List for discussions between the maintainers of database abstraction layers written in PHP ()

Hans Lellelid | 8 Mar 15:51

[phpdba] Re: Hello Database Abstractors

Hi guys,

By way of introduction, I am lead developer on Propel project [1].  Propel is an ORM layer for PHP based on
Apache Torque.   Similar to Manuel's Metastorage layer, it uses a code-generation approach that is a bit
different from other ActiveRecord/Rails-like tools.

I also wrote the Creole [2] DBAL (drawing heavily from the API for JDBC and implementations from PEAR DB,
MDB, ADOdb, etc.), which existed to provide a PHP5 (Exception-throwing, interface-using, etc.) DBAL
for use by the Propel project.  Since PDO has been released, however, we have moved Propel to use PDO for the
database API abstraction.  Creole is still a maintained project, but I certainly feel that supporting
these engine-level API layers is the right thing to do for everyone in the community.

I also *intend* to be helping Sven (also subscribed to list) on this new LinguaFranca [3] project, which is
meant to provide an abstraction for database metadata -- with the idea of parsers and renderers to pull in
from multiple sources (e.g. direct db connection, ddl dump, etc.) and then to write out to multiple
sources (e.g. create DDL, build Propel schema, MDB2 schema, etc.).  It's our solution for handling db
reverse-engineering for Propel2, but I think we all agreed that it made more sense for something like that
to be a standalone project.

Cheers,
Hans

[1] http://propel.phpdb.org/
[2] http://creole.phpdb.org/
[3] http://linguafranca.phpdb.org/

On Thu, 08 Mar 2007 10:18:59 +0100, Torsten Raudssus
<torsten@...> wrote:
> Hiho,
>

(Continue reading)

Torsten Raudssus | 8 Mar 16:19

Picon

Re: [phpdba] Re: Hello Database Abstractors

Hiho,
> By way of introduction, I am lead developer on Propel project [1].  Propel is an ORM layer for PHP based on
Apache Torque.   Similar to Manuel's Metastorage layer, it uses a code-generation approach that is a bit
different from other ActiveRecord/Rails-like tools.
>   
Code Generation we also use in DB_DO. I dont like the non code 
generation stuff anyway, cause that is really "poor" in usage. Wanna 
make me hot for propel? ;) <laugh>

Bye!

Hans Lellelid | 8 Mar 19:58

Re: [phpdba] Re: Hello Database Abstractors

On Thu, 08 Mar 2007 16:19:09 +0100, Torsten Raudssus
<torsten@...> wrote:
> Hiho,
>> By way of introduction, I am lead developer on Propel project [1]. 
> Propel is an ORM layer for PHP based on Apache Torque.   Similar to
> Manuel's Metastorage layer, it uses a code-generation approach that is a
> bit different from other ActiveRecord/Rails-like tools.
>>
> Code Generation we also use in DB_DO. I dont like the non code
> generation stuff anyway, cause that is really "poor" in usage. Wanna
> make me hot for propel? ;) <laugh>

Ah, I didn't realize.  I guess other tools like Doctrine also use code generation to a certain extent --
though it's a bit different there, since the code generation just builds the runtime database setup /
initialization.  Propel (and Metastorage, IIRC) builds base classes that are intimately aware of the
database structure.

Hans

Torsten Raudssus | 8 Mar 21:33

Picon

Re: [phpdba] Re: Hello Database Abstractors

Hiho,
> Ah, I didn't realize.  I guess other tools like Doctrine also use code generation to a certain extent --
though it's a bit different there, since the code generation just builds the runtime database setup /
initialization.  Propel (and Metastorage, IIRC) builds base classes that are intimately aware of the
database structure.
>   
In DB_DO(2) the concept is that the code generated just is for 
additional informations and class variables, so that you can sometimes 
even drop them, cause the setup of the fields is in an extra ini file. 
Tell me, did you ever took a look at DB_DataObject? Just to get some 
views of "others" :)

Hans Lellelid | 8 Mar 22:19

Re: [phpdba] Re: Hello Database Abstractors

On Thu, 08 Mar 2007 21:33:44 +0100, Torsten Raudssus
<torsten@...> wrote:
> Hiho,
>> Ah, I didn't realize.  I guess other tools like Doctrine also use code
> generation to a certain extent -- though it's a bit different there, since
> the code generation just builds the runtime database setup /
> initialization.  Propel (and Metastorage, IIRC) builds base classes that
> are intimately aware of the database structure.
>>
> In DB_DO(2) the concept is that the code generated just is for
> additional informations and class variables, so that you can sometimes
> even drop them, cause the setup of the fields is in an extra ini file.
> Tell me, did you ever took a look at DB_DataObject? Just to get some
> views of "others" :)

Yes, of course, but it was long ago.  Here were my (at that time) impressions of DB_DO (and others) &
motiviations for developing Propel: http://propel.phpdb.org/docs/en/user_guide/chapters/Introduction.html#Introduction.Motivations

Apparently, in reading that, I used to know that DB_DO did code generation. :)

Cheers,
Hans

David Zülke | 9 Mar 15:19

Pong (was: Re: [phpdba] Re: Hello Database Abstractors)

Hi everyone,

I'm also a lead developer of Propel, even though my project lead of  
the Agavi framework [1] has stolen most of my time recently, but I'll  
pick up development again next week, since Agavi is now stable and  
needs mostly documentation and an adequate website. It's about time I  
focus a bit on Propel again, we have great plans, especially for 2.0,  
and I'm curious if this list will give us a new perspective on  
certain matters.

One thing that _might_ be worth discussing here is a generic  
migrations library (similar to Rails') that can be used by any ORM  
layer, and any framework. Complicated, certainly, but we can  
rebuild^H^H^H^H^H^H^H think about it. We have the technology :)

Cheers,

David

[1] http://agavi.org/

Am 08.03.2007 um 15:51 schrieb Hans Lellelid:

> Hi guys,
>
> By way of introduction, I am lead developer on Propel project [1].   
> Propel is an ORM layer for PHP based on Apache Torque.   Similar to  
> Manuel's Metastorage layer, it uses a code-generation approach that  
> is a bit different from other ActiveRecord/Rails-like tools.
>

(Continue reading)

Lukas Kahwe Smith | 9 Mar 15:30

[phpdba] Re: Pong

David Zülke wrote:

> One thing that _might_ be worth discussing here is a generic migrations 
> library (similar to Rails') that can be used by any ORM layer, and any 
> framework. Complicated, certainly, but we can rebuild^H^H^H^H^H^H^H 
> think about it. We have the technology :)

Yes, this would be kind of cool indeed. I gave a talk on the topic. It 
does not contain the silver bullet, but instead is more a run down of 
the various approaches for SQL schema evolution management with their 
advantages and disadvantages:
http://pooteeweet.org/files/phptek06/database_schema_deployment.pdf

Since this is not run time dependent, end users would be more willing to 
use a common layer there, since its more a self contained "application" 
of sorts. However, this does not hide the fact that _we_ would have to 
choose one framework to build this on, and there we are back at square 
one. Since all the current DBAL/ORM efforts I know support the needed 
functionality partially.

So the only area for cooperation I see is in at least supporting the 
different serialization formats, since so far even those are fragmented. 
Metabase had the first format, but others still invented their own 
format for various reasons (some valid imho, some not so valid .. so it 
goes). And of course we can collaborate on best practices, SQL for 
specific databases, figureing out what data types should be chosen for 
portability etc.

regards,
Lukas

(Continue reading)

David Zülke | 9 Mar 15:43

Re: [phpdba] Re: Pong

It wouldn't use a framework. Instead, each ORM layer would generate  
migration rules that can then be extended with code that could also  
be specific to the respective framework. It's a huge undertaking, but  
with enough time and planning, it could be done. PHP lacks something  
like this. I'm okay with building a complete deployment solution,  
too, while we're at it (pretty much Rails' Capistrano).

David

Am 09.03.2007 um 15:30 schrieb Lukas Kahwe Smith:

> David Zülke wrote:
>
>> One thing that _might_ be worth discussing here is a generic  
>> migrations library (similar to Rails') that can be used by any ORM  
>> layer, and any framework. Complicated, certainly, but we can  
>> rebuild^H^H^H^H^H^H^H think about it. We have the technology :)
>
> Yes, this would be kind of cool indeed. I gave a talk on the topic.  
> It does not contain the silver bullet, but instead is more a run  
> down of the various approaches for SQL schema evolution management  
> with their advantages and disadvantages:
> http://pooteeweet.org/files/phptek06/database_schema_deployment.pdf
>
> Since this is not run time dependent, end users would be more  
> willing to use a common layer there, since its more a self  
> contained "application" of sorts. However, this does not hide the  
> fact that _we_ would have to choose one framework to build this on,  
> and there we are back at square one. Since all the current DBAL/ORM  
> efforts I know support the needed functionality partially.

(Continue reading)

Lukas Kahwe Smith | 9 Mar 15:50

Re: [phpdba] Re: Pong

David Zülke wrote:
> It wouldn't use a framework. Instead, each ORM layer would generate 
> migration rules that can then be extended with code that could also be 
> specific to the respective framework. It's a huge undertaking, but with 
> enough time and planning, it could be done. PHP lacks something like 
> this. I'm okay with building a complete deployment solution, too, while 
> we're at it (pretty much Rails' Capistrano).

migration rules?

not sure what these rules are supposed to cover. if its just about an 
array/object structure to define the necessary changes, then this is 
just yet another "serialization" format.

however while a good set of standards for this are important and coming 
up with good ones is a bit of work, the real work is in actually 
executing these changes. doing dependency resolutions for views, FKs, 
stored procedures etc.

regards,
Lukas

David Zülke | 9 Mar 16:13

Re: [phpdba] Re: Pong

Code to do the migration. Example: you built a small blog. Now you  
made it multi-user, and the postings table got a user_id field. Your  
migration code also inserts a first user into the users table, and  
then sets that user id on all entries in the postings table.

Another example. Your boss wants bigger thumbnails for product  
images. You simply sync the latest version to the production env, all  
files are copied, and then your migration code is run - nothing  
database-related here, it just generates new thumbnails for all  
fullsize versions of the images.

That's what Rails migrations and Capistrano can do. "rake deploy",  
and the new version is synced to production, "rake rollback" and  
we're back where we were before in case the latest version is foobar'd.

As I said, a huge task, and it would be clever if framework and ORM  
developers joined together in an effort to make a universal solution  
for that.

David

Am 09.03.2007 um 15:50 schrieb Lukas Kahwe Smith:

> David Zülke wrote:
>> It wouldn't use a framework. Instead, each ORM layer would  
>> generate migration rules that can then be extended with code that  
>> could also be specific to the respective framework. It's a huge  
>> undertaking, but with enough time and planning, it could be done.  
>> PHP lacks something like this. I'm okay with building a complete  
>> deployment solution, too, while we're at it (pretty much Rails'

(Continue reading)

Lukas Kahwe Smith | 9 Mar 16:42

Re: [phpdba] Re: Pong

David Zülke wrote:
> Code to do the migration. Example: you built a small blog. Now you made 
> it multi-user, and the postings table got a user_id field. Your 
> migration code also inserts a first user into the users table, and then 
> sets that user id on all entries in the postings table.

Yes, but how do you interact with the database? What ever migration code 
is generated .. it would need to use a specific database API .. and here 
I think we would quickly find us disagreeing about what database API to use.

> Another example. Your boss wants bigger thumbnails for product images. 
> You simply sync the latest version to the production env, all files are 
> copied, and then your migration code is run - nothing database-related 
> here, it just generates new thumbnails for all fullsize versions of the 
> images.

Hmm updating thumbnails .. this is really covered by rails? I mean I can 
see denormalization being handled by a "migration" layer, but also 
covering image resizing is kinda far off imho.

> That's what Rails migrations and Capistrano can do. "rake deploy", and 
> the new version is synced to production, "rake rollback" and we're back 
> where we were before in case the latest version is foobar'd.

Yeah, but as you can see in my talk, there are limitations in this 
approach. You cannot skip steps, which means you have to think very 
carefully when doing a release, because even if users skip releases, 
they might be hit by a change that for example removes a column, that is 
readded in the next release.

(Continue reading)

David Zülke | 9 Mar 17:02

Re: [phpdba] Re: Pong

Am 09.03.2007 um 16:42 schrieb Lukas Kahwe Smith:

> David Zülke wrote:
>> Code to do the migration. Example: you built a small blog. Now you  
>> made it multi-user, and the postings table got a user_id field.  
>> Your migration code also inserts a first user into the users  
>> table, and then sets that user id on all entries in the postings  
>> table.
>
> Yes, but how do you interact with the database? What ever migration  
> code is generated .. it would need to use a specific database  
> API .. and here I think we would quickly find us disagreeing about  
> what database API to use.

PDO, obviously :) But not exclusively, see my reply to Hans' mail.

>> Another example. Your boss wants bigger thumbnails for product  
>> images. You simply sync the latest version to the production env,  
>> all files are copied, and then your migration code is run -  
>> nothing database-related here, it just generates new thumbnails  
>> for all fullsize versions of the images.
>
> Hmm updating thumbnails .. this is really covered by rails? I mean  
> I can see denormalization being handled by a "migration" layer, but  
> also covering image resizing is kinda far off imho.

Not at all! The new software version requires different thumbnail  
sizes, so you have to create them during the upgrading process.  
That's what the so-called migrations are for.

(Continue reading)

Lukas Kahwe Smith | 9 Mar 17:12

Re: [phpdba] Re: Pong

David Zülke wrote:
> Am 09.03.2007 um 16:42 schrieb Lukas Kahwe Smith:
> 
>> David Zülke wrote:
>>> Code to do the migration. Example: you built a small blog. Now you 
>>> made it multi-user, and the postings table got a user_id field. Your 
>>> migration code also inserts a first user into the users table, and 
>>> then sets that user id on all entries in the postings table.
>>
>> Yes, but how do you interact with the database? What ever migration 
>> code is generated .. it would need to use a specific database API .. 
>> and here I think we would quickly find us disagreeing about what 
>> database API to use.
> 
> PDO, obviously :) But not exclusively, see my reply to Hans' mail.

I do not think this is feasible at all. PDO does not provide any real 
abstraction, so the end result would be creating yet another DBAL on top 
of PDO for this, though maybe optimized for this specific task.

Also keep in mind that I would want this layer to be able to do some 
smart stuff, that would need to integrate nicely with the DBAL/ORM at 
run time. Things like tree management (nested sets, adjacency lists 
etc), autoincrement/sequence emulation, specific portable data types 
(like the good old datetime vs. timestamp issues in MySQL) etc. So this 
framework would need to be very configureable to be able to handle all 
of the DBAL/ORM approaches to portability.

Finally some applications do want to manage DDL and things like that at 
runtime. So writing this framework would not spare all DBAL/ORM

(Continue reading)

David Zülke | 9 Mar 17:27

Re: [phpdba] Re: Pong

Am 09.03.2007 um 17:12 schrieb Lukas Kahwe Smith:

> David Zülke wrote:
>> Am 09.03.2007 um 16:42 schrieb Lukas Kahwe Smith:
>>> David Zülke wrote:
>>>> Code to do the migration. Example: you built a small blog. Now  
>>>> you made it multi-user, and the postings table got a user_id  
>>>> field. Your migration code also inserts a first user into the  
>>>> users table, and then sets that user id on all entries in the  
>>>> postings table.
>>>
>>> Yes, but how do you interact with the database? What ever  
>>> migration code is generated .. it would need to use a specific  
>>> database API .. and here I think we would quickly find us  
>>> disagreeing about what database API to use.
>> PDO, obviously :) But not exclusively, see my reply to Hans' mail.
>
> I do not think this is feasible at all. PDO does not provide any  
> real abstraction, so the end result would be creating yet another  
> DBAL on top of PDO for this, though maybe optimized for this  
> specific task.

Then you're wrong ;) PDO and native queries (there is no other way!)  
would be used to do the _schema_ changes. Drop a table, add a  
constraint, change a column. That is auto-generated by the respective  
counterpart in the ORM system - Propel would provide a "driver",  
Doctrine would provide a driver, etc. They'd all generate tasks for  
this system.

Next step: the user wants to add custom stuff. Then he uses the

(Continue reading)

Lukas Kahwe Smith | 9 Mar 17:31

Re: [phpdba] Re: Pong

David Zülke wrote:

>> I do not think this is feasible at all. PDO does not provide any real 
>> abstraction, so the end result would be creating yet another DBAL on 
>> top of PDO for this, though maybe optimized for this specific task.
> 
> Then you're wrong ;) PDO and native queries (there is no other way!) 
> would be used to do the _schema_ changes. Drop a table, add a 
> constraint, change a column. That is auto-generated by the respective 
> counterpart in the ORM system - Propel would provide a "driver", 
> Doctrine would provide a driver, etc. They'd all generate tasks for this 
> system.
> 
> Next step: the user wants to add custom stuff. Then he uses the engine 
> of his choice. Likely the same one he used to generate the "delta" task, 
> but not necessarily (for the very brave...)

I dont get what you want to do. Could you maybe provide some mockup code?

regards,
Lukas

David Zülke | 12 Mar 17:55

Re: [phpdba] Re: Pong

Am 09.03.2007 um 17:31 schrieb Lukas Kahwe Smith:

> David Zülke wrote:
>
>>> I do not think this is feasible at all. PDO does not provide any  
>>> real abstraction, so the end result would be creating yet another  
>>> DBAL on top of PDO for this, though maybe optimized for this  
>>> specific task.
>> Then you're wrong ;) PDO and native queries (there is no other  
>> way!) would be used to do the _schema_ changes. Drop a table, add  
>> a constraint, change a column. That is auto-generated by the  
>> respective counterpart in the ORM system - Propel would provide a  
>> "driver", Doctrine would provide a driver, etc. They'd all  
>> generate tasks for this system.
>> Next step: the user wants to add custom stuff. Then he uses the  
>> engine of his choice. Likely the same one he used to generate the  
>> "delta" task, but not necessarily (for the very brave...)
>
> I dont get what you want to do. Could you maybe provide some mockup  
> code?

Will do in a couple of days, sorry for the delay

David

Hans Lellelid | 9 Mar 16:24

Re: [phpdba] Re: Pong

Hi David ;)

So, just to clarify, what you're proposing is more about migrating *data* and less about migrating the
schemas themselves, right?  I assume in this model, you would simply make the db changes and then run this
migration PHP code to actually fix the data?  Or do you envision that these ORM frameworks would also create
alter DDL?

This solution strikes me as far more elegant (in its simplicity) than the idea of building alter DDL which
won't cause (undesired) data loss -- and it takes advantage of the fact that the ORM layer is more abstract
and flexible than the underlying database.

Hans

On Fri, 9 Mar 2007 16:13:42 +0100, David Zülke <dz@...> wrote:
> Code to do the migration. Example: you built a small blog. Now you  
> made it multi-user, and the postings table got a user_id field. Your  
> migration code also inserts a first user into the users table, and  
> then sets that user id on all entries in the postings table.
> 
> Another example. Your boss wants bigger thumbnails for product  
> images. You simply sync the latest version to the production env, all  
> files are copied, and then your migration code is run - nothing  
> database-related here, it just generates new thumbnails for all  
> fullsize versions of the images.
> 
> That's what Rails migrations and Capistrano can do. "rake deploy",  
> and the new version is synced to production, "rake rollback" and  
> we're back where we were before in case the latest version is foobar'd.
> 
> As I said, a huge task, and it would be clever if framework and ORM

(Continue reading)

David Zülke | 9 Mar 17:10

Re: [phpdba] Re: Pong

Well... remember the offer Dominik and I made to implement schema  
deltas and automated database upgrading for 2.0? That would be the  
heart of this migration system.

My idea was that each ORM layer would generate migration... tasks...?  
that are then run by the migration engine. These tasks can be  
extended using code - either plain PHP and PDO, or Propel (if you're  
a Propel user), for Agavi or symfony (not that easy there, but  
doable) you'd have a special context to access even more application- 
related stuff etc. Imagine you add a row to the table, but you need  
relatively complicated logic to determine what goes into that new  
cloumn for each of the existing rows.

The fundamental design principle would be that it is totally ORM- and  
framework-agnostic, but with the ability to tie in any of these.  
People could then even use the system without any ORM layer or  
framework, they would just have to write the tasks (= SQL, code, ...)  
themselves. For instance, Phing could be the execution base for that  
task system.

It would offer the ability to also deploy these changes to the  
production environment. Capistrano does that very well, but it's  
Ruby. If we tackled that for PHP, it would not only make our own  
lives easier, but we'd also drown in an ocean of love letters from  
happy developers all over the world ;) Problem is, it's way too large  
a task to tackle for one framework/orm/whatever team alone. It's only  
possible with collaboration, even though we probably can copy large  
chunks of Capistrano and Rails' migrations engine.

David

(Continue reading)

Lukas Kahwe Smith | 9 Mar 17:14

Re: [phpdba] Re: Pong

David Zülke wrote:

> My idea was that each ORM layer would generate migration... tasks...? 
> that are then run by the migration engine. These tasks can be extended 
> using code - either plain PHP and PDO, or Propel (if you're a Propel 
> user), for Agavi or symfony (not that easy there, but doable) you'd have 
> a special context to access even more application-related stuff etc. 
> Imagine you add a row to the table, but you need relatively complicated 
> logic to determine what goes into that new cloumn for each of the 
> existing rows.

Aha .. so this validates my point. You seem to assume that developers 
from other ORM's would just jump on the opportunity to write such a 
framework on top of Propel. While Doctrine developers will probably 
claim, that they would be just as good or even better base etc.

regards,
Lukas

David Zülke | 9 Mar 17:30

Re: [phpdba] Re: Pong

Am 09.03.2007 um 17:14 schrieb Lukas Kahwe Smith:

> David Zülke wrote:
>
>> My idea was that each ORM layer would generate migration...  
>> tasks...? that are then run by the migration engine. These tasks  
>> can be extended using code - either plain PHP and PDO, or Propel  
>> (if you're a Propel user), for Agavi or symfony (not that easy  
>> there, but doable) you'd have a special context to access even  
>> more application-related stuff etc. Imagine you add a row to the  
>> table, but you need relatively complicated logic to determine what  
>> goes into that new cloumn for each of the existing rows.
>
> Aha .. so this validates my point. You seem to assume that  
> developers from other ORM's would just jump on the opportunity to  
> write such a framework on top of Propel. While Doctrine developers  
> will probably claim, that they would be just as good or even better  
> base etc.

No, no, it doesn't validate your point, and nowhere did I say it  
would be Propel only! I thought it was pretty obvious that the "or"  
indicated choice. It could be Doctrine, too, and whatever else there  
is. It would be a (fully functional!) foundation ORM and framework  
developers could offer additional integration for (if necessary,  
which it probably isn't anyway, see the other mail).

David

Return

Return to gmane.comp.php.dbabstraction.

Project Web Page

List for discussions between the maintainers of database abstraction layers written in PHP ()

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane