gmane.comp.web.zope.plone.devel

Florian Schulze | 20 Nov 17:23

fake-zope-eggs harmful for your buildout!


Hi!

Today I narrowed down a bug which hunted us in several more complex  
buildouts. The problem is that some egg dependencies aren't picked up  
properly and we had to add them directly to the buildout.cfg eggs list  
instead of relying on buildout/setuptools. I finally found the cause today!

If you depend on a fake egg, dependencies break.

I attached a very simple buildout which demonstrates the problem. After  
you ran buildout you can check bin/instance and see that two eggs (bar and  
blubb) are missing in the sys.path. In the zc.recipe.egg created bin/foo  
script they are added correctly.

Wichert just added that Plone 3.2 depends on fake eggs, so this is  
critical to fix and any help is appreciated.

Regards,
Florian Schulze

Attachment (buildouttest.tgz): application/x-gzip, 3471 bytes

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/

_______________________________________________
Plone-developers mailing list

(Continue reading)

Martijn Pieters | 20 Nov 20:28

Re: fake-zope-eggs harmful for your buildout!

On Thu, Nov 20, 2008 at 17:24, Florian Schulze <florian.schulze@...> wrote:
>> Today I narrowed down a bug which hunted us in several more complex
> buildouts. The problem is that some egg dependencies aren't picked up
> properly and we had to add them directly to the buildout.cfg eggs list
> instead of relying on buildout/setuptools. I finally found the cause today!
>
> If you depend on a fake egg, dependencies break.

Could it be that the way fake eggs are generated is at fault? I
remember vaguely creating my own fake egg solution (low tech) to fake
the ZODB3 egg after I found the then-current recipe method for fake
eggs didn't actually work:

  [zodb3-fakeegg]
  recipe = plone.recipe.command
  command =
      echo Metadata-Version: 1.0 >
${zope2:location}/lib/python/ZODB/ZODB3.egg-info
      echo Name: ZODB3 >> ${zope2:location}/lib/python/ZODB/ZODB3.egg-info
      echo Version: 3.7.0 >> ${zope2:location}/lib/python/ZODB/ZODB3.egg-info
      echo ${zope2:location}/lib/python/ZODB >
${buildout:develop-eggs-directory}/ZODB3.egg-link
  update-command = ${zodb3-fakeegg:command}

What do zope2instance fake eggs look like?

--

-- 
Martijn Pieters

-------------------------------------------------------------------------

(Continue reading)

Jean-François Roche | 20 Nov 20:46

Re: fake-zope-eggs harmful for your buildout!

Martijn Pieters wrote:
> On Thu, Nov 20, 2008 at 17:24, Florian Schulze <florian.schulze@...> wrote:
>>> Today I narrowed down a bug which hunted us in several more complex
>> buildouts. The problem is that some egg dependencies aren't picked up
>> properly and we had to add them directly to the buildout.cfg eggs list
>> instead of relying on buildout/setuptools. I finally found the cause today!
>>
>> If you depend on a fake egg, dependencies break.
> 
> Could it be that the way fake eggs are generated is at fault? I
> remember vaguely creating my own fake egg solution (low tech) to fake
> the ZODB3 egg after I found the then-current recipe method for fake
> eggs didn't actually work:
> 
>   [zodb3-fakeegg]
>   recipe = plone.recipe.command
>   command =
>       echo Metadata-Version: 1.0 >
> ${zope2:location}/lib/python/ZODB/ZODB3.egg-info
>       echo Name: ZODB3 >> ${zope2:location}/lib/python/ZODB/ZODB3.egg-info
>       echo Version: 3.7.0 >> ${zope2:location}/lib/python/ZODB/ZODB3.egg-info
>       echo ${zope2:location}/lib/python/ZODB >
> ${buildout:develop-eggs-directory}/ZODB3.egg-link
>   update-command = ${zodb3-fakeegg:command}
> 
> What do zope2instance fake eggs look like?
> 

It's here:

(Continue reading)

Jean-François Roche | 20 Nov 23:42

Re: fake-zope-eggs harmful for your buildout!

Hello,

After 1h30 of debugging in zc.buildout & setuptools I still don't see
excaclty what is happining.

It's nearby the resolve(requirement, env, installer) [pkg_resources
L494] of setuptools 0.6c9 which is called by zc.buildout
[zc.buildout.easy_install L622]. bar isn't added to the WorkingSet as
pkg_resources.DistributionNotFound isn't raise for bar but I don't see why.

		Jeff

Jean-François Roche wrote:
> Martijn Pieters wrote:
>> On Thu, Nov 20, 2008 at 17:24, Florian Schulze <florian.schulze@...> wrote:
>>>> Today I narrowed down a bug which hunted us in several more complex
>>> buildouts. The problem is that some egg dependencies aren't picked up
>>> properly and we had to add them directly to the buildout.cfg eggs list
>>> instead of relying on buildout/setuptools. I finally found the cause today!
>>>
>>> If you depend on a fake egg, dependencies break.
>> Could it be that the way fake eggs are generated is at fault? I
>> remember vaguely creating my own fake egg solution (low tech) to fake
>> the ZODB3 egg after I found the then-current recipe method for fake
>> eggs didn't actually work:
>>
>>   [zodb3-fakeegg]
>>   recipe = plone.recipe.command
>>   command =
>>       echo Metadata-Version: 1.0 >

(Continue reading)

Wichert Akkerman | 24 Nov 08:50

Re: fake-zope-eggs harmful for your buildout!

FYI: this is now holding up the Plone 3.2rc1 release. We have several
packages in 3.2 which depend on zope.* packages and require fake eggs.

Has anyone approach Jim Fulton about this?

Wichert.

Previously Jean-François Roche wrote:
> Hello,
> 
> After 1h30 of debugging in zc.buildout & setuptools I still don't see
> excaclty what is happining.
> 
> It's nearby the resolve(requirement, env, installer) [pkg_resources
> L494] of setuptools 0.6c9 which is called by zc.buildout
> [zc.buildout.easy_install L622]. bar isn't added to the WorkingSet as
> pkg_resources.DistributionNotFound isn't raise for bar but I don't see why.
> 
> 
> 		Jeff
> 
> Jean-François Roche wrote:
> > Martijn Pieters wrote:
> >> On Thu, Nov 20, 2008 at 17:24, Florian Schulze <florian.schulze@...t> wrote:
> >>>> Today I narrowed down a bug which hunted us in several more complex
> >>> buildouts. The problem is that some egg dependencies aren't picked up
> >>> properly and we had to add them directly to the buildout.cfg eggs list
> >>> instead of relying on buildout/setuptools. I finally found the cause today!
> >>>
> >>> If you depend on a fake egg, dependencies break.

(Continue reading)

Hanno Schlichting | 27 Nov 14:32

Re: fake-zope-eggs harmful for your buildout!

Hi.

Jean-François Roche wrote:
> After 1h30 of debugging in zc.buildout & setuptools I still don't see
> excaclty what is happining.
> 
> It's nearby the resolve(requirement, env, installer) [pkg_resources
> L494] of setuptools 0.6c9 which is called by zc.buildout
> [zc.buildout.easy_install L622]. bar isn't added to the WorkingSet as
> pkg_resources.DistributionNotFound isn't raise for bar but I don't see why.

Is the test case Florian provided clear and small? Did someone post it
to the distutils-sig list or contact Jim Fulton directly?

This is blocking the 3.2 release right now :(

Hanno

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Plone-developers mailing list
Plone-developers <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plone-developers

Jean-François Roche | 27 Nov 15:25

Re: fake-zope-eggs harmful for your buildout!

Hello,

Hanno Schlichting wrote:
> Hi.
> 
> Jean-François Roche wrote:
>> After 1h30 of debugging in zc.buildout & setuptools I still don't see
>> excaclty what is happining.
>>
>> It's nearby the resolve(requirement, env, installer) [pkg_resources
>> L494] of setuptools 0.6c9 which is called by zc.buildout
>> [zc.buildout.easy_install L622]. bar isn't added to the WorkingSet as
>> pkg_resources.DistributionNotFound isn't raise for bar but I don't see why.
> 
> Is the test case Florian provided clear and small?

Yes it is. Will try to dig further tonight and if I still can't make it
I will ask help to Jim.

Regards,

		Jeff

> Did someone post it
> to the distutils-sig list or contact Jim Fulton directly?
> 
> This is blocking the 3.2 release right now :(
> 
> Hanno
>

(Continue reading)

Florian Schulze | 27 Nov 16:46

Re: fake-zope-eggs harmful for your buildout!

On Thu, 20 Nov 2008 23:42:11 +0100, Jean-François Roche  
<jfroche@...> wrote:

> Hello,
>
> After 1h30 of debugging in zc.buildout & setuptools I still don't see
> excaclty what is happining.
>
> It's nearby the resolve(requirement, env, installer) [pkg_resources
> L494] of setuptools 0.6c9 which is called by zc.buildout
> [zc.buildout.easy_install L622]. bar isn't added to the WorkingSet as
> pkg_resources.DistributionNotFound isn't raise for bar but I don't see  
> why.

I'm also not sure why it's not raising DistributionNotFound, it seems like  
it's picked up from the environment which contains the debug eggs, but  
that environment is only initialized if the fake egg is installed, not  
without out. Anyway, I have at least a fix for the symptoms, if we can't  
find the real cause.

the ws.resolve call returns a list of eggs to activate, I'm going through  
them and add them to the working set, it is possible that this is the  
wrong approach for non develop eggs, but I couldn't create a buildout  
which shows the error with non develop eggs yet. I was also unable to  
write a test case for zc.buildout.

Comments welcome!

Regards,
Florian Schulze

(Continue reading)

Tarek Ziadé | 7 Dec 16:16

Re: fake-zope-eggs harmful for your buildout!


Don't touch zc.buildout here.

The fake zope stuff introduced an unexpected behavior because the way it
created the fake eggs
is wrong:

when zc.buildout and setuptools scans the develop-eggs folder, the path used
for each fack eggs 
is also the path where the regular develop eggs are. 

This makes the resolve code pass when DistributionsError should be raised.

I made your buildout work with a slight change in the way fake zope eggs are
created.

This is my solution:

instead of creating *-info files inside the develop eggs folder, this tool
should:

- create it in a distinct folder (let's say "fake-eggs" by default maybe)
- create a distinct folder with the name of the egg, with the *-info stuff
inside that folder
- add a *-link file inside the develop-eggs folder, pointing to each folder
located in fake-eggs

this will fake the eggs the right way and won't brake the way setuptools
scans the
folders to build its environement, therefore the way zc.buildout deals with

(Continue reading)

Florian Schulze | 10 Dec 02:20

Re: fake-zope-eggs harmful for your buildout!

On Sun, 07 Dec 2008 16:16:17 +0100, Tarek Ziadé  
<ziade.tarek <at> gmail.com> wrote:

> The fake zope stuff introduced an unexpected behavior because the way it
> created the fake eggs is wrong:

...

> this will fake the eggs the right way and won't brake the way setuptools
> scans the
> folders to build its environement, therefore the way zc.buildout deals  
> with resolve.

Thank you very much for this explanation.

> Let me know if you would like me to write this patch for this recipe, I  
> have it in mind,
>
> (but I guess JFroche can do it or chat with me on IRC if it's unclear)

If we can get this fixed, then the Plone 3.2 release can finally be done.

Regards,
Florian Schulze

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/

(Continue reading)

Jean-François Roche | 10 Dec 09:33

Re: fake-zope-eggs harmful for your buildout!

Thanks Tarek for the explanation, I will look at that today (might be
coming back to you by irc).

			Jeff

Florian Schulze wrote:
> On Sun, 07 Dec 2008 16:16:17 +0100, Tarek Ziadé  
> <ziade.tarek <at> gmail.com> wrote:
> 
>> The fake zope stuff introduced an unexpected behavior because the way it
>> created the fake eggs is wrong:
> 
> ...
> 
>> this will fake the eggs the right way and won't brake the way setuptools
>> scans the
>> folders to build its environement, therefore the way zc.buildout deals  
>> with resolve.
> 
> Thank you very much for this explanation.
> 
>> Let me know if you would like me to write this patch for this recipe, I  
>> have it in mind,
>>
>> (but I guess JFroche can do it or chat with me on IRC if it's unclear)
> 
> If we can get this fixed, then the Plone 3.2 release can finally be done.
> 
> Regards,
> Florian Schulze

(Continue reading)

Jean-François Roche | 11 Dec 22:28

Re: fake-zope-eggs harmful for your buildout!

It's fixed. kudos to Tarek !

Regards,

		Jeff

Jean-François Roche wrote:
> Thanks Tarek for the explanation, I will look at that today (might be
> coming back to you by irc).
> 
> 
> 			Jeff
> 
> Florian Schulze wrote:
>> On Sun, 07 Dec 2008 16:16:17 +0100, Tarek Ziadé  
>> <ziade.tarek <at> gmail.com> wrote:
>>
>>> The fake zope stuff introduced an unexpected behavior because the way it
>>> created the fake eggs is wrong:
>> ...
>>
>>> this will fake the eggs the right way and won't brake the way setuptools
>>> scans the
>>> folders to build its environement, therefore the way zc.buildout deals  
>>> with resolve.
>> Thank you very much for this explanation.
>>
>>> Let me know if you would like me to write this patch for this recipe, I  
>>> have it in mind,
>>>

(Continue reading)

Tarek Ziade | 11 Dec 22:40

Re: fake-zope-eggs harmful for your buildout!

Thanks Jeff ! ;)

2008/12/11 Jean-François Roche <jfroche <at> affinitic.be>:
> It's fixed. kudos to Tarek !
>
> Regards,
>
>                Jeff
>
> Jean-François Roche wrote:
>> Thanks Tarek for the explanation, I will look at that today (might be
>> coming back to you by irc).
>>
>>
>>                       Jeff
>>
>> Florian Schulze wrote:
>>> On Sun, 07 Dec 2008 16:16:17 +0100, Tarek Ziadé
>>> <ziade.tarek <at> gmail.com> wrote:
>>>
>>>> The fake zope stuff introduced an unexpected behavior because the way it
>>>> created the fake eggs is wrong:
>>> ...
>>>
>>>> this will fake the eggs the right way and won't brake the way setuptools
>>>> scans the
>>>> folders to build its environement, therefore the way zc.buildout deals
>>>> with resolve.
>>> Thank you very much for this explanation.
>>>

(Continue reading)

Florian Schulze | 12 Dec 13:18

Re: fake-zope-eggs harmful for your buildout!

On Thu, 11 Dec 2008 22:28:09 +0100, Jean-François Roche  
<jfroche <at> affinitic.be> wrote:

> It's fixed. kudos to Tarek !

I can confirm the fix and made a 2.6 release of plone.recipe.zope2install.

Thanks to all involved!

*WARNING*: You have to remove your 'develop-eggs' folder before running  
buildout, or the fix won't work.

Regards,
Florian Schulze

------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you.  Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
Plone-developers mailing list
Plone-developers <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/plone-developers

Tarek Ziade | 12 Dec 13:37

Re: fake-zope-eggs harmful for your buildout!

I suggest that the recipe raises an error if it finds a .egg-info
folder inside the
develop-eggs folder.

2008/12/12 Florian Schulze <florian.schulze <at> gmx.net>:
> On Thu, 11 Dec 2008 22:28:09 +0100, Jean-François Roche
> <jfroche <at> affinitic.be> wrote:
>
>> It's fixed. kudos to Tarek !
>
> I can confirm the fix and made a 2.6 release of plone.recipe.zope2install.
>
> Thanks to all involved!
>
> *WARNING*: You have to remove your 'develop-eggs' folder before running
> buildout, or the fix won't work.
>
> Regards,
> Florian Schulze
>
>
> ------------------------------------------------------------------------------
> SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
> The future of the web can't happen without you.  Join us at MIX09 to help
> pave the way to the Next Web now. Learn more and register at
> http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
> _______________________________________________
> Plone-developers mailing list
> Plone-developers <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/plone-developers

(Continue reading)

Return

Return to gmane.comp.web.zope.plone.devel.

Advertisement

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane