Development of Minimal Python (written in python)

headers Marcus Smith | 31 Jul 2012 05:04

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev

2012/7/30 Marcus Smith <qwcode <at> gmail.com>:
> Hello:
>
> I'm working on trying to get the pip test suite passing for pypy.
> I need to figure out the mechanism that turns on/off  the use of
> __pycache__ folders.
> locally in my pypy v1.9 install, I only see normal *.pyc files for installed
> distributions.
> In Travis service builds which also reports v1.9, there are __pycache__
> folders with *.pypy-19.pyc files.
>
> How is this set or determined in pypy?

py.test is probably creating them.

--

-- 
Regards,
Benjamin

IMHO, *.pyc files should not be distributed with a binary pypy. This is 
probably an issue for the team who supply the pypy for travis-ci.org 
(since the __pycache__ files do not appear in the nightly or release 
builds on pypy.org) - I think they live at https://launchpad.net/~pypy 
<https://launchpad.net/%7Epypy>
Matti

On 31/07/2012 1:28 PM, Benjamin Peterson wrote:
> 2012/7/30 Marcus Smith <qwcode <at> gmail.com>:
>> Hello:
>>
>> I'm working on trying to get the pip test suite passing for pypy.
>> I need to figure out the mechanism that turns on/off  the use of
>> __pycache__ folders.
>> locally in my pypy v1.9 install, I only see normal *.pyc files for installed
>> distributions.
>> In Travis service builds which also reports v1.9, there are __pycache__
>> folders with *.pypy-19.pyc files.
>>
>> How is this set or determined in pypy?
> py.test is probably creating them.
>
>   
>

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Hi Marcus,

PyPy contains no logic to create or search for __pycache__.  A grep
tells me that neither does CPython 2.7.  You are probably confused by
py.test creating them.

A bientôt,

Armin.

Hi Marcus, Armin, 

On Tue, Jul 31, 2012 at 09:25 +0200, Armin Rigo wrote:
> Hi Marcus,
> 
> PyPy contains no logic to create or search for __pycache__.  A grep
> tells me that neither does CPython 2.7.  You are probably confused by
> py.test creating them.

... creating them during a pypy test run - the artificats (*.pyc files) of 
such runs should not be distributed as Matti points out.  Not sure it helps
in this case but in general you can set PYTHONDONTWRITEBYTECODE=1 to
prevent writing of such files.

best,
holger

> 
> A bientôt,
> 
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev <at> python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
> 
Hi Armin (2012.07.31_09:25:08_+0200)
> PyPy contains no logic to create or search for __pycache__.  A grep
> tells me that neither does CPython 2.7.  You are probably confused by
> py.test creating them.

A Debian/Ubuntu pypy does, though. And IIRC, that's what travis CI is
using.

It's the pep3147 patches:
http://anonscm.debian.org/gitweb/?p=collab-maint/pypy.git;a=tree;f=debian/patches

I thought it would be a useful feature, to make transition between pypy
versions in Debian easier. Proposed the patches in #pypy, and the
reception was lukewarm, so I never pushed harder.

SR

--

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

Hi Stefano,

On Tue, Jul 31, 2012 at 11:44 AM, Stefano Rivera <stefano <at> rivera.za.net> wrote:
>> PyPy contains no logic to create or search for __pycache__.  A grep
>> tells me that neither does CPython 2.7.
>
> A Debian/Ubuntu pypy does, though.

I think that PyPy shouldn't do that, because CPython doesn't.  If
Debian/Ubuntu also hacked CPython to do that, then fine; we are then
free to point users to them (blame them?) for issues 

A bientôt,

Armin.

Hi Armin (2012.07.31_12:10:01_+0200)
> I think that PyPy shouldn't do that, because CPython doesn't.  If
> Debian/Ubuntu also hacked CPython to do that, then fine; we are then
> free to point users to them (blame them?) for issues 

We did seriously consider hacking cpython to do that, before introducing
2.7. But by that point, there was no real benefit, as 2.7 is the last of
the 2.x series.
https://lists.debian.org/debian-python/2010/04/msg00046.html

I did it for pypy, as I thought it fairly likely that there would be a
benefit (there's no guarantee that pypy's bytecode format won't change
between pypy releases).

Only very esoteric code would break as a result of this change.

PyPy already has slightly different .pyc semantics to cpython, and has
PEP3149, a vaguely similar change, for C extensions.

It's worth stating that I did ask if there was any objection to the
patches I was applying, more than once, before the first upload to
Debian, and received none. The general opinion seemed to be that .pyc
files aren't liked much in the pypy community, at all, and this was
vaguely inline

SR

--

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

On Tue, Jul 31, 2012 at 12:35 PM, Stefano Rivera <stefano <at> rivera.za.net> wrote:
> Hi Armin (2012.07.31_12:10:01_+0200)
>> I think that PyPy shouldn't do that, because CPython doesn't.  If
>> Debian/Ubuntu also hacked CPython to do that, then fine; we are then
>> free to point users to them (blame them?) for issues 
>
> We did seriously consider hacking cpython to do that, before introducing
> 2.7. But by that point, there was no real benefit, as 2.7 is the last of
> the 2.x series.
> https://lists.debian.org/debian-python/2010/04/msg00046.html
>
> I did it for pypy, as I thought it fairly likely that there would be a
> benefit (there's no guarantee that pypy's bytecode format won't change
> between pypy releases).
>
> Only very esoteric code would break as a result of this change.

Never underestimate users. Just saying

>
> PyPy already has slightly different .pyc semantics to cpython, and has
> PEP3149, a vaguely similar change, for C extensions.
>
> It's worth stating that I did ask if there was any objection to the
> patches I was applying, more than once, before the first upload to
> Debian, and received none. The general opinion seemed to be that .pyc
> files aren't liked much in the pypy community, at all, and this was
> vaguely inline
>

On 31.07.12 12:35, Stefano Rivera wrote:
> Hi Armin (2012.07.31_12:10:01_+0200)
>> I think that PyPy shouldn't do that, because CPython doesn't.  If
>> Debian/Ubuntu also hacked CPython to do that, then fine; we are then
>> free to point users to them (blame them?) for issues 
> We did seriously consider hacking cpython to do that, before introducing
> 2.7. But by that point, there was no real benefit, as 2.7 is the last of
> the 2.x series.
> https://lists.debian.org/debian-python/2010/04/msg00046.html
>
> I did it for pypy, as I thought it fairly likely that there would be a
> benefit (there's no guarantee that pypy's bytecode format won't change
> between pypy releases).
>
> Only very esoteric code would break as a result of this change.
>
> PyPy already has slightly different .pyc semantics to cpython, and has
> PEP3149, a vaguely similar change, for C extensions.
>
> It's worth stating that I did ask if there was any objection to the
> patches I was applying, more than once, before the first upload to
> Debian, and received none. The general opinion seemed to be that .pyc
> files aren't liked much in the pypy community, at all, and this was
> vaguely inline

Well, good to know.

Your backport makes sense, but the reasoning should be identical
between cpython and pypy.
If cpython does not do it, pypy does not do it.

The bytecode compatibility is also identical between cpython and pypy,
there is no issue here that is not there.

pypy currently shows itself as python 2.7.2, and that is the truth.
Different behavior introduced by a distribution is IMHO not a
benefit, but goes into an incompatible direction.

If a backport happens, then in CPython, and PyPy will adjusted, accordingly.
One of the incompatibility issues _is_ the unexpected behavior of tests.

To Marcus:
Now that the issue is clear, it probably makes sense to add some extra check
to setup.py that does not inquire the python version but does some probing.
Better to be defensive 

cheers - chris

--

-- 
Christian Tismer             :^)   <mailto:tismer <at> stackless.com>
Software Consulting          :     Have a break! Take a ride on Python's
Karl-Liebknecht-Str. 121     :    *Starship* http://starship.python.net/
14482 Potsdam                :     PGP key -> http://pgp.uni-mainz.de
phone +49 173 24 18 776  fax +49 (30) 700143-0023
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

Hi Christian (2012.07.31_13:05:48_+0200)
> >vaguely inline

Hrm, I never finished that sentance, because I went back and made the
point earlier in the mail.

> Your backport makes sense, but the reasoning should be identical
> between cpython and pypy.
> If cpython does not do it, pypy does not do it.

cpython has a long legacy in Debian, and many existing packages that
depend on particular semantics. PyPy can afford to make changes that
cpython can't. Using PEP3147 + 3149 means that I only need to put python
modules in one place, for all versions of PyPy to consume.

I don't want to end up having to carry multiple versions of pypy in the
archive, as we have to with cpython 2.x. With 3.x, we've already seen
the benefit of PEP3147 + 3149.

> The bytecode compatibility is also identical between cpython and pypy,
> there is no issue here that is not there.

I believe that isn't the case:
https://bitbucket.org/pypy/pypy/src/6bb09a17e2aa/pypy/module/imp/importing.py#cl-832

> pypy currently shows itself as python 2.7.2, and that is the truth.
> Different behavior introduced by a distribution is IMHO not a
> benefit, but goes into an incompatible direction.

And I do my best to minimize that. In this case, I thought it was worth
the delta.

SR

--

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

Hi Stefano,

On Tue, Jul 31, 2012 at 1:34 PM, Stefano Rivera <stefano <at> rivera.za.net> wrote:
> And I do my best to minimize that. In this case, I thought it was worth
> the delta.

My point of view is that this goes in the same pack of
incompatibilities introduced by some distributions as splitting the
install path (which we recommend to install as one block in
/opt/pypy-1.9).  It may be useful to do that according to some
distribution-specific rules, but either you do it as a distribution
hack and we don't care about it, or you try to come here and discuss
more in details the problems that you have and how we could solve it
together.  As far as I know the __pycache__ is another example of
distribution-specific hack that we were was not aware of.  I don't
think we are by principle necessarily completely opposed to it:
whereas it still looks like a good idea to stick to CPython's
behavior, failing that, if there are good reasons, the next best
solution is probably to come with something that we can integrate into
the "real" pypy, rather than keep on your own.  If you don't, users
complain here and you are just making us hate .pyc files even more.

A bientôt,

Armin.

Hi Armin (2012.07.31_16:26:50_+0200)
> My point of view is that this goes in the same pack of
> incompatibilities introduced by some distributions as splitting the
> install path (which we recommend to install as one block in
> /opt/pypy-1.9).

Most distros won't use /opt. We tend to follow the FHS.
/opt makes a lot of sense for the binary pypy you distribute. Just as it
makes sense for the binary junk that Adobe et al. distribute.

> It may be useful to do that according to some distribution-specific
> rules, but either you do it as a distribution hack and we don't care
> about it, or you try to come here and discuss more in details the
> problems that you have and how we could solve it together.

Having the same installation layout on all distros would be fantastic.
Practically, that gets non-trivial, though. Debian has the weird
dist-packages layout for cpython, that I copied for pypy, but that
approach hasn't been documented in upstream cpython or adopted by any
other distributions.

> As far as I know the __pycache__ is another example of
> distribution-specific hack that we were was not aware of.

I did my best to make you all aware of it, and get some feedback.
I mentioned the PEP3147 support here [0], and a zillion times on IRC.
But I have no problem having that conversation now that people seem to
be aware of it :)

[0]: http://mail.python.org/pipermail/pypy-dev/2012-January/009104.html

> I don't think we are by principle necessarily completely opposed to it:
> whereas it still looks like a good idea to stick to CPython's
> behavior, failing that, if there are good reasons, the next best
> solution is probably to come with something that we can integrate into
> the "real" pypy, rather than keep on your own.

Yup, that's the opinion I felt before. And the approach I'm taking. But
the patch is fairly gnarly due to test-suite changes, so I haven't
forwarded it up to you.

SR

--

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

Hi,

<rant>
i would dare to say that Distributions that manage semi-compatible,
but distro specific "forks" of open source projects
instead of aiming for a distro compatible upstream
create quite some grief for developers.

after all users don't tend to notice those incompatibilities,
but it breaks the tooling of developers

I'm at a point where i need to run self-compiled pythons,
since the ones in Debian for example are broken
for some of my use-cases by Debian specific patches/changes

what makes it worse is that the error reporting paths for the distro 
specific "forks" aren't made clear
</rant>

-- Ronny

On 07/31/2012 04:26 PM, Armin Rigo wrote:
> Hi Stefano,
>
> On Tue, Jul 31, 2012 at 1:34 PM, Stefano Rivera<stefano <at> rivera.za.net>  wrote:
>> And I do my best to minimize that. In this case, I thought it was worth
>> the delta.
>
> My point of view is that this goes in the same pack of
> incompatibilities introduced by some distributions as splitting the
> install path (which we recommend to install as one block in
> /opt/pypy-1.9).  It may be useful to do that according to some
> distribution-specific rules, but either you do it as a distribution
> hack and we don't care about it, or you try to come here and discuss
> more in details the problems that you have and how we could solve it
> together.  As far as I know the __pycache__ is another example of
> distribution-specific hack that we were was not aware of.  I don't
> think we are by principle necessarily completely opposed to it:
> whereas it still looks like a good idea to stick to CPython's
> behavior, failing that, if there are good reasons, the next best
> solution is probably to come with something that we can integrate into
> the "real" pypy, rather than keep on your own.  If you don't, users
> complain here and you are just making us hate .pyc files even more.
> 
>
>
> A bientôt,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev <at> python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

Hi Ronny (2012.07.31_17:12:15_+0200)
> i would dare to say that Distributions that manage semi-compatible,
> but distro specific "forks" of open source projects
> instead of aiming for a distro compatible upstream
> create quite some grief for developers.

Weighed up against making grief for users.

> after all users don't tend to notice those incompatibilities,
> but it breaks the tooling of developers

We really try not to make changes that break tooling. And when we do, to
fix the tools, upstream.

> I'm at a point where i need to run self-compiled pythons,
> since the ones in Debian for example are broken
> for some of my use-cases by Debian specific patches/changes

And that was the point of the dist-packages change.

> what makes it worse is that the error reporting paths for the distro
> specific "forks" aren't made clear

Distros have bug trackers.

SR

--

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559

pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

__pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

Re: __pycache__ folders

pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders

Re: pycache folders