headers
Scott Kitterman | 22 May 2012 16:49

dh_python2 and /usr/share/pyshared in quantal

In https://launchpad.net/ubuntu/+source/python-defaults/2.7.3-0ubuntu3 Ubuntu 
modified dh_python2 to drop creation of /usr/share/pyshared and creation of 
python version specific symlinks to /usr/lib/python2.7/dist-packages/.

I believe this change should be reverted, but rather than just upload, wanted 
to discuss it first.  See https://bugs.launchpad.net/ubuntu/+source/python-
defaults/+bug/1001912 for additional discussion.

I've checked with Piotr Ożarowski (POX), the upstream developer for dh_python2 
and he does not support removing this feature of dh_python2 until after 
pysupport and pycentral are removed.  Even with a single supported python 
version (as Ubuntu has now) it's still useful because pysupport installs files 
in the same location.  It avoids some namespace issues.

This change breaks dozens of packages and has negligible (if any) advantages.

Additionally, the Python policy lists /usr/share/pyshared as the correct 
location to install version independent python files, so removing it moves away 
from the documented policy.

I'd like to understand if there's some compelling reason to make this change 
for quantal.  If not, it should be reverted sooner rather than later as 
packages built with this version are being misbuilt.

Scott K

--

-- 
ubuntu-devel mailing list
ubuntu-devel <at> lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
(Continue reading)

Permalink | Reply |
headers
Barry Warsaw | 22 May 2012 17:33
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On May 22, 2012, at 10:49 AM, Scott Kitterman wrote:

>In https://launchpad.net/ubuntu/+source/python-defaults/2.7.3-0ubuntu3 Ubuntu 
>modified dh_python2 to drop creation of /usr/share/pyshared and creation of 
>python version specific symlinks to /usr/lib/python2.7/dist-packages/.
>
>I believe this change should be reverted, but rather than just upload, wanted 
>to discuss it first.  See https://bugs.launchpad.net/ubuntu/+source/python-
>defaults/+bug/1001912 for additional discussion.
>
>I've checked with Piotr Ożarowski (POX), the upstream developer for
>dh_python2 and he does not support removing this feature of dh_python2 until
>after pysupport and pycentral are removed.  Even with a single supported
>python version (as Ubuntu has now) it's still useful because pysupport
>installs files in the same location.  It avoids some namespace issues.
>
>This change breaks dozens of packages and has negligible (if any) advantages.
>
>Additionally, the Python policy lists /usr/share/pyshared as the correct
>location to install version independent python files, so removing it moves
>away from the documented policy.
>
>I'd like to understand if there's some compelling reason to make this change
>for quantal.  If not, it should be reverted sooner rather than later as
>packages built with this version are being misbuilt.

According to this message in the bug:

https://bugs.launchpad.net/ubuntu/+source/python-defaults/+bug/1001912/comments/2
(Continue reading)

Permalink | Reply |
headers
Scott Kitterman | 22 May 2012 17:42

Re: dh_python2 and /usr/share/pyshared in quantal

On Tuesday, May 22, 2012 11:33:18 AM Barry Warsaw wrote:
> For example, I took a look at authres which uses dh_python2.  debian/rules
> creates a symlink in pyshared to its test suite, and then
> d/python-authres.install references pyshared.  Even after reading the
> changelog entry, it's not clear to me why this particular solution was
> chosen. Fortunately the maintainer is here and can probably answer that
> <wink>.

In that case it's because I had to install an extra, non-code, file for the 
test suite that dh_python2 didn't recognize should go there.  Arguably that's 
something I should be able to accomplish by configuring dh_python2, but it's a 
rare enough situation installing directly to the correct location makes sense.

I've got another case (pyspf) where I needed a symlink that would work 
regardless of if the default Python version was used.

/usr/share/pyshared is the correct location for such things.  That's what 
Python policy says.  pyshared is not an internal implementation detail of 
dh_python2.  It's where Python policy says to find such things.

I think it's an open question about we want to remove the general ability in 
the Debian Python system to support multiple Python versions.  It's a much 
broader question than just /usr/share/pyshared.  I don't think that there is 
any reason for Ubuntu to try and get ahead of Debian for this.

Scott K
Permalink | Reply |
headers
Barry Warsaw | 22 May 2012 20:49
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

Just to be clear, I was mostly trying to reverse engineer the rationale for
the change, not defend it.  I agree with the decision to revert it.

On May 22, 2012, at 11:42 AM, Scott Kitterman wrote:

>On Tuesday, May 22, 2012 11:33:18 AM Barry Warsaw wrote:
>> For example, I took a look at authres which uses dh_python2.  debian/rules
>> creates a symlink in pyshared to its test suite, and then
>> d/python-authres.install references pyshared.  Even after reading the
>> changelog entry, it's not clear to me why this particular solution was
>> chosen. Fortunately the maintainer is here and can probably answer that
>> <wink>.
>
>In that case it's because I had to install an extra, non-code, file for the
>test suite that dh_python2 didn't recognize should go there.  Arguably that's
>something I should be able to accomplish by configuring dh_python2, but it's
>a rare enough situation installing directly to the correct location makes
>sense.
>
>I've got another case (pyspf) where I needed a symlink that would work
>regardless of if the default Python version was used.

Thanks for the explanation.

>/usr/share/pyshared is the correct location for such things.  That's what
>Python policy says.  pyshared is not an internal implementation detail of
>dh_python2.  It's where Python policy says to find such things.

Actually, I read python-policy as being a bit ambiguous here.  To me, it
clearly says pyshared is the location for source code common for multiple
(Continue reading)

Permalink | Reply |
headers
Scott Kitterman | 22 May 2012 21:21

Re: dh_python2 and /usr/share/pyshared in quantal

On Tuesday, May 22, 2012 02:49:53 PM Barry Warsaw wrote:
> >/usr/share/pyshared is the correct location for such things.  That's what
> >Python policy says.  pyshared is not an internal implementation detail of
> >dh_python2.  It's where Python policy says to find such things.
> 
> Actually, I read python-policy as being a bit ambiguous here.  To me, it
> clearly says pyshared is the location for source code common for multiple
> Python (2) versions, but I think it's less than clear that this also applies
> to non-code common files.
> 
> The other two hits for 'pyshared' in python-policy.txt are in the deprecated
> python-support and python-central sections, so might be misconstrued as
> being implementation details of those helpers respectively (there's no
> mention of it in the dh_python2/3 section).

My hope (as one of the Python policy authors) is to eventually divorce the 
policy from any implementation specifics.  To the extent implementation specific 
details of pysupport or pycentral are mentioned, I consider them historical 
warts to be gotten rid of as soon as we can.

Scott K

P.S. re the other message - understand about the reverse engineering.
Permalink | Reply |
headers
Barry Warsaw | 22 May 2012 21:32
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On May 22, 2012, at 03:21 PM, Scott Kitterman wrote:

>My hope (as one of the Python policy authors) is to eventually divorce the
>policy from any implementation specifics.  To the extent implementation
>specific details of pysupport or pycentral are mentioned, I consider them
>historical warts to be gotten rid of as soon as we can.

+1

-Barry
Permalink | Reply |
headers
Matthias Klose | 23 May 2012 03:49
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On 22.05.2012 23:42, Scott Kitterman wrote:
> On Tuesday, May 22, 2012 11:33:18 AM Barry Warsaw wrote:
>> For example, I took a look at authres which uses dh_python2.  debian/rules
>> creates a symlink in pyshared to its test suite, and then
>> d/python-authres.install references pyshared.  Even after reading the
>> changelog entry, it's not clear to me why this particular solution was
>> chosen. Fortunately the maintainer is here and can probably answer that
>> <wink>.
> 
> In that case it's because I had to install an extra, non-code, file for the 
> test suite that dh_python2 didn't recognize should go there.  Arguably that's 
> something I should be able to accomplish by configuring dh_python2, but it's a 
> rare enough situation installing directly to the correct location makes sense.
> 
> I've got another case (pyspf) where I needed a symlink that would work 
> regardless of if the default Python version was used.
> 
> /usr/share/pyshared is the correct location for such things.  That's what 
> Python policy says.  pyshared is not an internal implementation detail of 
> dh_python2.  It's where Python policy says to find such things.

that is plain wrong. pyshared *is* an implementation detail for the working of
the package. Every package having pyshared on sys.path for testing, building,
installation is buggy.

So i'll modify the interpreter do disallow imports from pyshared to catch
exactly these misuses.

> I think it's an open question about we want to remove the general ability in 
> the Debian Python system to support multiple Python versions.
(Continue reading)

Permalink | Reply |
headers
Scott Kitterman | 23 May 2012 04:10

Re: dh_python2 and /usr/share/pyshared in quantal

On Wednesday, May 23, 2012 09:49:07 AM Matthias Klose wrote:
> PS: Scott, if you choose to move a discussion from a bug report to a ML,
> please mention this in the bug report next time.

Next time you decide to gratuitously break things, please have some discussion 
of it first.

Scott K
Permalink | Reply |
headers
Barry Warsaw | 23 May 2012 04:57
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On May 23, 2012, at 09:49 AM, Matthias Klose wrote:

>On 22.05.2012 23:42, Scott Kitterman wrote:
>> /usr/share/pyshared is the correct location for such things.  That's what 
>> Python policy says.  pyshared is not an internal implementation detail of 
>> dh_python2.  It's where Python policy says to find such things.
>
>that is plain wrong. pyshared *is* an implementation detail for the working of
>the package. Every package having pyshared on sys.path for testing, building,
>installation is buggy.
>
>So i'll modify the interpreter do disallow imports from pyshared to catch
>exactly these misuses.

Before you do this, I think some groundwork needs to be laid.  First, you need
to get general consensus that eradication of pyshared is a goal to be
achieved.  You don't need (and won't get) 100% consensus, but I think you need
the majority of developers to agree that this is a goal, and to commit to
helping make this happen.  The debian-python mailing list is the right place
to have this discussion.  Without consensus, bugs will just get Won't Fixed
and/or Ubuntu will just carry even more deltas from Debian, which I think we
all agree is not good.

(Take the dh_python2 transition as a model.  We didn't get - and still don't
have - 100% agreement about it, but we got a majority of developers to agree
that it was the right thing to do, and with official deprecation of
python-central and python-support, it became the defacto standard helper for
Python 2.  These facts can be used in bug reports, patches, wiki pages, and
other means of communication to rally developers around the transition.
Ubuntu can still lead, but it makes it easier to push patches back to Debian,
(Continue reading)

Permalink | Reply |
headers
Scott Kitterman | 22 May 2012 17:50

Re: dh_python2 and /usr/share/pyshared in quantal

On Tuesday, May 22, 2012 11:33:18 AM Barry Warsaw wrote:
> I'm sympathetic to Matthias's goal to stop using pyshared, and I think his
> intent was to use this change to gather information rather than use the
> breakage to force the package maintainers to change their packages (or for
> Ubuntu to carry deltas in all the packages).  Julian's list seems like a
> good first cut on that information gathering:
> 
> https://launchpadlibrarian.net/105736622/pyshared
> 
> So it probably makes the most sense at this point to file bugs against each
> of these packages in Ubuntu and Debian (assuming we're not carrying a
> dh_python2 delta, which would be a separate report in itself), and then
> revert the change to unbreak the packages.

Almost forgot ...

It seems to me that making changes that are known to break other packages to 
gather data is not consistent with the stated goal of keeping the development 
release in a functional state at ~all times.

I disagree that direct references to pyshared are a bug of any kind.

Also, Julian's list was limited to dpmt/papt packages which is a subset of 
packages that are affected by this.  IIRC, there are ~400 packages in those 
teams, so it hits ~10% of dh_python2 packages based on that sample.  That's a 
lot.

Scott K
(Continue reading)

Permalink | Reply |
headers
Steve Langasek | 22 May 2012 18:22
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On Tue, May 22, 2012 at 11:50:51AM -0400, Scott Kitterman wrote:
> Almost forgot ...

> It seems to me that making changes that are known to break other packages to 
> gather data is not consistent with the stated goal of keeping the development 
> release in a functional state at ~all times.

> I disagree that direct references to pyshared are a bug of any kind.

> Also, Julian's list was limited to dpmt/papt packages which is a subset of
> packages that are affected by this.  IIRC, there are ~400 packages in
> those teams, so it hits ~10% of dh_python2 packages based on that sample. 
> That's a lot.

Between this and the observation that it's resulting in misbuilt packages,
I think the right thing to do is revert the change for now while the
discussion is ongoing.  If in the end there's a solution that lets us gather
information in the archive without causing package misbuilds, we can
consider re-introducing that; or we can do test rebuilds in a ppa for this
information.  But we should not be misbuilding packages in the main archive
purely as an information-gathering exercise.

--

-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
Ubuntu Developer                                    http://www.debian.org/
slangasek <at> ubuntu.com                                     vorlon <at> debian.org
Permalink | Reply |
headers
Scott Kitterman | 22 May 2012 18:32

Re: dh_python2 and /usr/share/pyshared in quantal

On Tuesday, May 22, 2012 09:22:09 AM Steve Langasek wrote:
> On Tue, May 22, 2012 at 11:50:51AM -0400, Scott Kitterman wrote:
> > Almost forgot ...
> > 
> > It seems to me that making changes that are known to break other packages
> > to gather data is not consistent with the stated goal of keeping the
> > development release in a functional state at ~all times.
> > 
> > I disagree that direct references to pyshared are a bug of any kind.
> > 
> > Also, Julian's list was limited to dpmt/papt packages which is a subset of
> > packages that are affected by this.  IIRC, there are ~400 packages in
> > those teams, so it hits ~10% of dh_python2 packages based on that sample.
> > That's a lot.
> 
> Between this and the observation that it's resulting in misbuilt packages,
> I think the right thing to do is revert the change for now while the
> discussion is ongoing.  If in the end there's a solution that lets us gather
> information in the archive without causing package misbuilds, we can
> consider re-introducing that; or we can do test rebuilds in a ppa for this
> information.  But we should not be misbuilding packages in the main archive
> purely as an information-gathering exercise.

Thanks.  Done.

I do not have time to go back and look at what packages are affected by this, 
but someone should.

Scott K
(Continue reading)

Permalink | Reply |
headers
Barry Warsaw | 22 May 2012 20:50
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On May 22, 2012, at 11:50 AM, Scott Kitterman wrote:

>It seems to me that making changes that are known to break other packages to
>gather data is not consistent with the stated goal of keeping the development
>release in a functional state at ~all times.

This is a really excellent point.

>I disagree that direct references to pyshared are a bug of any kind.

Fair enough.

-Barry
Permalink | Reply |
headers
Matthias Klose | 23 May 2012 03:54
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On 23.05.2012 02:50, Barry Warsaw wrote:
> On May 22, 2012, at 11:50 AM, Scott Kitterman wrote:
> 
>> It seems to me that making changes that are known to break other packages to
>> gather data is not consistent with the stated goal of keeping the development
>> release in a functional state at ~all times.
> 
> This is a really excellent point.

no, not that excellent. For larger changes we do these changes in the archive,
like the glib header changes during the late precise cycle, which can but have
not to be reverted later.

>> I disagree that direct references to pyshared are a bug of any kind.
> 
> Fair enough.

again, any use of pyshared in sys.path is a bug.

  Matthias
Permalink | Reply |
headers
Scott Kitterman | 23 May 2012 04:12

Re: dh_python2 and /usr/share/pyshared in quantal

On Wednesday, May 23, 2012 09:54:07 AM Matthias Klose wrote:
> >> I disagree that direct references to pyshared are a bug of any kind.
> >
> > 
> >
> > Fair enough.
> 
> again, any use of pyshared in sys.path is a bug.

Of the four packages on the list that I've touched, none of them do that.  In 
one of them, python-qt4, debian/rules only mentions pyshared to remove empty 
directories left there.  It's an artifact of the way the build system works, 
not a bug related to sys.path.

Scott K
Permalink | Reply |
headers
Sebastien Bacher | 22 May 2012 19:22
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

Le 22/05/2012 17:33, Barry Warsaw a écrit :
> Matthias made the change to try to catch unnecessary (invalid?) uses of
> pyshared, which might a worthy goal
Hey,

Do we really need to break the archive version to figure those out, 
can't we do an archive rebuild of some way using a custom or ppa build?

Sebastien Bacher

--

-- 
ubuntu-devel mailing list
ubuntu-devel <at> lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Permalink | Reply |
headers
Micah Gersten | 23 May 2012 04:02
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On 05/22/2012 12:22 PM, Sebastien Bacher wrote:
> Le 22/05/2012 17:33, Barry Warsaw a écrit :
>> Matthias made the change to try to catch unnecessary (invalid?) uses of
>> pyshared, which might a worthy goal
> Hey,
>
> Do we really need to break the archive version to figure those out,
> can't we do an archive rebuild of some way using a custom or ppa build?
>
> Sebastien Bacher
>
An i386 rebuild should be sufficient for this.  As the builders have
been mostly idle and a rebuild on i386 should be done before alpha1
week, maybe now is a good time?

--

-- 
ubuntu-devel mailing list
ubuntu-devel <at> lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
Permalink | Reply |
headers
Stefano Rivera | 23 May 2012 04:07
Favicon
Gravatar

Re: dh_python2 and /usr/share/pyshared in quantal

Hi Micah (2012.05.23_04:02:36_+0200)
> > Do we really need to break the archive version to figure those out,
> > can't we do an archive rebuild of some way using a custom or ppa build?
>
> An i386 rebuild should be sufficient for this.

I'd have thought a grep through the lintian lab would be sufficient?

SR

--

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  H: +27 21 461 1230 C: +27 72 419 8559
Permalink | Reply |
headers
Micah Gersten | 23 May 2012 04:14
Favicon

Re: dh_python2 and /usr/share/pyshared in quantal

On 05/22/2012 09:07 PM, Stefano Rivera wrote:
>>> Do we really need to break the archive version to figure those out,
>>> can't we do an archive rebuild of some way using a custom or ppa build?
>> An i386 rebuild should be sufficient for this.
> I'd have thought a grep through the lintian lab would be sufficient?
That would probably work too :)

Micah
Permalink | Reply |

Gmane