gmane.linux.gentoo.devel

Re: RFC: auto-detection of unpack dependencies

Marius Mauch wrote:
> As a result of Cardoes earlier mail we talked a bit about possible
> solutions in #gento-portage, and I suggested to let portage
> automatically inject the deps based on SRC_URI pattern matching.
> A mapping of extensions and their unpack deps would be kept in the tree
> (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'
>
> Benefits:
> - simplified depstrings for most packages (I assume >90% of the tree)
> - easier maintenance if dependencies are changing, or additional
> implementations become supported (this could also be achieved with
> virtuals though)
> - potentially more accurate dependencies, as some common errors would
> be eliminated (e.g. proper treatment of use-conditionals, not adding
> unpack deps to RDEPEND)
> - long-term adds the possibility to remove bzip2, gzip and tar from
> @system
>
> Potential problems:
> - might cause trouble for some packages that use custom code for
> unpacking, or due to circular deps, this could simply be solved with a
> new RESTRICT value though.
> - automagic deps could be confusing to devs/users
> - not available for existing EAPIs (due to the mentioned problems)
> - slightly slower dep calculations due to the extra processing
> - possible match errors
>
> So, is this something ebuild maintainers would like in general, or does
> such a feature cause you nightmares?
>

(Continue reading)

headers

Petteri Räty | 15 Jul 18:14

Re: RFC: auto-detection of unpack dependencies

Marius Mauch kirjoitti:
> 
> So, is this something ebuild maintainers would like in general, or does
> such a feature cause you nightmares?
> 

I have actually been thinking about writing support for this into the 
Java eclasses because most Java upstreams use .zip files and we need 
app-arch/unzip in DEPEND for that.

Regards,
Petteri

headers

Tiziano Müller | 15 Jul 21:40

Re: RFC: auto-detection of unpack dependencies

Marius Mauch wrote:

> As a result of Cardoes earlier mail we talked a bit about possible
> solutions in #gento-portage, and I suggested to let portage
> automatically inject the deps based on SRC_URI pattern matching.
> A mapping of extensions and their unpack deps would be kept in the tree
> (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'
> 
> Benefits:
> - simplified depstrings for most packages (I assume >90% of the tree)
> - easier maintenance if dependencies are changing, or additional
> implementations become supported (this could also be achieved with
> virtuals though)
> - potentially more accurate dependencies, as some common errors would
> be eliminated (e.g. proper treatment of use-conditionals, not adding
> unpack deps to RDEPEND)
> - long-term adds the possibility to remove bzip2, gzip and tar from
> @system
> 
> Potential problems:
> - might cause trouble for some packages that use custom code for
> unpacking, or due to circular deps, this could simply be solved with a
> new RESTRICT value though.
> - automagic deps could be confusing to devs/users
> - not available for existing EAPIs (due to the mentioned problems)
> - slightly slower dep calculations due to the extra processing
> - possible match errors
> 
> So, is this something ebuild maintainers would like in general, or does
> such a feature cause you nightmares?

(Continue reading)

headers

Patrick Börjesson | 15 Jul 20:10

Re: Re: RFC: auto-detection of unpack dependencies

On 2008-07-15 21:40, Tiziano Müller uttered these thoughts:
> Marius Mauch wrote:
> 
> > As a result of Cardoes earlier mail we talked a bit about possible
> > solutions in #gento-portage, and I suggested to let portage
> > automatically inject the deps based on SRC_URI pattern matching.
> > A mapping of extensions and their unpack deps would be kept in the tree
> > (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'
[snip]
> > So, is this something ebuild maintainers would like in general, or does
> > such a feature cause you nightmares?
> 
> Yes. I think that's something which should be done manually.

Indeed, the correct solution would be to state the deps manually in each
ebuild that requires the dep. But in this case it would mean adjusting
the DEPEND string of pretty much the entire tree. Until such measures
are stated required, this would be a good middle ground, no?

The same thing would apply to gcc if all "real" depends were to be
required in all ebuilds, but that would pretty much have to be manually
stated since the PM wouldn't be able to judge that by automatic
measures. This, on the other hand, can (at least partially) be handled
automatically for the ebuild-devs on the PM side of things. 

Patrick B

--

-- 
()  The ASCII Ribbon Campaign - against HTML Email
/\  and proprietary formats.

(Continue reading)

headers

Donnie Berkholz | 15 Jul 21:23

Re: Re: RFC: auto-detection of unpack dependencies

\On 20:10 Tue 15 Jul     , Patrick Börjesson wrote:
> On 2008-07-15 21:40, Tiziano Müller uttered these thoughts:
> > Yes. I think that's something which should be done manually.
> 
> Indeed, the correct solution would be to state the deps manually in each
> ebuild that requires the dep. But in this case it would mean adjusting
> the DEPEND string of pretty much the entire tree. Until such measures
> are stated required, this would be a good middle ground, no?

Could someone explain why manually doing work is better than 
automatically detecting the deps? This sounds like an argument against 
automation, and I'm not following it.

--

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

headers

Ciaran McCreesh | 15 Jul 21:25

Re: Re: RFC: auto-detection of unpack dependencies

On Tue, 15 Jul 2008 12:23:08 -0700
Donnie Berkholz <dberkholz <at> gentoo.org> wrote:
> Could someone explain why manually doing work is better than 
> automatically detecting the deps? This sounds like an argument
> against automation, and I'm not following it.

Sometimes the magic will be wrong. For example, packages that have
a .zip file in SRC_URI but that don't unpack that file (say, if they
install it into sharedir as-is instead) don't need a dep upon unzip.

--

-- 
Ciaran McCreesh

headers

Donnie Berkholz | 15 Jul 22:06

Re: Re: RFC: auto-detection of unpack dependencies

On 20:25 Tue 15 Jul     , Ciaran McCreesh wrote:
> On Tue, 15 Jul 2008 12:23:08 -0700
> Donnie Berkholz <dberkholz <at> gentoo.org> wrote:
> > Could someone explain why manually doing work is better than 
> > automatically detecting the deps? This sounds like an argument
> > against automation, and I'm not following it.
> 
> Sometimes the magic will be wrong. For example, packages that have
> a .zip file in SRC_URI but that don't unpack that file (say, if they
> install it into sharedir as-is instead) don't need a dep upon unzip.

Indeed, that is a great argument for the RESTRICT. I wonder how many 
packages do this? To get a quick idea, I did this grep:

  grep '\.zip' /usr/portage/*/*/*ebuild \
    | grep -v -e http -e ftp -e mirror -e unpack -e SRC_URI -e unzip

It produced a reasonable number of results, and I didn't investigate 
further.

--

-- 
Thanks,
Donnie

Donnie Berkholz
Developer, Gentoo Linux
Blog: http://dberkholz.wordpress.com

headers

Richard Freeman | 15 Jul 22:45

Re: Re: RFC: auto-detection of unpack dependencies

Ciaran McCreesh wrote:
> On Tue, 15 Jul 2008 12:23:08 -0700
> Donnie Berkholz <dberkholz <at> gentoo.org> wrote:
>> Could someone explain why manually doing work is better than 
>> automatically detecting the deps? This sounds like an argument
>> against automation, and I'm not following it.
> 
> Sometimes the magic will be wrong. For example, packages that have
> a .zip file in SRC_URI but that don't unpack that file (say, if they
> install it into sharedir as-is instead) don't need a dep upon unzip.
> 

Couldn't the ebuild be wrong?  For example, if the package manager uses 
fancy-unzip-replacement to unzip packages, but the ebuild depends on 
unzip, then wouldn't it fail?  It seems like we're trying to have 
ebuilds DEPEND on something that in reality the package manager is doing.

Shouldn't the package manager depend on unzip instead?  If circular 
references are the problem, then wouldn't it be better to find a better 
way to handle circular references (ie more specific ways of defining 
dependencies)?

The advantage I see with automagic PM dependency calculations is that 
the PM is the program calling unzip, so the PM ought to be able to 
figure out whether it needs to call unzip.

Attachment (smime.p7s): application/x-pkcs7-signature, 3670 bytes

headers

Ciaran McCreesh | 15 Jul 22:53

Re: Re: RFC: auto-detection of unpack dependencies

On Tue, 15 Jul 2008 16:45:38 -0400
Richard Freeman <rich <at> thefreemanclan.net> wrote:
> Couldn't the ebuild be wrong?  For example, if the package manager
> uses fancy-unzip-replacement to unzip packages, but the ebuild
> depends on unzip, then wouldn't it fail?  It seems like we're trying
> to have ebuilds DEPEND on something that in reality the package
> manager is doing.

For current EAPIs, we pretty much mandate that 'unzipping .zip files is
done using app-arch/unzip'. It's not ideal.

For future EAPIs, we can have the package manager define
super-magic/thing-i-use-to-unzip.

> Shouldn't the package manager depend on unzip instead?  If circular 
> references are the problem, then wouldn't it be better to find a
> better way to handle circular references (ie more specific ways of
> defining dependencies)?

Having the package manager depend upon things like 7z for the two
packages in the tree that use it isn't sane.

One could argue that having package manager support for extracting 7z
is also not sane, of course, but that's something we're stuck with in
current EAPIs.

--

-- 
Ciaran McCreesh

headers

Richard Freeman | 16 Jul 00:00

Re: Re: RFC: auto-detection of unpack dependencies

Ciaran McCreesh wrote:
> For future EAPIs, we can have the package manager define
> super-magic/thing-i-use-to-unzip.

++

> 
> One could argue that having package manager support for extracting 7z
> is also not sane, of course, but that's something we're stuck with in
> current EAPIs.
> 

Or we could allow users to decide for themselves with IUSE="7z" in the 
package manager ebuild.  Granted, it would be a bit of a pain predicting 
what formats your package manager will need to handle.

Attachment (smime.p7s): application/x-pkcs7-signature, 3670 bytes

headers

Richard Freeman | 16 Jul 00:00

Re: Re: RFC: auto-detection of unpack dependencies

Ciaran McCreesh wrote:
> For future EAPIs, we can have the package manager define
> super-magic/thing-i-use-to-unzip.

++

> 
> One could argue that having package manager support for extracting 7z
> is also not sane, of course, but that's something we're stuck with in
> current EAPIs.
> 

Or we could allow users to decide for themselves with IUSE="7z" in the
package manager ebuild.  Granted, it would be a bit of a pain predicting
what formats your package manager will need to handle.
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Tiziano Müller | 15 Jul 22:58

Re: Re: RFC: auto-detection of unpack dependencies

Patrick Börjesson wrote:

> On 2008-07-15 21:40, Tiziano Müller uttered these thoughts:
>> Marius Mauch wrote:
>> 
>> > As a result of Cardoes earlier mail we talked a bit about possible
>> > solutions in #gento-portage, and I suggested to let portage
>> > automatically inject the deps based on SRC_URI pattern matching.
>> > A mapping of extensions and their unpack deps would be kept in the tree
>> > (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'
> [snip]
>> > So, is this something ebuild maintainers would like in general, or does
>> > such a feature cause you nightmares?
>> 
>> Yes. I think that's something which should be done manually.
> 
> Indeed, the correct solution would be to state the deps manually in each
> ebuild that requires the dep. But in this case it would mean adjusting
> the DEPEND string of pretty much the entire tree. Until such measures
> are stated required, this would be a good middle ground, no?
no. How about just introducing the new deps on their next version or
revision bump? (I assume that more than half of the packages would be fixed
within the next half year and that's more than fast enough).

> 
> The same thing would apply to gcc if all "real" depends were to be
> required in all ebuilds, but that would pretty much have to be manually
> stated since the PM wouldn't be able to judge that by automatic
> measures. This, on the other hand, can (at least partially) be handled
> automatically for the ebuild-devs on the PM side of things.

(Continue reading)

headers

Patrick Börjesson | 16 Jul 02:32

Re: Re: Re: RFC: auto-detection of unpack dependencies

On 2008-07-15 22:58, Tiziano Müller uttered these thoughts:
> Patrick Börjesson wrote:
> > On 2008-07-15 21:40, Tiziano Müller uttered these thoughts:
> > The same thing would apply to gcc if all "real" depends were to be
> > required in all ebuilds, but that would pretty much have to be manually
> > stated since the PM wouldn't be able to judge that by automatic
> > measures. 
> That's a different thing:
> A dependency on gcc just ensures that gcc is installed not that it is
> actually used to build a package.

Not quite sure what you mean here. I'm just saying that if you want to
go the route of stating all deps explicitly, you have to state in the
ebuild (DEPENDS) that gcc is needed to build the package, if that's the
case. I'm not against this at all (I'm not an ebuild-maintainer), i just
gave an example for when there's no sane way for the PM to automatically
inject a dependency. 

> And for such a dependency we'd need new ways to express deps since gcc is
> only needed when building packages not when it gets installed from a
> binpkg.

Portage (or whichever PM you want) uses it's own way of packaging
binpkgs, so for it to be able to extract those binpkgs, a RDEPEND on the
applications used for that specific task has to be stated in the _PM_
itself. It isn't the ebuild deciding which format it's gonna be
packaged down into. 
I'm far from sure about this, but DEPENDS aren't really taken into
consideration when installing from a binpkg, so stating (f.ex) gcc in
DEPENDS wouldn't draw it in when you install the package from a binpkg.

(Continue reading)

headers

Alec Warner | 16 Jul 05:33

Re: Re: Re: RFC: auto-detection of unpack dependencies

On Tue, Jul 15, 2008 at 1:58 PM, Tiziano Müller <dev-zero <at> gentoo.org> wrote:
> Patrick Börjesson wrote:
>
>> On 2008-07-15 21:40, Tiziano Müller uttered these thoughts:
>>> Marius Mauch wrote:
>>>
>>> > As a result of Cardoes earlier mail we talked a bit about possible
>>> > solutions in #gento-portage, and I suggested to let portage
>>> > automatically inject the deps based on SRC_URI pattern matching.
>>> > A mapping of extensions and their unpack deps would be kept in the tree
>>> > (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'
>> [snip]
>>> > So, is this something ebuild maintainers would like in general, or does
>>> > such a feature cause you nightmares?
>>>
>>> Yes. I think that's something which should be done manually.
>>
>> Indeed, the correct solution would be to state the deps manually in each
>> ebuild that requires the dep. But in this case it would mean adjusting
>> the DEPEND string of pretty much the entire tree. Until such measures
>> are stated required, this would be a good middle ground, no?
> no. How about just introducing the new deps on their next version or
> revision bump? (I assume that more than half of the packages would be fixed
> within the next half year and that's more than fast enough).

Why would you do a bunch of manual work when you can script it easily
with few false positives?
Doubly so when false positives are relatively cheap; adding an extra
dep on a package you probably
already have installed won't hurt much and those it does hurt can file

(Continue reading)

headers

Rémi Cardona | 15 Jul 19:53

Re: RFC: auto-detection of unpack dependencies

Marius Mauch wrote:
> Potential problems:
> - might cause trouble for some packages that use custom code for
> unpacking, or due to circular deps, this could simply be solved with a
> new RESTRICT value though.

As long as this is done to allow a 100% manual override, then this is a 
_very_ good idea.

Rémi
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Fabian Groffen | 15 Jul 20:00

Re: RFC: auto-detection of unpack dependencies

On 15-07-2008 19:53:47 +0200, Rémi Cardona wrote:
> Marius Mauch wrote:
>> Potential problems:
>> - might cause trouble for some packages that use custom code for
>> unpacking, or due to circular deps, this could simply be solved with a
>> new RESTRICT value though.
>
> As long as this is done to allow a 100% manual override, then this is a  
> _very_ good idea.

Manual override as in "emerge --nodeps" or something.  I have no
examples at hand, but during bootstrapping I have more than often a
problem with dependencies, which requires careful manual emerging of
packages (often ignoring deps) because those deps cannot be compiled
or emerged yet.

-- 
Fabian Groffen
Gentoo on a different level
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Rémi Cardona | 15 Jul 20:20

Re: RFC: auto-detection of unpack dependencies

Fabian Groffen wrote:
> Manual override as in "emerge --nodeps" or something.

No, manual override as in "the ebuild turns off auto-detection and 
specifically asks for app-arch/{bzip2,gzip,tar,whatever} using DEPEND"

Cheers,

Rémi
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Gokdeniz Karadag | 15 Jul 22:31

Re: RFC: auto-detection of unpack dependencies

If an ebuild lists bzip2 in DEPEND, the package manager has to bring it in.
The proposed method would only add automatically determined dependencies, not 
remove what you listed in DEPEND.

A hypothetical problem is; If a package source file has a bz extension but does 
not need bzip2 in any way, an extra dependency is in effect. But can this 
situation ever happen ? I guess not.

Rémi Cardona demis ki::
> Fabian Groffen wrote:
>> Manual override as in "emerge --nodeps" or something.
> 
> No, manual override as in "the ebuild turns off auto-detection and 
> specifically asks for app-arch/{bzip2,gzip,tar,whatever} using DEPEND"
> 
> Cheers,
> 
> Rémi

-- 
Gokdeniz Karadag

--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Ciaran McCreesh | 15 Jul 20:12

Re: RFC: auto-detection of unpack dependencies

On Tue, 15 Jul 2008 04:14:18 +0200
Marius Mauch <genone <at> gentoo.org> wrote:
> As a result of Cardoes earlier mail we talked a bit about possible
> solutions in #gento-portage, and I suggested to let portage
> automatically inject the deps based on SRC_URI pattern matching.
> A mapping of extensions and their unpack deps would be kept in the
> tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'

Could do it just as an eclass...

inherit work-out-my-unpack-deps-for-me

In principle, there's nothing that can't be done on the ebuild side
here.

For the future EAPI side, you could require package managers to define
super-magic/package-manager-unpack-bzip2 etc for whatever they use to do
the unpacking, and to make it work with existing EAPIs, use || deps
with what existing package managers happen to use as the second item.
For current EAPIs it's no worse than the existing situation, where
ebuilds have to guess that package managers use 'app-arch/unzip' to
unpack zip files.

--

-- 
Ciaran McCreesh

headers

Doug Goldstein | 15 Jul 20:28

Re: RFC: auto-detection of unpack dependencies

Ciaran McCreesh wrote:
> On Tue, 15 Jul 2008 04:14:18 +0200
> Marius Mauch <genone <at> gentoo.org> wrote:
>   
>> As a result of Cardoes earlier mail we talked a bit about possible
>> solutions in #gento-portage, and I suggested to let portage
>> automatically inject the deps based on SRC_URI pattern matching.
>> A mapping of extensions and their unpack deps would be kept in the
>> tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'
>>     
>
> Could do it just as an eclass...
>
> inherit work-out-my-unpack-deps-for-me
>
> In principle, there's nothing that can't be done on the ebuild side
> here.
>
> For the future EAPI side, you could require package managers to define
> super-magic/package-manager-unpack-bzip2 etc for whatever they use to do
> the unpacking, and to make it work with existing EAPIs, use || deps
> with what existing package managers happen to use as the second item.
> For current EAPIs it's no worse than the existing situation, where
> ebuilds have to guess that package managers use 'app-arch/unzip' to
> unpack zip files.
>
>   
This is pretty close to what I had brought up in #gentoo-portage 
yesterday as well and seems like the best step forward.
--

--

(Continue reading)

headers

Petteri Räty | 15 Jul 21:15

Re: RFC: auto-detection of unpack dependencies

Ciaran McCreesh kirjoitti:
> On Tue, 15 Jul 2008 04:14:18 +0200
> Marius Mauch <genone <at> gentoo.org> wrote:
>> As a result of Cardoes earlier mail we talked a bit about possible
>> solutions in #gento-portage, and I suggested to let portage
>> automatically inject the deps based on SRC_URI pattern matching.
>> A mapping of extensions and their unpack deps would be kept in the
>> tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'
> 
> Could do it just as an eclass...
> 
> inherit work-out-my-unpack-deps-for-me
> 
> In principle, there's nothing that can't be done on the ebuild side
> here.
> 

Wouldn't this also require having a variable like SRC_URI or 
AUTODEP_SRC_URI above inherit?

Regards,
Petteri

headers

Ciaran McCreesh | 15 Jul 21:19

Re: RFC: auto-detection of unpack dependencies

On Tue, 15 Jul 2008 22:15:16 +0300
Petteri Räty <betelgeuse <at> gentoo.org> wrote:
> > Could do it just as an eclass...
> > 
> > inherit work-out-my-unpack-deps-for-me
> > 
> > In principle, there's nothing that can't be done on the ebuild side
> > here.
> 
> Wouldn't this also require having a variable like SRC_URI or 
> AUTODEP_SRC_URI above inherit?

Yep. Although, you could do something like:

inherit normal-eclass-stuff

SRC_URI="blah"

inherit fancy-extra-eclass-stuff

--

-- 
Ciaran McCreesh

headers

Marius Mauch | 15 Jul 23:23

Re: RFC: auto-detection of unpack dependencies

On Tue, 15 Jul 2008 19:12:37 +0100
Ciaran McCreesh <ciaran.mccreesh <at> googlemail.com> wrote:

> On Tue, 15 Jul 2008 04:14:18 +0200
> Marius Mauch <genone <at> gentoo.org> wrote:
> > As a result of Cardoes earlier mail we talked a bit about possible
> > solutions in #gento-portage, and I suggested to let portage
> > automatically inject the deps based on SRC_URI pattern matching.
> > A mapping of extensions and their unpack deps would be kept in the
> > tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'
> 
> Could do it just as an eclass...
> 
> inherit work-out-my-unpack-deps-for-me
> 
> In principle, there's nothing that can't be done on the ebuild side
> here.

Right, just I'd expect the parsing of SRC_URI (with conditionals) to be
a bit tricky in bash, not something I'm going to work on. An
eclass-based solution would have a few benefits though wrt the metadata
cache.

Marius
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Ciaran McCreesh | 15 Jul 23:34

Re: RFC: auto-detection of unpack dependencies

On Tue, 15 Jul 2008 23:23:26 +0200
Marius Mauch <genone <at> gentoo.org> wrote:
> Right, just I'd expect the parsing of SRC_URI (with conditionals) to
> be a bit tricky in bash, not something I'm going to work on. An
> eclass-based solution would have a few benefits though wrt the
> metadata cache.

Well... You don't really have to parse it.

for p in $SRC_URI ; do
    if [[ "${p}" == "(" ]] || [[ "${p}" == ")" ]] || \
        [[ "${p%\?}" != "${p}" ]] ; then
        UNPACK_DEPENDS="${UNPACK_DEPENDS} $p "
    elif [[ "${p%.zip}" != "${p}" ]] ; then
        UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/unzip "
    elif [[ "${p%.bz2}" != "${p}" ]] ; then
        UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/bzip2 "
    fi
done

Granted, it'll generate invalid output if SRC_URI is invalid (for
example, if SRC_URI has mismatched parens, the output will too), but I
can't think of any situation where breaking DEPEND if SRC_URI is
already broken is a problem.

--

-- 
Ciaran McCreesh

headers

Marius Mauch | 16 Jul 00:43

Re: RFC: auto-detection of unpack dependencies

On Tue, 15 Jul 2008 22:34:33 +0100
Ciaran McCreesh <ciaran.mccreesh <at> googlemail.com> wrote:

> On Tue, 15 Jul 2008 23:23:26 +0200
> Marius Mauch <genone <at> gentoo.org> wrote:
> > Right, just I'd expect the parsing of SRC_URI (with conditionals) to
> > be a bit tricky in bash, not something I'm going to work on. An
> > eclass-based solution would have a few benefits though wrt the
> > metadata cache.
> 
> Well... You don't really have to parse it.
> 
> for p in $SRC_URI ; do
>     if [[ "${p}" == "(" ]] || [[ "${p}" == ")" ]] || \
>         [[ "${p%\?}" != "${p}" ]] ; then
>         UNPACK_DEPENDS="${UNPACK_DEPENDS} $p "
>     elif [[ "${p%.zip}" != "${p}" ]] ; then
>         UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/unzip "
>     elif [[ "${p%.bz2}" != "${p}" ]] ; then
>         UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/bzip2 "
>     fi
> done
> 
> Granted, it'll generate invalid output if SRC_URI is invalid (for
> example, if SRC_URI has mismatched parens, the output will too), but I
> can't think of any situation where breaking DEPEND if SRC_URI is
> already broken is a problem.

Interesting idea. Unfortunately our depstring parser doesn't like empty
parentheses (as they are usually problem indicators), so it doesn't

(Continue reading)

headers

Ciaran McCreesh | 16 Jul 01:02

Re: RFC: auto-detection of unpack dependencies

On Wed, 16 Jul 2008 00:43:28 +0200
Marius Mauch <genone <at> gentoo.org> wrote:
> Interesting idea. Unfortunately our depstring parser doesn't like
> empty parentheses (as they are usually problem indicators), so it
> doesn't work out.

Bleh. You can hack around that using a second (looping) pass then. I
think this is right, assuming extglob:

for p in $SRC_URI ; do
    if [[ "${p}" == "(" ]] || [[ "${p}" == ")" ]] || \
        [[ "${p%\?}" != "${p}" ]] ; then
        UNPACK_DEPENDS="${UNPACK_DEPENDS} $p "
    elif [[ "${p%.zip}" != "${p}" ]] ; then
        UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/unzip "
    elif [[ "${p%.bz2}" != "${p}" ]] ; then
        UNPACK_DEPENDS="${UNPACK_DEPENDS} app-arch/bzip2 "
    fi
done

while true ; do
    UNPACK_DEPENDS=" ${UNPACK_DEPENDS} "
    u="${UNPACK_DEPENDS}"
    UNPACK_DEPENDS="${UNPACK_DEPENDS//?(+( )+([^ ])\?)+( )(+( ))+( )/ }"
    [[ "$u" == "${UNPACK_DEPENDS}" ]] && break
done

Although... Allowing ( ) makes much more sense...

--

--

(Continue reading)

headers

Marius Mauch | 17 Jul 04:51

Re: RFC: auto-detection of unpack dependencies

On Tue, 15 Jul 2008 04:14:18 +0200
Marius Mauch <genone <at> gentoo.org> wrote:

> As a result of Cardoes earlier mail we talked a bit about possible
> solutions in #gento-portage, and I suggested to let portage
> automatically inject the deps based on SRC_URI pattern matching.
> A mapping of extensions and their unpack deps would be kept in the
> tree (e.g. mapping '.tar.bz2' to '( app-arch/tar app-arch/bzip2 )'

At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass
to implement this feature. It doesn't modify DEPEND by itself, ebuild
have to call unpack_update_depend on their own for that, as automatic
updates are problematic when you have to take indirect and multiple
inheritance of eclasses into account. As automatic updates aren't
implemented there is also no need for a RESTRICT setting or similar,
and also no requirements on the order of SRC_URI/RESTRICT/inherit.

The eclass also contains it's own implementation of unpack (renamed to
unpack2) and src_unpack so the logic which tools/packages are used for
unpacking can be maintained in a single place instead of being
splitted between the package manager and the tree.

If someone wants to use it I can add it on the tree (after the normal
review process and being better tested), but I'll only be doing it
when there is an actual demand for it (no point in adding an eclass that
nobody uses).

It's still possible to come up with a package-manager-based solution in
the future, but for the moment the eclass should help with the
immideate problem (though the problem isn't exactly new).

(Continue reading)

headers

Joe Peterson | 17 Jul 14:00

Re: RFC: auto-detection of unpack dependencies

Marius Mauch wrote:
> The eclass also contains it's own implementation of unpack (renamed to
> unpack2) and src_unpack so the logic which tools/packages are used for
> unpacking can be maintained in a single place instead of being
> splitted between the package manager and the tree.

Marius, these look like nice functions.  A couple of questions:

1) Since it requires altering an ebuild to use these features (i.e. no
automatic), what is the advantage over just including the dep the usual way?

2) The name "unpack2" is intended to be temporary, right?  ;)  I'd guess that
after this functionality is integrated into portage, there would be no need
for the extra unpack func.  But then we'd probably have to keep "unpack2" for
a long time as an alias.  Any way to avoid that?

3) Same would go for the needed call for unpack_update_depend, correct?  I.e.
it would not be needed after portage has this functionality (and at that
point, the unpack and dep code would be unified cleanly).  So this function
would then become a stub, correct?  The only thing here is that ebuilds could
linger for some time without being "cleaned up".

						-Joe
--

-- 
gentoo-dev <at> lists.gentoo.org mailing list

headers

Marius Mauch | 17 Jul 19:06

Re: RFC: auto-detection of unpack dependencies

On Thu, 17 Jul 2008 07:00:32 -0500
Joe Peterson <lavajoe <at> gentoo.org> wrote:

> Marius Mauch wrote:
> > The eclass also contains it's own implementation of unpack (renamed
> > to unpack2) and src_unpack so the logic which tools/packages are
> > used for unpacking can be maintained in a single place instead of
> > being splitted between the package manager and the tree.
> 
> Marius, these look like nice functions.  A couple of questions:
> 
> 1) Since it requires altering an ebuild to use these features (i.e. no
> automatic), what is the advantage over just including the dep the
> usual way?

You wouldn't have to maintain the relevant deps, e.g. when you change
SRC_URI, when the unpack implementation changes (e.g. adding support
for new unpackers) or keeping the conditionals in sync with SRC_URI.

> 2) The name "unpack2" is intended to be temporary, right?  ;)  I'd
> guess that after this functionality is integrated into portage, there
> would be no need for the extra unpack func.  But then we'd probably
> have to keep "unpack2" for a long time as an alias.  Any way to avoid
> that?

Well, it can't be named unpack to avoid conflicts with the internal
portage function. If this functionality would be added on the PM side
then there would be no more need for the eclass (you wouldn't want to
use both implementations at the same time), and it would be opt-in via a
new EAPI, so you'd have to change the ebuilds anyway to benefit from

(Continue reading)

headers

Petteri Räty | 19 Jul 01:35

Re: RFC: auto-detection of unpack dependencies

Marius Mauch kirjoitti:
> 
> If someone wants to use it I can add it on the tree (after the normal
> review process and being better tested), but I'll only be doing it
> when there is an actual demand for it (no point in adding an eclass that
> nobody uses).
> 

I have been long thinking about adding support for .zip handling to java 
eclasses so let's get this in and it will get a user from java-pkg-2 and 
java-pkg-opt-2 eclasses at least.

Regards,
Petteri

headers

Peter Volkov | 20 Jul 15:41

Re: RFC: auto-detection of unpack dependencies

В Чтв, 17/07/2008 в 04:51 +0200, Marius Mauch пишет: 
> At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass
> to implement this feature. 

Marius, although it's possible to do this things in eclass why is eclass
better? As I see portage's unpack() already has extension <-> "program
to unpack" relation. Basically unpack() in ebuild.sh has the following
code:

unpack() {
[snip]
        case "${x##*.}" in
            tar)
                tar xof "${srcdir}${x}" ${tar_opts} || die "$myfail"
                ;;
            tgz)
                tar xozf "${srcdir}${x}" ${tar_opts} || die "$myfail"
[snip and so on...]

This means that every ebuild which uses 'unpack ${A}' should have in
DEPEND a program which is selected by filename extension of sources. So
as I understood your initial suggestion was to make this happen
automatically. And this is very logical as makes ebuilds cleaner and
more terse. So why did you changed your mind and try to write another
eclass (which then should sit in the tree forever), to create duplicate
unpack function instead of just making step you suggested initially? Is
there any intension to remove unpack logic from package manager? And if
yes, why?

--

--

(Continue reading)

headers

Marius Mauch | 20 Jul 17:29

Re: RFC: auto-detection of unpack dependencies

On Sun, 20 Jul 2008 17:41:58 +0400
Peter Volkov <pva <at> gentoo.org> wrote:

> В Чтв, 17/07/2008 в 04:51 +0200, Marius Mauch пишет: 
> > At dev.gentoo.org/~genone/unpack.eclass is the draft for an eclass
> > to implement this feature. 
> 
> Marius, although it's possible to do this things in eclass why is
> eclass better? As I see portage's unpack() already has extension <->
> "program to unpack" relation. Basically unpack() in ebuild.sh has the
> following code:
> 
> unpack() {
> [snip]
>         case "${x##*.}" in
>             tar)
>                 tar xof "${srcdir}${x}" ${tar_opts} || die "$myfail"
>                 ;;
>             tgz)
>                 tar xozf "${srcdir}${x}" ${tar_opts} || die "$myfail"
> [snip and so on...]
> 
> This means that every ebuild which uses 'unpack ${A}' should have in
> DEPEND a program which is selected by filename extension of sources.
> So as I understood your initial suggestion was to make this happen
> automatically. And this is very logical as makes ebuilds cleaner and
> more terse. So why did you changed your mind and try to write another
> eclass (which then should sit in the tree forever), to create
> duplicate unpack function instead of just making step you suggested
> initially? Is there any intension to remove unpack logic from package

(Continue reading)

headers

Joe Peterson | 20 Jul 19:25

Re: RFC: auto-detection of unpack dependencies

Peter Volkov wrote:
> This means that every ebuild which uses 'unpack ${A}' should have in
> DEPEND a program which is selected by filename extension of sources. So
> as I understood your initial suggestion was to make this happen
> automatically. And this is very logical as makes ebuilds cleaner and
> more terse. So why did you changed your mind and try to write another
> eclass (which then should sit in the tree forever), to create duplicate
> unpack function instead of just making step you suggested initially? Is
> there any intension to remove unpack logic from package manager? And if
> yes, why?

++

I also was wondering this.  And if we add "unpack2()", which could be a
little hard to explain in the documentation, it will need to be there at
least until ebuilds stop using it (when portage gets this functionality
ultimately).

For my uses, I'd rather do deps manually and use the official portage
unpack() until portage has such features in order to keep my ebuilds
looking a bit more clean.

					-Joe