Zac Medico | 31 Aug 22:03 2012
Picon

HDEPEND (host dependencies for cross-compilation) for EAPI 5?

For those who may not know, chromium-os currently uses a
hard-host-depends ebuild as a workaround for our lack of HDEPEND support
[1]. We could easily add HDEPEND in EAPI 5 if we want, since we already
have a Portage patch attached to bug #317337 [2]. Here is a summary of
what that Portage patch will do:

In EAPI 5 or later, DEPEND has been divided into two parts:
DEPEND for build-time target dependencies, and HDEPEND for
build-time host dependencies. This division is designed
specifically to minimize difficulty in the process of
adapting ebuilds that were written for earlier EAPIs,
and therefore it also minimizes the adjustments that
ebuild developers will have to make to the thought
processes involved when writing ebuilds from scratch. In
an environment that does not involve cross-compilation,
HDEPEND behaves the same as DEPEND. When an ebuild is
converted from EAPI 4 or earlier to EAPI 5 or later,
in order to support cross-compilation environments, some
dependencies may need to be migrated to HDEPEND.

For ebuilds that have EAPI 5 or later, the emerge
--root-deps option has no effect since it is made obsolete
by division between DEPEND and HDEPEND. If EAPI 4 or
earlier ebuilds are used in combination with EAPI 5 or
later ebuilds, the --root-deps behavior will still be
applied to the EAPI 4 or earlier ebuilds (there is no
behavior change for ebuilds having older EAPIs).

[1]
http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/portage-build-faq
(Continue reading)

Richard Yao | 31 Aug 22:39 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On 08/31/2012 04:03 PM, Zac Medico wrote:
> For those who may not know, chromium-os currently uses a
> hard-host-depends ebuild as a workaround for our lack of HDEPEND support
> [1]. We could easily add HDEPEND in EAPI 5 if we want, since we already
> have a Portage patch attached to bug #317337 [2]. Here is a summary of
> what that Portage patch will do:
> 
> In EAPI 5 or later, DEPEND has been divided into two parts:
> DEPEND for build-time target dependencies, and HDEPEND for
> build-time host dependencies. This division is designed
> specifically to minimize difficulty in the process of
> adapting ebuilds that were written for earlier EAPIs,
> and therefore it also minimizes the adjustments that
> ebuild developers will have to make to the thought
> processes involved when writing ebuilds from scratch. In
> an environment that does not involve cross-compilation,
> HDEPEND behaves the same as DEPEND. When an ebuild is
> converted from EAPI 4 or earlier to EAPI 5 or later,
> in order to support cross-compilation environments, some
> dependencies may need to be migrated to HDEPEND.
> 
> For ebuilds that have EAPI 5 or later, the emerge
> --root-deps option has no effect since it is made obsolete
> by division between DEPEND and HDEPEND. If EAPI 4 or
> earlier ebuilds are used in combination with EAPI 5 or
> later ebuilds, the --root-deps behavior will still be
> applied to the EAPI 4 or earlier ebuilds (there is no
> behavior change for ebuilds having older EAPIs).
> 
> [1]
(Continue reading)

Ciaran McCreesh | 31 Aug 22:46 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, 31 Aug 2012 13:03:00 -0700
Zac Medico <zmedico <at> gentoo.org> wrote:
> For those who may not know, chromium-os currently uses a
> hard-host-depends ebuild as a workaround for our lack of HDEPEND
> support [1]. We could easily add HDEPEND in EAPI 5 if we want, since
> we already have a Portage patch attached to bug #317337 [2]. Here is
> a summary of what that Portage patch will do:
> 
> In EAPI 5 or later, DEPEND has been divided into two parts:
> DEPEND for build-time target dependencies, and HDEPEND for
> build-time host dependencies. This division is designed
> specifically to minimize difficulty in the process of
> adapting ebuilds that were written for earlier EAPIs,
> and therefore it also minimizes the adjustments that
> ebuild developers will have to make to the thought
> processes involved when writing ebuilds from scratch. In
> an environment that does not involve cross-compilation,
> HDEPEND behaves the same as DEPEND. When an ebuild is
> converted from EAPI 4 or earlier to EAPI 5 or later,
> in order to support cross-compilation environments, some
> dependencies may need to be migrated to HDEPEND.
> 
> For ebuilds that have EAPI 5 or later, the emerge
> --root-deps option has no effect since it is made obsolete
> by division between DEPEND and HDEPEND. If EAPI 4 or
> earlier ebuilds are used in combination with EAPI 5 or
> later ebuilds, the --root-deps behavior will still be
> applied to the EAPI 4 or earlier ebuilds (there is no
> behavior change for ebuilds having older EAPIs).

(Continue reading)

Zac Medico | 31 Aug 23:11 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On 08/31/2012 01:46 PM, Ciaran McCreesh wrote:
> On Fri, 31 Aug 2012 13:03:00 -0700
> What exactly would the rules be for handling a package that is in both
> DEPEND and HDEPEND, when ROOT is in effect? Would the versions be
> expected to match? What about use flags?

For the sake of simplicity, I would treat them as entirely independent.
It should be easy enough for users to apply manual configuration
adjustments in order to resolve any conflicts of this nature that may
arise. If there turns out to be a strong demand for additional
constraints, we can consider adding them in a future EAPI (possibly
using a combined DEPENDENCIES variable).

> Also, we're getting rather a lot of *DEPEND variables here... If we're
> making people make major changes to their deps, which for HDEPEND we
> definitely would be,

Well, I not sure that "major changes" is a really good characterization.
We're just talking about migrating a few things from DEPEND to HDEPEND,
and it's not strictly required. The migration is only needed when
fulfilling a request to support cross-compilation in a particular ebuild.

> then the "it's expensive since people would have
> to redo their deps" argument against a combined DEPENDENCIES variable
> goes out of the window, so we should rethink that too.
--

-- 
Thanks,
Zac

(Continue reading)

Fabio Erculiani | 31 Aug 23:40 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

I like this as well.
However, since we're going to introduce a *DEPEND split, how about
splitting PDEPEND as well?

As far as I've seen, PDEPEND has two (or more?) different meanings:
- advisory (for instance, informing users about plugins)
- cycle-breaking to help the dependency solver

Would it be possible to add support for ODEPEND (as in "optional"
dependencies -- I don't really care about the variable name) as well?
This would be quite beneficial under certain circumstances. One of
these is when ebuilds are shipped with PDEPENDs which are not required
at runtime nor for cycle-breaking...

Another scenario in where ODEPEND would be nice to have is with
systemd init files pulled in by USE=systemd (and generally use? (
sys-apps/systemd ) in *DEPEND). Providing full systemd support for all
the packages without forcing users to have it installed, given that
openrc is the de-facto standard init system in Gentoo (and we don't
have any openrc? ( sys-apps/openrc )), would be a nice features for
binpkg repos. Users could then choose to enable or disable ODEPEND
during dependencies calculation via make.conf or argv.

I don't want to diverge too much from the HDEPEND discussion, but I
think that if we're going to split *DEPEND, it might be a good
opportunity to do it right _once_ and _for all_.

--

-- 
Fabio Erculiani

(Continue reading)

Ciaran McCreesh | 31 Aug 23:50 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, 31 Aug 2012 23:40:27 +0200
Fabio Erculiani <lxnay <at> gentoo.org> wrote:
> Would it be possible to add support for ODEPEND (as in "optional"
> dependencies -- I don't really care about the variable name) as well?
> This would be quite beneficial under certain circumstances. One of
> these is when ebuilds are shipped with PDEPENDs which are not required
> at runtime nor for cycle-breaking...

This is what we call "suggestions" and SDEPEND. There are already two
proposals for dealing with this general issue.

--

-- 
Ciaran McCreesh
Zac Medico | 31 Aug 23:58 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On 08/31/2012 02:40 PM, Fabio Erculiani wrote:
> I like this as well.
> However, since we're going to introduce a *DEPEND split, how about
> splitting PDEPEND as well?
> 
> As far as I've seen, PDEPEND has two (or more?) different meanings:
> - advisory (for instance, informing users about plugins)
> - cycle-breaking to help the dependency solver
> 
> Would it be possible to add support for ODEPEND (as in "optional"
> dependencies -- I don't really care about the variable name) as well?
> This would be quite beneficial under certain circumstances. One of
> these is when ebuilds are shipped with PDEPENDs which are not required
> at runtime nor for cycle-breaking...
> 
> Another scenario in where ODEPEND would be nice to have is with
> systemd init files pulled in by USE=systemd (and generally use? (
> sys-apps/systemd ) in *DEPEND). Providing full systemd support for all
> the packages without forcing users to have it installed, given that
> openrc is the de-facto standard init system in Gentoo (and we don't
> have any openrc? ( sys-apps/openrc )), would be a nice features for
> binpkg repos. Users could then choose to enable or disable ODEPEND
> during dependencies calculation via make.conf or argv.
> 
> I don't want to diverge too much from the HDEPEND discussion, but I
> think that if we're going to split *DEPEND, it might be a good
> opportunity to do it right _once_ and _for all_.

For optional dependencies, I'm pretty happy with the "runtime-switchable
USE flags" proposal:
(Continue reading)

Ciaran McCreesh | 1 Sep 00:15 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, 31 Aug 2012 14:58:49 -0700
Zac Medico <zmedico <at> gentoo.org> wrote:
> For optional dependencies, I'm pretty happy with the
> "runtime-switchable USE flags" proposal:
> 
>   https://gist.github.com/2945569

Do we have an implementation of this yet? I have extreme doubts about
the viability of the idea...

--

-- 
Ciaran McCreesh
Zac Medico | 1 Sep 00:58 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On 08/31/2012 03:15 PM, Ciaran McCreesh wrote:
> On Fri, 31 Aug 2012 14:58:49 -0700
> Zac Medico <zmedico <at> gentoo.org> wrote:
>> For optional dependencies, I'm pretty happy with the
>> "runtime-switchable USE flags" proposal:
>>
>>   https://gist.github.com/2945569
> 
> Do we have an implementation of this yet? I have extreme doubts about
> the viability of the idea...

No, but there's a request here:

  https://bugs.gentoo.org/show_bug.cgi?id=424283
--

-- 
Thanks,
Zac

Fabio Erculiani | 1 Sep 00:18 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, Aug 31, 2012 at 11:58 PM, Zac Medico <zmedico <at> gentoo.org> wrote:
>
> For optional dependencies, I'm pretty happy with the "runtime-switchable
> USE flags" proposal:
>
>   https://gist.github.com/2945569

runtime-switchable USE flags for optional dependencies o.O? It sounds
like using a spoon to eat spaghetti to me.
I think SDEPEND is a much simpler approach to the issue, why
introducing a new kind of USE flags to address what really belongs to
*DEPEND?

> --
> Thanks,
> Zac
>

--

-- 
Fabio Erculiani

Michał Górny | 1 Sep 00:59 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Sat, 1 Sep 2012 00:18:07 +0200
Fabio Erculiani <lxnay <at> gentoo.org> wrote:

> On Fri, Aug 31, 2012 at 11:58 PM, Zac Medico <zmedico <at> gentoo.org>
> wrote:
> >
> > For optional dependencies, I'm pretty happy with the
> > "runtime-switchable USE flags" proposal:
> >
> >   https://gist.github.com/2945569
> 
> runtime-switchable USE flags for optional dependencies o.O? It sounds
> like using a spoon to eat spaghetti to me.
> I think SDEPEND is a much simpler approach to the issue, why
> introducing a new kind of USE flags to address what really belongs to
> *DEPEND?

Because otherwise you can't use USE flags. The rationale is there.

--

-- 
Best regards,
Michał Górny
Zac Medico | 1 Sep 01:03 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On 08/31/2012 03:18 PM, Fabio Erculiani wrote:
> On Fri, Aug 31, 2012 at 11:58 PM, Zac Medico <zmedico <at> gentoo.org> wrote:
>>
>> For optional dependencies, I'm pretty happy with the "runtime-switchable
>> USE flags" proposal:
>>
>>   https://gist.github.com/2945569
> 
> runtime-switchable USE flags for optional dependencies o.O? It sounds
> like using a spoon to eat spaghetti to me.

All suggested deps are not equal, so USE flags give you the ability to
pick and choose the ones that you want.

> I think SDEPEND is a much simpler approach to the issue, why
> introducing a new kind of USE flags to address what really belongs to
> *DEPEND?

I guess we could combine the two ideas if we allow USE conditionals
inside SDEPEND.
--

-- 
Thanks,
Zac

Ciaran McCreesh | 1 Sep 01:07 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, 31 Aug 2012 16:03:25 -0700
Zac Medico <zmedico <at> gentoo.org> wrote:
> > runtime-switchable USE flags for optional dependencies o.O? It
> > sounds like using a spoon to eat spaghetti to me.
> 
> All suggested deps are not equal, so USE flags give you the ability to
> pick and choose the ones that you want.

So does --take / --ignore with suggested dependencies, with the added
advantage that suggested packages don't end up being brought in without
user request just because a user has a particular use flag enabled
globally.

--

-- 
Ciaran McCreesh
Zac Medico | 1 Sep 03:45 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On 08/31/2012 04:07 PM, Ciaran McCreesh wrote:
> On Fri, 31 Aug 2012 16:03:25 -0700
> Zac Medico <zmedico <at> gentoo.org> wrote:
>>> runtime-switchable USE flags for optional dependencies o.O? It
>>> sounds like using a spoon to eat spaghetti to me.
>>
>> All suggested deps are not equal, so USE flags give you the ability to
>> pick and choose the ones that you want.
> 
> So does --take / --ignore with suggested dependencies, with the added
> advantage that suggested packages don't end up being brought in without
> user request just because a user has a particular use flag enabled
> globally.

If the USE flags have ambiguous meanings doesn't that mean that they've
been poorly named?
--

-- 
Thanks,
Zac

Ciaran McCreesh | 1 Sep 18:00 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, 31 Aug 2012 18:45:59 -0700
Zac Medico <zmedico <at> gentoo.org> wrote:
> On 08/31/2012 04:07 PM, Ciaran McCreesh wrote:
> > On Fri, 31 Aug 2012 16:03:25 -0700
> > Zac Medico <zmedico <at> gentoo.org> wrote:
> >>> runtime-switchable USE flags for optional dependencies o.O? It
> >>> sounds like using a spoon to eat spaghetti to me.
> >>
> >> All suggested deps are not equal, so USE flags give you the
> >> ability to pick and choose the ones that you want.
> > 
> > So does --take / --ignore with suggested dependencies, with the
> > added advantage that suggested packages don't end up being brought
> > in without user request just because a user has a particular use
> > flag enabled globally.
> 
> If the USE flags have ambiguous meanings doesn't that mean that
> they've been poorly named?

It's not like that. It's that in practice, suggestions are mostly for a
particular specific feature (such as git-send-email support), not for a
general concept (such as email in general).

It also defeats the point of suggestions, if they're not made visible.
For users, suggestions should look like suggestions, and they should
be able to see them easily.

--

-- 
Ciaran McCreesh
(Continue reading)

Zac Medico | 1 Sep 20:15 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On 09/01/2012 09:00 AM, Ciaran McCreesh wrote:
> On Fri, 31 Aug 2012 18:45:59 -0700
> Zac Medico <zmedico <at> gentoo.org> wrote:
>> On 08/31/2012 04:07 PM, Ciaran McCreesh wrote:
>>> On Fri, 31 Aug 2012 16:03:25 -0700
>>> Zac Medico <zmedico <at> gentoo.org> wrote:
>>>>> runtime-switchable USE flags for optional dependencies o.O? It
>>>>> sounds like using a spoon to eat spaghetti to me.
>>>>
>>>> All suggested deps are not equal, so USE flags give you the
>>>> ability to pick and choose the ones that you want.
>>>
>>> So does --take / --ignore with suggested dependencies, with the
>>> added advantage that suggested packages don't end up being brought
>>> in without user request just because a user has a particular use
>>> flag enabled globally.
>>
>> If the USE flags have ambiguous meanings doesn't that mean that
>> they've been poorly named?
> 
> It's not like that. It's that in practice, suggestions are mostly for a
> particular specific feature (such as git-send-email support), not for a
> general concept (such as email in general).
> 
> It also defeats the point of suggestions, if they're not made visible.
> For users, suggestions should look like suggestions, and they should
> be able to see them easily.

This sounds more like a user-interface issue than a problem with
runtime-switchable USE flags (GLEP 62). The nice thing about
(Continue reading)

Ciaran McCreesh | 1 Sep 20:28 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Sat, 01 Sep 2012 11:15:16 -0700
Zac Medico <zmedico <at> gentoo.org> wrote:
> This sounds more like a user-interface issue than a problem with
> runtime-switchable USE flags (GLEP 62). The nice thing about
> runtime-switchable USE flags is that makes it possible to allow users
> to unify all of their optional dependency choices in their USE flag
> settings.

The nice thing about GLEP 62 is that no-one has implemented it and tried
it with lots of packages and a bunch of users, thus figuring out just
how much of a pain in the ass getting this right is... Right now we're
debating the merits of a tried and tested solution versus an entirely
hypothetical idea.

If you really think unification is an advantage, you could treat
exheres-style suggestion names as a special USE_EXPAND group. But
practical experience suggests that suggestions *shouldn't* be unified,
and that the way to make the feature useful to users is to get the user
to explicitly accept or reject suggestions, and to make suggestions look
like suggestions.

--

-- 
Ciaran McCreesh
Michał Górny | 1 Sep 09:42 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Sat, 1 Sep 2012 00:07:38 +0100
Ciaran McCreesh <ciaran.mccreesh <at> googlemail.com> wrote:

> On Fri, 31 Aug 2012 16:03:25 -0700
> Zac Medico <zmedico <at> gentoo.org> wrote:
> > > runtime-switchable USE flags for optional dependencies o.O? It
> > > sounds like using a spoon to eat spaghetti to me.
> > 
> > All suggested deps are not equal, so USE flags give you the ability
> > to pick and choose the ones that you want.
> 
> So does --take / --ignore with suggested dependencies, with the added
> advantage that suggested packages don't end up being brought in
> without user request just because a user has a particular use flag
> enabled globally.

Yes because runtime SSL support is *so much* different than build-time
one.

--

-- 
Best regards,
Michał Górny
Ciaran McCreesh | 1 Sep 01:09 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, 31 Aug 2012 16:03:25 -0700
Zac Medico <zmedico <at> gentoo.org> wrote:
> > I think SDEPEND is a much simpler approach to the issue, why
> > introducing a new kind of USE flags to address what really belongs
> > to *DEPEND?
> 
> I guess we could combine the two ideas if we allow USE conditionals
> inside SDEPEND.

But you don't want to do that... The point of suggestions is that they
can be considered on their own merits.

--

-- 
Ciaran McCreesh
Michał Górny | 1 Sep 01:00 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, 31 Aug 2012 14:58:49 -0700
Zac Medico <zmedico <at> gentoo.org> wrote:

> On 08/31/2012 02:40 PM, Fabio Erculiani wrote:
> > I like this as well.
> > However, since we're going to introduce a *DEPEND split, how about
> > splitting PDEPEND as well?
> > 
> > As far as I've seen, PDEPEND has two (or more?) different meanings:
> > - advisory (for instance, informing users about plugins)
> > - cycle-breaking to help the dependency solver
> > 
> > Would it be possible to add support for ODEPEND (as in "optional"
> > dependencies -- I don't really care about the variable name) as
> > well? This would be quite beneficial under certain circumstances.
> > One of these is when ebuilds are shipped with PDEPENDs which are
> > not required at runtime nor for cycle-breaking...
> > 
> > Another scenario in where ODEPEND would be nice to have is with
> > systemd init files pulled in by USE=systemd (and generally use? (
> > sys-apps/systemd ) in *DEPEND). Providing full systemd support for
> > all the packages without forcing users to have it installed, given
> > that openrc is the de-facto standard init system in Gentoo (and we
> > don't have any openrc? ( sys-apps/openrc )), would be a nice
> > features for binpkg repos. Users could then choose to enable or
> > disable ODEPEND during dependencies calculation via make.conf or
> > argv.
> > 
> > I don't want to diverge too much from the HDEPEND discussion, but I
> > think that if we're going to split *DEPEND, it might be a good
(Continue reading)

Ciaran McCreesh | 31 Aug 23:53 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Fri, 31 Aug 2012 14:11:38 -0700
Zac Medico <zmedico <at> gentoo.org> wrote:
> On 08/31/2012 01:46 PM, Ciaran McCreesh wrote:
> > On Fri, 31 Aug 2012 13:03:00 -0700
> > What exactly would the rules be for handling a package that is in
> > both DEPEND and HDEPEND, when ROOT is in effect? Would the versions
> > be expected to match? What about use flags?
> 
> For the sake of simplicity, I would treat them as entirely
> independent. It should be easy enough for users to apply manual
> configuration adjustments in order to resolve any conflicts of this
> nature that may arise. If there turns out to be a strong demand for
> additional constraints, we can consider adding them in a future EAPI
> (possibly using a combined DEPENDENCIES variable).

The thing is... Without some kind of "the same" constraint, we'd be
adding a feature which would probably work most of the time only by
coincidence.

> > Also, we're getting rather a lot of *DEPEND variables here... If
> > we're making people make major changes to their deps, which for
> > HDEPEND we definitely would be,
> 
> Well, I not sure that "major changes" is a really good
> characterization. We're just talking about migrating a few things
> from DEPEND to HDEPEND, and it's not strictly required. The migration
> is only needed when fulfilling a request to support cross-compilation
> in a particular ebuild.

Where are you getting "a few" from? Is this "a few seems to be enough
(Continue reading)

Zac Medico | 1 Sep 00:16 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On 08/31/2012 02:53 PM, Ciaran McCreesh wrote:
> On Fri, 31 Aug 2012 14:11:38 -0700
> Zac Medico <zmedico <at> gentoo.org> wrote:
>> On 08/31/2012 01:46 PM, Ciaran McCreesh wrote:
>>> On Fri, 31 Aug 2012 13:03:00 -0700
>>> What exactly would the rules be for handling a package that is in
>>> both DEPEND and HDEPEND, when ROOT is in effect? Would the versions
>>> be expected to match? What about use flags?
>>
>> For the sake of simplicity, I would treat them as entirely
>> independent. It should be easy enough for users to apply manual
>> configuration adjustments in order to resolve any conflicts of this
>> nature that may arise. If there turns out to be a strong demand for
>> additional constraints, we can consider adding them in a future EAPI
>> (possibly using a combined DEPENDENCIES variable).
> 
> The thing is... Without some kind of "the same" constraint, we'd be
> adding a feature which would probably work most of the time only by
> coincidence.

Shrug, I don't do any cross-compilation myself, but maybe someone who
does could comment on the usefulness of this kind of constraint. Mike?
Brian?

>>> Also, we're getting rather a lot of *DEPEND variables here... If
>>> we're making people make major changes to their deps, which for
>>> HDEPEND we definitely would be,
>>
>> Well, I not sure that "major changes" is a really good
>> characterization. We're just talking about migrating a few things
(Continue reading)

Jorge Manuel B. S. Vicetto | 5 Sep 02:06 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?


On 31-08-2012 20:46, Ciaran McCreesh wrote:

<snip>

> Also, we're getting rather a lot of *DEPEND variables here... If
> we're making people make major changes to their deps, which for
> HDEPEND we definitely would be, then the "it's expensive since
> people would have to redo their deps" argument against a combined
> DEPENDENCIES variable goes out of the window, so we should rethink
> that too.

I have to agree with Ciaran, instead of multiplying DEPEND variables,
it's probably time we move to a single DEPENDENCIES variable.

--

-- 
Regards,

Jorge Vicetto (jmbsvicetto) - jmbsvicetto at gentoo dot org
Gentoo- forums / Userrel / Devrel / KDE / Elections / RelEng
Fabio Erculiani | 5 Sep 09:19 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, Sep 5, 2012 at 2:06 AM, Jorge Manuel B. S. Vicetto
<jmbsvicetto <at> gentoo.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 31-08-2012 20:46, Ciaran McCreesh wrote:
>
> <snip>
>
>> Also, we're getting rather a lot of *DEPEND variables here... If
>> we're making people make major changes to their deps, which for
>> HDEPEND we definitely would be, then the "it's expensive since
>> people would have to redo their deps" argument against a combined
>> DEPENDENCIES variable goes out of the window, so we should rethink
>> that too.
>
> I have to agree with Ciaran, instead of multiplying DEPEND variables,
> it's probably time we move to a single DEPENDENCIES variable.

So, let's say that I only want to apply a filter function against the
buildtime dependencies, in that case I'd need to parse *all* the
dependencies anyway?
The complexity would become:
O((b + r + p) * P)
b = amount of buildtime dependencies
r = amount of runtime dependencies
p = amount of post-dependencies
P = amount of packages i need to apply the filter to

Instead of just:
(Continue reading)

Ciaran McCreesh | 5 Sep 09:27 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, 5 Sep 2012 09:19:45 +0200
Fabio Erculiani <lxnay <at> gentoo.org> wrote:
> So, let's say that I only want to apply a filter function against the
> buildtime dependencies, in that case I'd need to parse *all* the
> dependencies anyway?

Yes, and? If your dependency parser's time isn't "so tiny it makes
absolutely no difference compared to the filesystem access time needed
to get the metadata in the first place" then you're doing something
severely wrong.

And if we are going the "parser time" route, then given the heavy
intersection between build and run dependencies, the overall amount of
data to be processed in the common case that all dependencies are
required is smaller with unified DEPENDENCIES.

--

-- 
Ciaran McCreesh
Rich Freeman | 5 Sep 12:44 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, Sep 5, 2012 at 3:19 AM, Fabio Erculiani <lxnay <at> gentoo.org> wrote:
> The complexity would become:
> O((b + r + p) * P)
> b = amount of buildtime dependencies
> r = amount of runtime dependencies
> p = amount of post-dependencies
> P = amount of packages i need to apply the filter to
>
> Instead of just:
> O(b * P)

Well, actually, in both cases the complexity is O(n), assuming you
only need to make a constant number of passes through the deps per
package.

The whole point of O notation is that it is about how it scales, not
how long it takes.  An O(n) algorithm can actually be slower than an
O(n^n) algorithm even on a large dataset.  However, the key is that at
some point if the dataset gets large enough the O(n) algorithm will
always become faster.

I tend to agree with Cirian though - the time for some code to run
through the dependencies array and do something isn't going to be very
high.  If a piece of code has to do it many times there is nothing
that says the package manager can't index it.

Rich

Fabio Erculiani | 5 Sep 13:23 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, Sep 5, 2012 at 12:44 PM, Rich Freeman <rich0 <at> gentoo.org> wrote:
> On Wed, Sep 5, 2012 at 3:19 AM, Fabio Erculiani <lxnay <at> gentoo.org> wrote:
>> The complexity would become:
>> O((b + r + p) * P)
>> b = amount of buildtime dependencies
>> r = amount of runtime dependencies
>> p = amount of post-dependencies
>> P = amount of packages i need to apply the filter to
>>
>> Instead of just:
>> O(b * P)
>
> Well, actually, in both cases the complexity is O(n), assuming you
> only need to make a constant number of passes through the deps per
> package.
>
> The whole point of O notation is that it is about how it scales, not
> how long it takes.  An O(n) algorithm can actually be slower than an
> O(n^n) algorithm even on a large dataset.  However, the key is that at
> some point if the dataset gets large enough the O(n) algorithm will
> always become faster.
>
> I tend to agree with Cirian though - the time for some code to run
> through the dependencies array and do something isn't going to be very
> high.  If a piece of code has to do it many times there is nothing
> that says the package manager can't index it.

I don't want to diverge (again) from the main topic, but I think that
you're just oversimplifying it.
If you consider parsing an ebuild something hidden behind a lot of
(Continue reading)

Ciaran McCreesh | 5 Sep 13:27 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, 5 Sep 2012 13:23:19 +0200
Fabio Erculiani <lxnay <at> gentoo.org> wrote:
> If you consider parsing an ebuild something hidden behind a lot of
> abstraction layers, O(n) vs O(n/2) is a big difference, even if both,
> normalized, are still O(n). And I would never design an API which
> assumes that O(n/2) equals to O(n), because you don't know how that is
> going to be used in upper layers, today, tomorrow and in 5 years.

Uhm. O(n) == O(n/2). Anything assuming they're different is just wrong.

--

-- 
Ciaran McCreesh
Rich Freeman | 5 Sep 14:46 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, Sep 5, 2012 at 7:27 AM, Ciaran McCreesh
<ciaran.mccreesh <at> googlemail.com> wrote:
> Uhm. O(n) == O(n/2). Anything assuming they're different is just wrong.

We're basically debating definitions.  O notation is used to indicate
how algorithms scale and nobody uses O(n/2) and such as a result.

An algorithm that is twice as slow is still twice as slow.  That might
or might not matter.  However,  this isn't the domain where O notation
is used.  You use O notation when your algorithm takes 30 seconds to
run and you want to know what happens with the dataset doubles 3000
times.  It generally doesn't matter if the result is that it will take
1 trillion years to operate or 2 trillion years.  You care more about
whether it will take minutes, hours, weeks, years, or whatever.

I can't really think of any practical examples where multiplying the
time to parse a list of maybe 50 items vs 5 lists of 10 items is going
to make that big of a difference.  They're just lines in a text file -
your CPU can compare a few billions characters per second.  Sure, if
you add 75 layers of abstraction you might be able to find just the
right point where a factor of 5 is going to make it intolerable but a
factor of 1 is almost acceptable, but go ahead and add/remove a few
layers and suddenly it is all fine or all horrible anyway.  That is a
bit contrived.  That's why everybody ignores constant factors in O
notation anyway.

Rich

Alexis Ballier | 5 Sep 15:28 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, 5 Sep 2012 08:46:13 -0400
Rich Freeman <rich0 <at> gentoo.org> wrote:

> On Wed, Sep 5, 2012 at 7:27 AM, Ciaran McCreesh
> <ciaran.mccreesh <at> googlemail.com> wrote:
> > Uhm. O(n) == O(n/2). Anything assuming they're different is just
> > wrong.
> 
> We're basically debating definitions.  O notation is used to indicate
> how algorithms scale and nobody uses O(n/2) and such as a result.
> 
> An algorithm that is twice as slow is still twice as slow.  That might
> or might not matter.  However,  this isn't the domain where O notation
> is used.  You use O notation when your algorithm takes 30 seconds to
> run and you want to know what happens with the dataset doubles 3000
> times.  It generally doesn't matter if the result is that it will take
> 1 trillion years to operate or 2 trillion years.  You care more about
> whether it will take minutes, hours, weeks, years, or whatever.
> 
> I can't really think of any practical examples where multiplying the
> time to parse a list of maybe 50 items vs 5 lists of 10 items is going
> to make that big of a difference.  They're just lines in a text file -
> your CPU can compare a few billions characters per second.  Sure, if
> you add 75 layers of abstraction you might be able to find just the
> right point where a factor of 5 is going to make it intolerable but a
> factor of 1 is almost acceptable, but go ahead and add/remove a few
> layers and suddenly it is all fine or all horrible anyway.  That is a
> bit contrived.  That's why everybody ignores constant factors in O
> notation anyway.

(Continue reading)

Alec Warner | 5 Sep 17:24 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, Sep 5, 2012 at 2:46 PM, Rich Freeman <rich0 <at> gentoo.org> wrote:
> On Wed, Sep 5, 2012 at 7:27 AM, Ciaran McCreesh
> <ciaran.mccreesh <at> googlemail.com> wrote:
>> Uhm. O(n) == O(n/2). Anything assuming they're different is just wrong.
>
> We're basically debating definitions.  O notation is used to indicate
> how algorithms scale and nobody uses O(n/2) and such as a result.
>
> An algorithm that is twice as slow is still twice as slow.  That might
> or might not matter.  However,  this isn't the domain where O notation
> is used.  You use O notation when your algorithm takes 30 seconds to
> run and you want to know what happens with the dataset doubles 3000
> times.  It generally doesn't matter if the result is that it will take
> 1 trillion years to operate or 2 trillion years.  You care more about
> whether it will take minutes, hours, weeks, years, or whatever.
>
> I can't really think of any practical examples where multiplying the
> time to parse a list of maybe 50 items vs 5 lists of 10 items is going
> to make that big of a difference.  They're just lines in a text file -
> your CPU can compare a few billions characters per second.  Sure, if
> you add 75 layers of abstraction you might be able to find just the
> right point where a factor of 5 is going to make it intolerable but a
> factor of 1 is almost acceptable, but go ahead and add/remove a few
> layers and suddenly it is all fine or all horrible anyway.  That is a
> bit contrived.  That's why everybody ignores constant factors in O
> notation anyway.

so tl;dr, this doesn't matter because string comparison is very fast.

-A
(Continue reading)

Michał Górny | 5 Sep 18:15 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, 05 Sep 2012 00:06:45 +0000
"Jorge Manuel B. S. Vicetto" <jmbsvicetto <at> gentoo.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 31-08-2012 20:46, Ciaran McCreesh wrote:
> 
> <snip>
> 
> > Also, we're getting rather a lot of *DEPEND variables here... If
> > we're making people make major changes to their deps, which for
> > HDEPEND we definitely would be, then the "it's expensive since
> > people would have to redo their deps" argument against a combined
> > DEPENDENCIES variable goes out of the window, so we should rethink
> > that too.
> 
> I have to agree with Ciaran, instead of multiplying DEPEND variables,
> it's probably time we move to a single DEPENDENCIES variable.

Well, if you really insist that Gentoo is the right place to reinvent
the wheel of bash variables...

If we really want to go this route, then please at least require
explicit label at start of DEPENDENCIES. And the same when appending to
DEPENDENCIES -- just so 'unlikely' mistakes will leave us with hours of
debugging.

Furthermore, please think of eclasses. They *all* will have to append
dependencies in two ways, in an EAPI-conditional way. Exherbo doesn't
(Continue reading)

Ciaran McCreesh | 6 Sep 07:58 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, 5 Sep 2012 18:15:43 +0200
Michał Górny <mgorny <at> gentoo.org> wrote:
> If we really want to go this route, then please at least require
> explicit label at start of DEPENDENCIES. And the same when appending
> to DEPENDENCIES -- just so 'unlikely' mistakes will leave us with
> hours of debugging.

We should take the exheres-0 rules for labels and eclasses, which limit
labels' scopes to blocks, and which introduce an extra ( ) block around
the outside when doing eclass variable merging.

> Not that appending dependencies in eclasses is really that good idea.

Dependencies aren't appended over eclasses, they're merged.

(And I have a sneaking recollection of PMS not documenting this
properly...)

> Remember that this requirement will actually cause migration to EAPI 5
> to be even harder than to any previous EAPIs. Migrating a single
> ebuild will require rewriting the dependencies, and migrating an
> eclass will require adding a lot of dirty code.

Migrating to EAPI 5 requires rewriting dependencies anyway if we're
adding in HDEPEND. Also, earlier EAPIs have introduced new phase
functions, which is a far ickier change for ebuilds than this.

> Especially if it is python.eclass.

You know what the solution there is...
(Continue reading)

Michał Górny | 6 Sep 09:39 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Thu, 6 Sep 2012 06:58:51 +0100
Ciaran McCreesh <ciaran.mccreesh <at> googlemail.com> wrote:

> On Wed, 5 Sep 2012 18:15:43 +0200
> Michał Górny <mgorny <at> gentoo.org> wrote:
> > If we really want to go this route, then please at least require
> > explicit label at start of DEPENDENCIES. And the same when appending
> > to DEPENDENCIES -- just so 'unlikely' mistakes will leave us with
> > hours of debugging.
> 
> We should take the exheres-0 rules for labels and eclasses, which
> limit labels' scopes to blocks, and which introduce an extra ( )
> block around the outside when doing eclass variable merging.

Because? I believe we should take 'Gentoo rules', including required
explicit build+run at the start.

> > Not that appending dependencies in eclasses is really that good
> > idea.
> 
> Dependencies aren't appended over eclasses, they're merged.

Thanks for correcting my wording, like the naming was really relevant
to the topic.

> (And I have a sneaking recollection of PMS not documenting this
> properly...)

Yes, I think PMS is pretty silent about this. I think it doesn't even
say that in phase functions the contents are implementation-defined.
(Continue reading)

Ciaran McCreesh | 6 Sep 10:00 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Thu, 6 Sep 2012 09:39:25 +0200
Michał Górny <mgorny <at> gentoo.org> wrote:
> On Thu, 6 Sep 2012 06:58:51 +0100
> Ciaran McCreesh <ciaran.mccreesh <at> googlemail.com> wrote:
> > On Wed, 5 Sep 2012 18:15:43 +0200
> > Michał Górny <mgorny <at> gentoo.org> wrote:
> > > If we really want to go this route, then please at least require
> > > explicit label at start of DEPENDENCIES. And the same when
> > > appending to DEPENDENCIES -- just so 'unlikely' mistakes will
> > > leave us with hours of debugging.
> > 
> > We should take the exheres-0 rules for labels and eclasses, which
> > limit labels' scopes to blocks, and which introduce an extra ( )
> > block around the outside when doing eclass variable merging.
> 
> Because? I believe we should take 'Gentoo rules', including required
> explicit build+run at the start.

You mean, you want to invent some new rules that don't quite work,
rather than using the ones that do... The whole "initial labels" thing
isn't an issue for exheres-0, since rather than appending, the
resulting metadata variable ends up with extra ( ) blocks like this:

    (
        stuff/from-the-exheres
    )
    (
        stuff/from-exlib-1
    )
    (
(Continue reading)

Michał Górny | 6 Sep 10:27 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Thu, 6 Sep 2012 09:00:40 +0100
Ciaran McCreesh <ciaran.mccreesh <at> googlemail.com> wrote:

> On Thu, 6 Sep 2012 09:39:25 +0200
> Michał Górny <mgorny <at> gentoo.org> wrote:
> > On Thu, 6 Sep 2012 06:58:51 +0100
> > Ciaran McCreesh <ciaran.mccreesh <at> googlemail.com> wrote:
> > > On Wed, 5 Sep 2012 18:15:43 +0200
> > > Michał Górny <mgorny <at> gentoo.org> wrote:
> > > > If we really want to go this route, then please at least require
> > > > explicit label at start of DEPENDENCIES. And the same when
> > > > appending to DEPENDENCIES -- just so 'unlikely' mistakes will
> > > > leave us with hours of debugging.
> > > 
> > > We should take the exheres-0 rules for labels and eclasses, which
> > > limit labels' scopes to blocks, and which introduce an extra ( )
> > > block around the outside when doing eclass variable merging.
> > 
> > Because? I believe we should take 'Gentoo rules', including required
> > explicit build+run at the start.
> 
> You mean, you want to invent some new rules that don't quite work,
> rather than using the ones that do... The whole "initial labels" thing
> isn't an issue for exheres-0, since rather than appending, the
> resulting metadata variable ends up with extra ( ) blocks like this:
> 
>     (
>         stuff/from-the-exheres
>     )
>     (
(Continue reading)

Ciaran McCreesh | 6 Sep 10:31 2012

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Thu, 6 Sep 2012 10:27:55 +0200
Michał Górny <mgorny <at> gentoo.org> wrote:
> But what I was saying is that I dislike the implicit 'no label ==
> build+run'. It's unclear, very unclear.
> 
> Why the heck:
> 
>     ( foo/bar )
> 
> introduces another label than:
> 
>     use? ( foo/bar )
> 
> ?

Labels are propagated into child blocks, so it doesn't introduce a new
label. I think you've misunderstood something here...

--

-- 
Ciaran McCreesh
Michał Górny | 6 Sep 10:42 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Thu, 6 Sep 2012 09:31:29 +0100
Ciaran McCreesh <ciaran.mccreesh <at> googlemail.com> wrote:

> On Thu, 6 Sep 2012 10:27:55 +0200
> Michał Górny <mgorny <at> gentoo.org> wrote:
> > But what I was saying is that I dislike the implicit 'no label ==
> > build+run'. It's unclear, very unclear.
> > 
> > Why the heck:
> > 
> >     ( foo/bar )
> > 
> > introduces another label than:
> > 
> >     use? ( foo/bar )
> > 
> > ?
> 
> Labels are propagated into child blocks, so it doesn't introduce a new
> label. I think you've misunderstood something here...

Right, too many ()s for me.

Then my argument that I want us to require explicit label at start of
global () still holds.

--

-- 
Best regards,
Michał Górny
(Continue reading)

Brian Harring | 6 Sep 10:11 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Wed, Sep 05, 2012 at 12:06:45AM +0000, Jorge Manuel B. S. Vicetto wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 31-08-2012 20:46, Ciaran McCreesh wrote:
> 
> <snip>
> 
> > Also, we're getting rather a lot of *DEPEND variables here... If
> > we're making people make major changes to their deps, which for
> > HDEPEND we definitely would be, then the "it's expensive since
> > people would have to redo their deps" argument against a combined
> > DEPENDENCIES variable goes out of the window, so we should rethink
> > that too.
> 
> I have to agree with Ciaran, instead of multiplying DEPEND variables,
> it's probably time we move to a single DEPENDENCIES variable.

Personally, my complaints re: it are that  1) while minor, the 
labels in some cases are overly verbose; recommendations instead of 
recommends, suggestions instead of suggests, etc.  2) An actual 
flaw in their design (imo): it tries to intermix two different 
forms of parsing, without any real justification for *why* beyond 
*hey look kids, I can!*;  The two can intersect in slightly fucked 
up ways, case in point:

DEPENDENCIES="
run+build:
  cat/the
  x? ( cat/cow
(Continue reading)

Michał Górny | 11 Sep 16:13 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Thu, 6 Sep 2012 01:11:45 -0700
Brian Harring <ferringb <at> gmail.com> wrote:

> A compatibility hack that stacks them is strongly advisable;
> something akin to the following:
> 
> Literally, we do the following:
> inherit() {
>   if eapi blah; then
>     local DEPEND PDEPEND RDEPEND
>     <usual saving/protection of DEPENDENCIES var>
>   else
>     <usual saving/protection of DEPEND/PDEPEND/RDEPEND vars>
>   fi
>   <normal sourcing machinery>
>   if eapi blah; then
>     local _deps=( ) _x
>     for _x in DEPENDENCIES DEPEND RDEPEND PDEPEND; do
>       [ -n "${!_x}" ] && deps+=( "${!_x}" )
>     done
>     [ ${#deps} -ne 0 ] && DEPENDENCIES="${deps[*]}"
>     unset DEPEND RDEPEND PDEPEND _x _deps
>     <normal stacking/restoration of DEPENDENCIES rules>
>   else
>     <normal stacking/restoration of RDEPEND/PDEPEND/DEPEND>
>   fi
> }
> 
> Via that, we eclasses that are pure DEPENDENCIES eapi wise, just set 
> the DEPENDENCIES directly; those that have to support multiple eapi's 
(Continue reading)

Brian Harring | 12 Sep 06:54 2012
Picon

Re: HDEPEND (host dependencies for cross-compilation) for EAPI 5?

On Tue, Sep 11, 2012 at 7:13 AM, Michał Górny <mgorny <at> gentoo.org> wrote:
> On Thu, 6 Sep 2012 01:11:45 -0700
> Brian Harring <ferringb <at> gmail.com> wrote:
>
>> A compatibility hack that stacks them is strongly advisable;
>> something akin to the following:
>>
>> Literally, we do the following:
>> inherit() {
>>   if eapi blah; then
>>     local DEPEND PDEPEND RDEPEND
>>     <usual saving/protection of DEPENDENCIES var>
>>   else
>>     <usual saving/protection of DEPEND/PDEPEND/RDEPEND vars>
>>   fi
>>   <normal sourcing machinery>
>>   if eapi blah; then
>>     local _deps=( ) _x
>>     for _x in DEPENDENCIES DEPEND RDEPEND PDEPEND; do
>>       [ -n "${!_x}" ] && deps+=( "${!_x}" )
>>     done
>>     [ ${#deps} -ne 0 ] && DEPENDENCIES="${deps[*]}"
>>     unset DEPEND RDEPEND PDEPEND _x _deps
>>     <normal stacking/restoration of DEPENDENCIES rules>
>>   else
>>     <normal stacking/restoration of RDEPEND/PDEPEND/DEPEND>
>>   fi
>> }
>>
>> Via that, we eclasses that are pure DEPENDENCIES eapi wise, just set
(Continue reading)


Gmane