Discussion about libraries for Haskell

Johan Tibell | 7 Dec 2012 17:20

The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Thu, Dec 6, 2012 at 11:52 PM, Michael Snoyman <michael <at> snoyman.com> wrote:
> I thought the natural response to this would be to put a lower bound on
> text. After all, following the PVP, there should be no problem with updating
> a minor version number. However, when I did so, many users of the Haskell
> Platform were no longer able to compile due to conflicting dependencies. The
> result: I had to put in some dirty conditional compilation tricks in
> persistent to avoid inlining `pack` when using older versions of text.

I've meant to write about this but haven't yet found time to do so.

Assuming that the goal of the PVP is to make sure that a packages that
previously compiled continues to compile, dependencies must be taken
into account when computing package version number bumps. For example,
given these packages:

A-1.0
B-1.0 depends on A-1.0.0.*
C-1.0 depends on A-1.0.0.* and B-1.0.0.*

Now assume that A-2.0 is released and B starts depending on the new
version of A (without changing its own API). He now have this
situation:

A-1.0
A-2.0
B-1.?.?.? depends on A-2.0.0.*
C-1.0 depends on A-1.0.0.* and B-1.0.0.*

If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
longer compiles (due to a version constraint failure) with B-1.0.0.1

(Continue reading)

headers

Ian Lynagh | 7 Dec 2012 17:28

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:
> 
> If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
> longer compiles (due to a version constraint failure) with B-1.0.0.1

Won't cabal-install select B-1.0 in that case, so installing C will
still work?

Thanks
Ian

headers

Johan Tibell | 7 Dec 2012 17:30

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh <ian <at> well-typed.com> wrote:
> On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:
>>
>> If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
>> longer compiles (due to a version constraint failure) with B-1.0.0.1
>
> Won't cabal-install select B-1.0 in that case, so installing C will
> still work?

See my aside at the end. Yes, it will backtrack and use the older
version, but the users might get confused why the new version isn't
used as it's version number suggests that it could be.

headers

Ian Lynagh | 7 Dec 2012 17:41

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Fri, Dec 07, 2012 at 08:30:45AM -0800, Johan Tibell wrote:
> On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh <ian <at> well-typed.com> wrote:
> > On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:
> >>
> >> If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
> >> longer compiles (due to a version constraint failure) with B-1.0.0.1
> >
> > Won't cabal-install select B-1.0 in that case, so installing C will
> > still work?
> 
> See my aside at the end. Yes, it will backtrack and use the older
> version, but the users might get confused why the new version isn't
> used as it's version number suggests that it could be.

I don't think most users would notice, and for those that do the "-v"
flag to cabal-install ought to tell them why.

I don't think this (theoretical?) confusion issue is worth the effort of
bumping major versions all the way up the dependency tree.

Thanks
Ian

headers

Conrad Parker | 10 Dec 2012 05:15

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On 8 December 2012 00:41, Ian Lynagh <ian <at> well-typed.com> wrote:
> On Fri, Dec 07, 2012 at 08:30:45AM -0800, Johan Tibell wrote:
>> On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh <ian <at> well-typed.com> wrote:
>> > On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:
>> >>
>> >> If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
>> >> longer compiles (due to a version constraint failure) with B-1.0.0.1
>> >
>> > Won't cabal-install select B-1.0 in that case, so installing C will
>> > still work?
>>
>> See my aside at the end. Yes, it will backtrack and use the older
>> version, but the users might get confused why the new version isn't
>> used as it's version number suggests that it could be.
>
> I don't think most users would notice, and for those that do the "-v"
> flag to cabal-install ought to tell them why.
>
> I don't think this (theoretical?) confusion issue is worth the effort of
> bumping major versions all the way up the dependency tree.

What if the minor version bump in B is for fixing a security issue?

Conrad.

headers

Ian Lynagh | 10 Dec 2012 11:56

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Mon, Dec 10, 2012 at 12:15:47PM +0800, Conrad Parker wrote:
> On 8 December 2012 00:41, Ian Lynagh <ian <at> well-typed.com> wrote:
> > On Fri, Dec 07, 2012 at 08:30:45AM -0800, Johan Tibell wrote:
> >> On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh <ian <at> well-typed.com> wrote:
> >> > On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:
> >> >>
> >> >> If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
> >> >> longer compiles (due to a version constraint failure) with B-1.0.0.1
> >> >
> >> > Won't cabal-install select B-1.0 in that case, so installing C will
> >> > still work?
> >>
> >> See my aside at the end. Yes, it will backtrack and use the older
> >> version, but the users might get confused why the new version isn't
> >> used as it's version number suggests that it could be.
> >
> > I don't think most users would notice, and for those that do the "-v"
> > flag to cabal-install ought to tell them why.
> >
> > I don't think this (theoretical?) confusion issue is worth the effort of
> > bumping major versions all the way up the dependency tree.
> 
> What if the minor version bump in B is for fixing a security issue?

Then the proposal doesn't help: It doesn't matter whether you give the
new version of B a version number of 1.0.0.1 or 2.0.0.0, cabal-install
will still install B 1.0 and A 1.0 when asked to install C.

Thanks
Ian

(Continue reading)

headers

Duncan Coutts | 8 Dec 2012 21:11

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On 7 December 2012 16:30, Johan Tibell <johan.tibell <at> gmail.com> wrote:
> On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh <ian <at> well-typed.com> wrote:
>> On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:
>>>
>>> If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
>>> longer compiles (due to a version constraint failure) with B-1.0.0.1
>>
>> Won't cabal-install select B-1.0 in that case, so installing C will
>> still work?
>
> See my aside at the end. Yes, it will backtrack and use the older
> version, but the users might get confused why the new version isn't
> used as it's version number suggests that it could be.

I agree with Ian here, we can find the solution ok and we don't need
tighter constraints. Indeed it actually over-tightens, since if B can
work with A-1.0 or 2.0 then we've eliminated a possible solution.

Yes, the solver could and should tell us when it's not selecting the
"best" (usually latest) version (it already does this) and why (it
doesn't do that yet).

Also, I don't think it scales, if we followed it strictly and said
every major bump in any dependent package forces a bump on all things
that depend on it, it'd just be far too much, especially when it's
actually redundant.

But yes it is true that the "real" API of a package is in the version
number of the package itself, and the versions of all the packages
which are "exposed" by that package (meaning packages that define

(Continue reading)

headers

Simon Marlow | 10 Dec 2012 16:49

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On 08/12/12 20:11, Duncan Coutts wrote:
> On 7 December 2012 16:30, Johan Tibell <johan.tibell <at> gmail.com> wrote:
>> On Fri, Dec 7, 2012 at 8:28 AM, Ian Lynagh <ian <at> well-typed.com> wrote:
>>> On Fri, Dec 07, 2012 at 08:20:41AM -0800, Johan Tibell wrote:
>>>>
>>>> If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
>>>> longer compiles (due to a version constraint failure) with B-1.0.0.1
>>>
>>> Won't cabal-install select B-1.0 in that case, so installing C will
>>> still work?
>>
>> See my aside at the end. Yes, it will backtrack and use the older
>> version, but the users might get confused why the new version isn't
>> used as it's version number suggests that it could be.
>
> I agree with Ian here, we can find the solution ok and we don't need
> tighter constraints. Indeed it actually over-tightens, since if B can
> work with A-1.0 or 2.0 then we've eliminated a possible solution.
>
> Yes, the solver could and should tell us when it's not selecting the
> "best" (usually latest) version (it already does this) and why (it
> doesn't do that yet).
>
> Also, I don't think it scales, if we followed it strictly and said
> every major bump in any dependent package forces a bump on all things
> that depend on it, it'd just be far too much, especially when it's
> actually redundant.

I agree with all that.

(Continue reading)

headers

Johan Tibell | 10 Dec 2012 17:02

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Sat, Dec 8, 2012 at 12:11 PM, Duncan Coutts
<duncan.coutts <at> googlemail.com> wrote:
> But yes it is true that the "real" API of a package is in the version
> number of the package itself, and the versions of all the packages
> which are "exposed" by that package (meaning packages that define
> types that are mentioned in the api of the exposing package). But it
> that has the property that I only notice changes in the types in those
> dependencies if I also depend on them directly myself. So we can get
> away with specifying local info, and not have to encode the full
> transative info (I think).

It's worse than that. The failure to build in my example above is a
failure to find a valid install plan due to conflicting dependency
ranges, not due to clashing symbols from different package versions.
It happens even with private dependencies!

-- Johan

headers

Simon Marlow | 11 Dec 2012 10:45

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On 10/12/12 16:02, Johan Tibell wrote:
> On Sat, Dec 8, 2012 at 12:11 PM, Duncan Coutts
> <duncan.coutts <at> googlemail.com> wrote:
>> But yes it is true that the "real" API of a package is in the version
>> number of the package itself, and the versions of all the packages
>> which are "exposed" by that package (meaning packages that define
>> types that are mentioned in the api of the exposing package). But it
>> that has the property that I only notice changes in the types in those
>> dependencies if I also depend on them directly myself. So we can get
>> away with specifying local info, and not have to encode the full
>> transative info (I think).
>
> It's worse than that. The failure to build in my example above is a
> failure to find a valid install plan due to conflicting dependency
> ranges, not due to clashing symbols from different package versions.
> It happens even with private dependencies!

I'm a bit lost.  The example from your mail that started this thread 
*does* compile, because Cabal finds a valid install plan by avoiding the 
new version of B. It's working exactly as intended, isn't it?  Then you 
said that the user might get confused about why the new B wasn't being 
used, to which the response is that we need better diagnostics.

What's the problem you're getting at above?

Cheers,
	Simon

headers

Johan Tibell | 11 Dec 2012 17:14

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Tue, Dec 11, 2012 at 1:45 AM, Simon Marlow <marlowsd <at> gmail.com> wrote:
> I'm a bit lost.  The example from your mail that started this thread *does*
> compile, because Cabal finds a valid install plan by avoiding the new
> version of B. It's working exactly as intended, isn't it?  Then you said
> that the user might get confused about why the new B wasn't being used, to
> which the response is that we need better diagnostics.
>
> What's the problem you're getting at above?

The problem is one of two things, depending on how you look at it. On
the one hand, if we take "C-1.0 depends on A-1.0.0.*" to mean that C
states that it can work with any version of A-1.0.0, that is a lie (as
per the scenario in my original email) and the PVP needs to be
adjusted.

On the other hand, if you take "C-1.0 depends on A-1.0.0.*" to mean
that C states that it can work with the A-1.0.0 API then users and
developers might get confused when newer releases of libraries don't
get picked up even though the version bounds suggests that they would.
We could try to solve this by cabal telling the user about when this
situation appears, but understanding why package version conflicts
appear is already hard to understand for users.

I guess one reason I brought this up is that it hasn't really been
discussed before to my knowledge.

-- Johan

headers

Simon Marlow | 11 Dec 2012 17:34

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On 11/12/12 16:14, Johan Tibell wrote:
> On Tue, Dec 11, 2012 at 1:45 AM, Simon Marlow <marlowsd <at> gmail.com> wrote:
>> I'm a bit lost.  The example from your mail that started this thread *does*
>> compile, because Cabal finds a valid install plan by avoiding the new
>> version of B. It's working exactly as intended, isn't it?  Then you said
>> that the user might get confused about why the new B wasn't being used, to
>> which the response is that we need better diagnostics.
>>
>> What's the problem you're getting at above?
>
> The problem is one of two things, depending on how you look at it. On
> the one hand, if we take "C-1.0 depends on A-1.0.0.*" to mean that C
> states that it can work with any version of A-1.0.0, that is a lie (as
> per the scenario in my original email) and the PVP needs to be
> adjusted.

If users expect this interpretation, then I think we should tackle that 
with better documentation. It's important that the dependencies on a 
package are a *local* statement about the APIs that *this* package 
requires, and don't need to take into account constraints imposed elsewhere.

The consequences of the alternative are pretty severe: every package 
author basically has to do Cabal's job and perform a backtracking 
search, solving the constraints so they can put the results in their 
build-depends.  Furthermore they would have to re-do this every time a 
new constraint is added to the system (somebody uploads a new package 
that might be a dependency of one of yours).

> On the other hand, if you take "C-1.0 depends on A-1.0.0.*" to mean
> that C states that it can work with the A-1.0.0 API then users and

(Continue reading)

headers

Johan Tibell | 11 Dec 2012 17:58

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Tue, Dec 11, 2012 at 8:34 AM, Simon Marlow <marlowsd <at> gmail.com> wrote:
> On 11/12/12 16:14, Johan Tibell wrote:
>> The problem is one of two things, depending on how you look at it. On
>> the one hand, if we take "C-1.0 depends on A-1.0.0.*" to mean that C
>> states that it can work with any version of A-1.0.0, that is a lie (as
>> per the scenario in my original email) and the PVP needs to be
>> adjusted.
>
>
> If users expect this interpretation, then I think we should tackle that with
> better documentation. It's important that the dependencies on a package are
> a *local* statement about the APIs that *this* package requires, and don't
> need to take into account constraints imposed elsewhere.
>
> The consequences of the alternative are pretty severe: every package author
> basically has to do Cabal's job and perform a backtracking search, solving
> the constraints so they can put the results in their build-depends.
> Furthermore they would have to re-do this every time a new constraint is
> added to the system (somebody uploads a new package that might be a
> dependency of one of yours).

It's still a local statement: if we changed the PVP (which I don't
suggest we do at this point), it would be to require that every time
you bump a lower version bound on a dependency you have to bump your
own major version number. You don't have to recursively look at all
your dependencies as we assume they also follow the PVP.

> Right, better diagnostics is the solution.
>
>   I couldn't use B-1.0.0.1 because it depends on A-2.0

(Continue reading)

headers

Andres Löh | 11 Dec 2012 18:19

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

> Better diagnostic is probably they way forward for now. I feel that
> cabal used to be better at telling me why it made certain dependency
> choices. Now it tells me very little (i.e. basically a list of
> packages it tried, without information about why they were tried).

There's a bug in cabal-install-1.16.0 which I fixed in HEAD (I hope)
that caused cabal-install to no longer print the reason why a goal is
tried when -v3 is selected.

One of the problems in printing diagnostics after the fact is that
there's a lot of information during solving, but most of it is
discarded. Currently, most of the info about the actual install plan
is recomputed after the plan has been communicated back by the solver.
This has the advantage of making it easy to still support both
solvers, but it's making it unnecessarily difficult to include
additional info in the final output.

I have quite a number of ideas on how to improve the diagnostics in
principle, but I can currently give no estimate on how long it will
take me in practice to implement any of this.

Cheers,
  Andres

headers

Joachim Breitner | 11 Dec 2012 23:34

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

Hi,

Am Dienstag, den 11.12.2012, 08:14 -0800 schrieb Johan Tibell:
> On Tue, Dec 11, 2012 at 1:45 AM, Simon Marlow <marlowsd <at> gmail.com> wrote:
> > I'm a bit lost.  The example from your mail that started this thread *does*
> > compile, because Cabal finds a valid install plan by avoiding the new
> > version of B. It's working exactly as intended, isn't it?  Then you said
> > that the user might get confused about why the new B wasn't being used, to
> > which the response is that we need better diagnostics.
> >
> > What's the problem you're getting at above?
> 
> The problem is one of two things, depending on how you look at it. On
> the one hand, if we take "C-1.0 depends on A-1.0.0.*" to mean that C
> states that it can work with any version of A-1.0.0, that is a lie (as
> per the scenario in my original email) and the PVP needs to be
> adjusted.

I don’t think this is a good semantics for the dependency. I always
understand package dependencies as: If you find a set of packaging such
that all constraints are fulfilled, then things will build correctly.

What if A-1.0.0.1 would fail for other reasons, besides a bumped
dependency? E.g. A decides to implement an internal parser with happy,
and adds that to the list of tools required. Now people who could build
C before, but don’t have happy installed, cannot build C. But should
that really require A to bump a major version?

> On the other hand, if you take "C-1.0 depends on A-1.0.0.*" to mean
> that C states that it can work with the A-1.0.0 API then users and

(Continue reading)

headers

Michael Snoyman | 8 Dec 2012 20:48

Re: The PVP ought to mention dependency bounds (Was: Growing Haskell Platform)

On Fri, Dec 7, 2012 at 6:20 PM, Johan Tibell <johan.tibell <at> gmail.com> wrote:

On Thu, Dec 6, 2012 at 11:52 PM, Michael Snoyman <michael <at> snoyman.com> wrote:
> I thought the natural response to this would be to put a lower bound on
> text. After all, following the PVP, there should be no problem with updating
> a minor version number. However, when I did so, many users of the Haskell
> Platform were no longer able to compile due to conflicting dependencies. The
> result: I had to put in some dirty conditional compilation tricks in
> persistent to avoid inlining `pack` when using older versions of text.

I've meant to write about this but haven't yet found time to do so.

Assuming that the goal of the PVP is to make sure that a packages that
previously compiled continues to compile, dependencies must be taken
into account when computing package version number bumps. For example,
given these packages:

A-1.0
B-1.0 depends on A-1.0.0.*
C-1.0 depends on A-1.0.0.* and B-1.0.0.*

Now assume that A-2.0 is released and B starts depending on the new
version of A (without changing its own API). He now have this
situation:

A-1.0
A-2.0
B-1.?.?.? depends on A-2.0.0.*
C-1.0 depends on A-1.0.0.* and B-1.0.0.*

If B only bumps its patch-level version (i.e. to B-1.0.0.1), C no
longer compiles (due to a version constraint failure) with B-1.0.0.1
even though C's dependency on B (i.e. B-1.0.0.*) suggests that it
would. If B updates it's lower bound on a dependency it must bump its
own major version number.

We can summarize this under the slogan "versions are APIs"; the
versions you depend on are part of your API.

Aside: Even if cabal install would backtrack and *not* use the new
version of B (i.e. B-1.0.0.1) and thus be able to compile C-1.0 using
the old version of B things would still be bad as C's dependency
constraints would be a lie.

I think the PVP policy must be changed to require that increasing
lower bounds on dependencies requires an increase in the major version
number.

-- Johan

+1. I already try and follow this policy myself, I've seen the lack of clarity on it lead to versioning problems in the past. If anyone is interested, I can try to dredge up the actual incidents.

Michael

_______________________________________________
Libraries mailing list
Libraries <at> haskell.org
http://www.haskell.org/mailman/listinfo/libraries