Thomas Koch | 10 Feb 14:49 2011
Picon

Dependence on specific versions

Hi Stefane,

in another thread you write:

> > and that it is possible for projects to accumulate
> > technical debt by depending on strict version numbers.
> 
> OTOH, this is a HUGE source of instability and hard to debug bugs (as we've 
found to our expense) to depend on loose version numbers.

I totaly disagree with this. One day or another you need to update to newer 
versions of your dependencies. If you're specifying strict version numbers 
then you're just procrasinating the update.

Instead, you (as in all java projects) should include the expected behaviour 
of the dependencies in your test cases. How should you update them otherwise?

If your test cases work with any versions of the dependencies provided, then 
your product should work.

You may say that "business realities" are different. But then you tell me that 
business software just means: Software that happens to work today, but who 
knows about tomorrow?

Or am I totally wrong?

Best regards,

Thomas Koch, http://www.koch.ro

(Continue reading)

Stefane Fermigier | 10 Feb 15:25 2011

Re: Dependence on specific versions


On Feb 10, 2011, at 2:49 PM, Thomas Koch wrote:

> Hi Stefane,
> 
> in another thread you write:
> 
>>> and that it is possible for projects to accumulate
>>> technical debt by depending on strict version numbers.
>> 
>> OTOH, this is a HUGE source of instability and hard to debug bugs (as we've 
> found to our expense) to depend on loose version numbers.
> 
> I totaly disagree with this. One day or another you need to update to newer 
> versions of your dependencies. If you're specifying strict version numbers 
> then you're just procrasinating the update.
> 
> Instead, you (as in all java projects) should include the expected behaviour 
> of the dependencies in your test cases. How should you update them otherwise?
> 
> If your test cases work with any versions of the dependencies provided, then 
> your product should work.
> 
> You may say that "business realities" are different. But then you tell me that 
> business software just means: Software that happens to work today, but who 
> knows about tomorrow?
> 
> Or am I totally wrong?

Definitevely.
(Continue reading)

Russ Allbery | 10 Feb 19:50 2011
Picon

Re: Dependence on specific versions

Stefane Fermigier <sf <at> nuxeo.com> writes:

> Only by fixing version numbers of third-party libraries can you be sure
> that the same build that works today will still work next week, if you
> redo the build on the exact same version of the sources (and Maven, and
> Java, of course), any operating system.

> Yes, we do upgrade third-party lib versions from time to time, but only
> when there is a good reason to ("if it ain't broke, don't fix it").

> BTW: I used to think like you 3-4 years ago when I discovered Maven, but
> had to change my mind due to the reality.

For those of us who have been doing this sort of thing for a while, this
argument sounds very familiar.  I've heard this argument applied to C
libraries, Perl modules, Python modules, and most recently Ruby modules.
It always sounds persuasive when presented from a stability perspective.
It has always proven to be completely wrong in the long run.

--

-- 
Russ Allbery (rra <at> debian.org)               <http://www.eyrie.org/~eagle/>

Stefane Fermigier | 10 Feb 21:29 2011

Re: Dependence on specific versions


On Feb 10, 2011, at 7:50 PM, Russ Allbery wrote:

> Stefane Fermigier <sf <at> nuxeo.com> writes:
> 
>> Only by fixing version numbers of third-party libraries can you be sure
>> that the same build that works today will still work next week, if you
>> redo the build on the exact same version of the sources (and Maven, and
>> Java, of course), any operating system.
> 
>> Yes, we do upgrade third-party lib versions from time to time, but only
>> when there is a good reason to ("if it ain't broke, don't fix it").
> 
>> BTW: I used to think like you 3-4 years ago when I discovered Maven, but
>> had to change my mind due to the reality.
> 
> For those of us who have been doing this sort of thing for a while, this
> argument sounds very familiar.  I've heard this argument applied to C
> libraries, Perl modules, Python modules, and most recently Ruby modules.
> It always sounds persuasive when presented from a stability perspective.
> It has always proven to be completely wrong in the long run.

Please develop, unless you want me to believe you only based on your reputation, which I won't since I don't
know you.

  S.

--

-- 
Stefane Fermigier, Founder and Chairman, Nuxeo
Open Source, Java EE based, Enterprise Content Management (ECM)
(Continue reading)

Jesús M. Navarro | 10 Feb 22:40 2011
Picon

Re: Dependence on specific versions

Hi, Stefane:

On Thursday 10 February 2011 21:29:34 Stefane Fermigier wrote:
> On Feb 10, 2011, at 7:50 PM, Russ Allbery wrote:
> > Stefane Fermigier <sf <at> nuxeo.com> writes:
> >> Only by fixing version numbers of third-party libraries can you be sure
> >> that the same build that works today will still work next week, if you
> >> redo the build on the exact same version of the sources (and Maven, and
> >> Java, of course), any operating system.
> >>
> >> Yes, we do upgrade third-party lib versions from time to time, but only
> >> when there is a good reason to ("if it ain't broke, don't fix it").
> >>
> >> BTW: I used to think like you 3-4 years ago when I discovered Maven, but
> >> had to change my mind due to the reality.
> >
> > For those of us who have been doing this sort of thing for a while, this
> > argument sounds very familiar.  I've heard this argument applied to C
> > libraries, Perl modules, Python modules, and most recently Ruby modules.
> > It always sounds persuasive when presented from a stability perspective.
> > It has always proven to be completely wrong in the long run.
>
> Please develop, unless you want me to believe you only based on your
> reputation, which I won't since I don't know you.

I'll give my opinion here: both Russ and you are right.

Yes, you are right: in order to distribute a product you must have to be able 
to reproduce it unambiguosly from your sources.  This implies all your 
dependencies must be hardcoded.
(Continue reading)

Russ Allbery | 10 Feb 22:56 2011
Picon

Re: Dependence on specific versions

"Jesús M. Navarro" <jesus.navarro <at> undominio.net> writes:

> If you don't do that already is because you know that your app will fail
> day-in day-out because foo developer (and bar, and zoot...) is not
> sensible enough not to break backwards compatibility between foo_1.2.3
> and foo_1.2.4.  Russ is right in that when things go that way, things
> are completly wrong in the long run in so many manners it's not even
> funny.  The fact this behaviour is "business as usual" on Java
> development it's only a measure of its average quality.

I also feel like I should say here that many Java developers *do* do this
properly.  I've moved Java projects to new versions of common Java
libraries like Spring or various Apache infrastructure libraries and had
them just work.  Often even across major releases, but very commonly with
minor releases.  I don't think the current state of the art is actually as
dire as all that.  Java offers very good mechanisms for information hiding
and minimizing the exposed API so that libraries can adjust for
compatibility issues, and good Java programmers use those facilities.

> Oh! and another hint: "if ain't broken don't fix it" is not such a
> valuable knowledge with regards to open source if only because of the
> outer testing base and the work it takes to jump over more than a few
> releases (exponential, not linear, to the distance).

Right, what you end up with is five or six or more copies of the same
library at different versions, frozen there because no one dares to try to
update and test, and then when that library has a security vulnerability,
you have to backport the fix to all these slightly different versions.
It's a nightmare.

(Continue reading)

Stefane Fermigier | 10 Feb 23:09 2011

Re: Dependence on specific versions


On Feb 10, 2011, at 10:40 PM, Jesús M. Navarro wrote:

> Hi, Stefane:
> 
> On Thursday 10 February 2011 21:29:34 Stefane Fermigier wrote:
>> On Feb 10, 2011, at 7:50 PM, Russ Allbery wrote:
>>> Stefane Fermigier <sf <at> nuxeo.com> writes:
>>>> Only by fixing version numbers of third-party libraries can you be sure
>>>> that the same build that works today will still work next week, if you
>>>> redo the build on the exact same version of the sources (and Maven, and
>>>> Java, of course), any operating system.
>>>> 
>>>> Yes, we do upgrade third-party lib versions from time to time, but only
>>>> when there is a good reason to ("if it ain't broke, don't fix it").
>>>> 
>>>> BTW: I used to think like you 3-4 years ago when I discovered Maven, but
>>>> had to change my mind due to the reality.
>>> 
>>> For those of us who have been doing this sort of thing for a while, this
>>> argument sounds very familiar.  I've heard this argument applied to C
>>> libraries, Perl modules, Python modules, and most recently Ruby modules.
>>> It always sounds persuasive when presented from a stability perspective.
>>> It has always proven to be completely wrong in the long run.
>> 
>> Please develop, unless you want me to believe you only based on your
>> reputation, which I won't since I don't know you.
> 
> I'll give my opinion here: both Russ and you are right.
> 
(Continue reading)

Russ Allbery | 10 Feb 22:52 2011
Picon

Re: Dependence on specific versions

Stefane Fermigier <sf <at> nuxeo.com> writes:
> On Feb 10, 2011, at 7:50 PM, Russ Allbery wrote:

>> For those of us who have been doing this sort of thing for a while,
>> this argument sounds very familiar.  I've heard this argument applied
>> to C libraries, Perl modules, Python modules, and most recently Ruby
>> modules.  It always sounds persuasive when presented from a stability
>> perspective.  It has always proven to be completely wrong in the long
>> run.

> Please develop, unless you want me to believe you only based on your
> reputation, which I won't since I don't know you.

You're advocating for a (very significant) change in Debian Policy, and I
think the burden of proof is on showing why Java libraries should be
treated differently than every other library in Debian.  They pose the
same challenges for security support, for ensuring rebuildability of
software, and for keeping the whole distribution moving forward in an
integrated fashion, and therefore the same issues apply.

So, I think you have the burden of proof reveresed.  I think you have to
convince Debian that Java needs to be a weird special case, rather than
asking me to convince you that Debian's policy for every other language
should also apply to Java.

You can look at the infrastructure and handling of libraries in every
other language in Debian, and observe that they all follow the system that
we're asking Java libraries to follow as well.  Java isn't being singled
out here.  Historically, every one of those languages have presented the
same argument that you're presenting.  For example, including convenience
(Continue reading)

Torsten Werner | 10 Feb 21:38 2011
Picon

Re: Dependence on specific versions

Hi Stefane,

On Thu, Feb 10, 2011 at 3:25 PM, Stefane Fermigier <sf <at> nuxeo.com> wrote:
> Only by fixing version numbers of third-party libraries can you be sure that the same build that works
today will still work next week, if you redo the build on the exact same version of the sources (and Maven,
and Java, of course), any operating system.

that sounds good but at least Maven does not really support fixed
dependencies. Example:

a.jar (0.1) depends on b.jar (0.1)
c.jar (0.3) depends on b.jar (0.2)
d.jar (0.4) depends on a.jar (0.1) and c.jar (0.3)

What version of b.jar will be chosen by Maven? 0.1 or 0.2? You cannot
predict that. Neither a.jar nor c.jar can rely on getting the version
they want.

That is why the concept of fixed version dependencies is fully broken, sorry.

Cheers,
Torsten

Stefane Fermigier | 10 Feb 21:47 2011

Re: Dependence on specific versions


On Feb 10, 2011, at 9:38 PM, Torsten Werner wrote:

> Hi Stefane,
> 
> 
> On Thu, Feb 10, 2011 at 3:25 PM, Stefane Fermigier <sf <at> nuxeo.com> wrote:
>> Only by fixing version numbers of third-party libraries can you be sure that the same build that works
today will still work next week, if you redo the build on the exact same version of the sources (and Maven,
and Java, of course), any operating system.
> 
> that sounds good but at least Maven does not really support fixed
> dependencies. Example:
> 
> a.jar (0.1) depends on b.jar (0.1)
> c.jar (0.3) depends on b.jar (0.2)
> d.jar (0.4) depends on a.jar (0.1) and c.jar (0.3)
> 
> What version of b.jar will be chosen by Maven? 0.1 or 0.2? You cannot
> predict that. Neither a.jar nor c.jar can rely on getting the version
> they want.
> 
> That is why the concept of fixed version dependencies is fully broken, sorry.

A lot of things are wrong in Maven, but it this case, you just ask maven to use a fixed version of the dependency
in the dependencyManagement section of your POM, and voila.

See our master POM for examples: http://hg.nuxeo.org/nuxeo/file/20953aeee544/pom.xml

  S.
(Continue reading)

Torsten Werner | 10 Feb 22:02 2011
Picon

Re: Dependence on specific versions

On Thu, Feb 10, 2011 at 9:47 PM, Stefane Fermigier <sf <at> nuxeo.com> wrote:
> On Feb 10, 2011, at 9:38 PM, Torsten Werner wrote:
>> a.jar (0.1) depends on b.jar (0.1)
>> c.jar (0.3) depends on b.jar (0.2)
>> d.jar (0.4) depends on a.jar (0.1) and c.jar (0.3)
>
> A lot of things are wrong in Maven, but it this case, you just ask maven to use a fixed version of the
dependency in the dependencyManagement section of your POM, and voila.

No, if you choose b.jar (0.1) then c.jar (0.3) does not git its
preferred version. If you choose b.jar (0.2) then a.jar (0.1) does not
get its preferred version. That means either a.jar or c.jar will get a
non preferred version. What is the point of specifying fixed versions
if you cannot rely on getting them?

Torsten

Stefane Fermigier | 10 Feb 22:10 2011

Re: Dependence on specific versions


On Feb 10, 2011, at 10:02 PM, Torsten Werner wrote:

> On Thu, Feb 10, 2011 at 9:47 PM, Stefane Fermigier <sf <at> nuxeo.com> wrote:
>> On Feb 10, 2011, at 9:38 PM, Torsten Werner wrote:
>>> a.jar (0.1) depends on b.jar (0.1)
>>> c.jar (0.3) depends on b.jar (0.2)
>>> d.jar (0.4) depends on a.jar (0.1) and c.jar (0.3)
>> 
>> A lot of things are wrong in Maven, but it this case, you just ask maven to use a fixed version of the
dependency in the dependencyManagement section of your POM, and voila.
> 
> No, if you choose b.jar (0.1) then c.jar (0.3) does not git its
> preferred version. If you choose b.jar (0.2) then a.jar (0.1) does not
> get its preferred version. That means either a.jar or c.jar will get a
> non preferred version. What is the point of specifying fixed versions
> if you cannot rely on getting them?

As the developer of d.jar, I get to choose which version of a.jar, b.jar and c.jar I will assemble in my application.

When a situation such as the one you describe occurs, I have to make a choice, even if it's not the one
advertised by the dependencies authors.

Then it's my responsibility to test the hell out of the application and make sure that it's working.

Do not confuse application development and library development, BTW.

  S.

--

-- 
(Continue reading)

Thomas Zeeman | 10 Feb 22:23 2011
Picon
Picon

Re: Dependence on specific versions


On Feb 10, 2011, at 9:38 PM, Torsten Werner wrote:

> Hi Stefane,
> 
> 
> On Thu, Feb 10, 2011 at 3:25 PM, Stefane Fermigier <sf <at> nuxeo.com> wrote:
>> Only by fixing version numbers of third-party libraries can you be sure that the same build that works
today will still work next week, if you redo the build on the exact same version of the sources (and Maven,
and Java, of course), any operating system.
> 
> that sounds good but at least Maven does not really support fixed
> dependencies. Example:
> 
> a.jar (0.1) depends on b.jar (0.1)
> c.jar (0.3) depends on b.jar (0.2)
> d.jar (0.4) depends on a.jar (0.1) and c.jar (0.3)
> 
> What version of b.jar will be chosen by Maven? 0.1 or 0.2? You cannot
> predict that. Neither a.jar nor c.jar can rely on getting the version
> they want.
> 

Actually, you can predict it. Given the above order and no outside influences like a dependencyManagement
block in your (parent) pom,  it would be 0.1. See
http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html for a
more detailed explanation on why this is and what surprising results the dependency mediation can have.

If a and c depend on the same part of the API of b and it has changed from .1 to .2 you will be in for either some
spectacular failure at run time or some very subtle and hard to notice error in your application behaviour.
(Continue reading)

Torsten Werner | 11 Feb 07:19 2011
Picon

Re: Dependence on specific versions

Hi Thomas,

On Thu, Feb 10, 2011 at 10:23 PM, Thomas Zeeman <tjzeeman <at> xs4all.nl> wrote:
> Actually, you can predict it. Given the above order and no outside influences like a
dependencyManagement block in your (parent) pom,  it would be 0.1. See
http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html for a
more detailed explanation on why this is and what surprising results the dependency mediation can have.

what I wanted to say: you cannot predict that for any non trivial real
work project. Try to look at the dependency chain of maven itself. But
not for too long because of the danger of getting insane. :)

> If a and c depend on the same part of the API of b and it has changed from .1 to .2 you will be in for either some
spectacular failure at run time or some very subtle and hard to notice error in your application behaviour.

Yes, runtime is another step where the whole thing collapses. That is
probably why integration tests are more important in the Java world
[*] - the code uses different dependencies during unittest time
(provided by Maven) and after integration (something else).

Torsten

[*] I must confess that I have no hard scientific evidence for my claim.

Thomas J. Zeeman | 11 Feb 10:35 2011
Picon
Picon

Re: Dependence on specific versions

 On Fri, 11 Feb 2011 07:19:33 +0100, Torsten Werner wrote:
> Hi Thomas,
>
> On Thu, Feb 10, 2011 at 10:23 PM, Thomas Zeeman <tjzeeman <at> xs4all.nl> 
> wrote:
>> Actually, you can predict it. Given the above order and no outside 
>> influences like a dependencyManagement block in your (parent) pom,  it 
>> would be 0.1. See 
>> http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html 
>> for a more detailed explanation on why this is and what surprising 
>> results the dependency mediation can have.
>
> what I wanted to say: you cannot predict that for any non trivial 
> real
> work project.

 I think you can. Given that maven creates its dependency graph as a 
 tree where any transitive dependency which is also declared at the top 
 level is cut out at the transitive level and it prioritizes transitive 
 dependencies on depth in the tree first and on order of declaration in 
 the pom second. Granted. It is not going to be easy on larger projects, 
 but with enough patience and a large enough screen to show the entire 
 tree at once its doable. And yes, I've had to do just that one more 
 occasions than I would've liked already.

>  Try to look at the dependency chain of maven itself. But
> not for too long because of the danger of getting insane. :)

 Too late. ;) I've had to do just that for a project where I had to 
 mavenise a largish (~200 jars in the largest war, 300 different 
(Continue reading)


Gmane