Ian Lynagh | 27 Nov 15:52 2012

Dynamic libraries by default and GHC 7.8


Hi all,

GHC HEAD now has support for using dynamic libraries by default (and in
particular, using dynamic libraries and the system linker in GHCi) for a
number of platforms.

This has some advantages and some disadvantages, so we need to make a
decision about what we want to do in GHC 7.8. There are also some policy
questions we need to answer about how Cabal will work with a GHC that
uses dynamic libraries by default. We would like to make these as soon
as possible, so that GHC 7.6.2 can ship with a Cabal that works
correctly.

The various issues are described in a wiki page here:
    http://hackage.haskell.org/trac/ghc/wiki/DynamicByDefault

If you have a few minutes to read it then we'd be glad to hear your
feedback, to help us in making our decisions

Thanks
Ian
--

-- 
Ian Lynagh, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/
Joachim Breitner | 28 Nov 00:28 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

Hi,

Am Dienstag, den 27.11.2012, 14:52 +0000 schrieb Ian Lynagh:
> The various issues are described in a wiki page here:
>     http://hackage.haskell.org/trac/ghc/wiki/DynamicByDefault
> 
> If you have a few minutes to read it then we'd be glad to hear your
> feedback, to help us in making our decisions

here comes the obligatory butting in by the Debian Haskell Group:

Given the current sensitivity of the ABI hashes we really do not want to
have Programs written in Haskell have a runtime dependency on all the
included Haskell libraries. So I believe we should still link Haskell
programs statically in Debian.

Hence, Debian will continue to provide its libraries built the static
way.

Building them also in the dynamic way for the sake of GHCi users seems
possible.

Open question: What should GHC on Debian do when building binaries,
given that all libraries are likely available in both ways – shared or
static. Shared means that all locally built binaries (e.g. xmonad!) will
suddenly break when the user upgrades its Haskell packages, as the
package management is ignorant of unpackaged, locally built programs.
I’d feel more comfortable if that could not happen.

Other open question: Should we put the dynamic libraries in the normal
(Continue reading)

Tyson Whitehead | 28 Nov 03:57 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

On Wed, 2012-11-28 at 00:28 +0100, Joachim Breitner wrote:
> Am Dienstag, den 27.11.2012, 14:52 +0000 schrieb Ian Lynagh:
> > The various issues are described in a wiki page here:
> >     http://hackage.haskell.org/trac/ghc/wiki/DynamicByDefault
> > 
> > If you have a few minutes to read it then we'd be glad to hear your
> > feedback, to help us in making our decisions
> 
> here comes the obligatory butting in by the Debian Haskell Group:
> 
> Given the current sensitivity of the ABI hashes we really do not want to
> have Programs written in Haskell have a runtime dependency on all the
> included Haskell libraries. So I believe we should still link Haskell
> programs statically in Debian.
>
> Hence, Debian will continue to provide its libraries built the static
> way.

I was so excited for a bit thinking that this would finally mean that
Debian would move to a dynamic system.  Every haskell binary being 10s
of MBs (e.g., pandoc = 25MB executable) makes it look kind of bad.

From our previous discussion on this

http://lists.debian.org/debian-haskell/2010/03/msg00120.html

it seemed a big part of the problem is you really need to be able to
have multiple versions of the same package installed.

At the time, I believe the only way to do this was to include part of
(Continue reading)

Brandon Allbery | 28 Nov 05:39 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

On Tue, Nov 27, 2012 at 9:57 PM, Tyson Whitehead <twhitehead <at> gmail.com> wrote:
it seemed a big part of the problem is you really need to be able to
have multiple versions of the same package installed.

At the time, I believe the only way to do this was to include part of
the hash in the name, but that meant you had to constantly be creating
new packages which Debian didn't make easy.

Maybe it is time to explain the problem to the Debian packaging people
and see if they have any ideas from their level?

Not sure they do, because it's a GHC-specific and very nasty problem at core.

The fundamental issue is that GHC does aggressive cross-module inlining to make performance halfway reasonable -- but this means you end up requiring very precise dependencies, because different builds of the same library even with the same compiler might for some reason expose different things for inlining, and that ends up infecting the whole dependency chain.  Done statically, the result is the infamous Cabal hell; done dynamically, it means shared library hell.  I am not sure it can be made sane in either case.

--
brandon s allbery kf8nh                               sine nomine associates
allbery.b <at> gmail.com                                  ballbery <at> sinenomine.net
unix, openafs, kerberos, infrastructure, xmonad        http://sinenomine.net

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users <at> haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Joachim Breitner | 28 Nov 10:45 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

Hi,

Am Dienstag, den 27.11.2012, 21:57 -0500 schrieb Tyson Whitehead:
> I was so excited for a bit thinking that this would finally mean that
> Debian would move to a dynamic system.  Every haskell binary being 10s
> of MBs (e.g., pandoc = 25MB executable) makes it look kind of bad.

its not like dynamic libraries make the bytes disappear – the
(non-Haskell-developer) user who wants to use pandoc still has to
install all these bytes, but now they just come split in a dozen of
packages.

Or gix-annex, a more and more popular Haskell application: Building it
requires 94 Haskell library packages. Now imagine this to be dynamically
built: Now installing git-annex will require 94 strage sounding packages
that the user most likely has no idea what they are about, and chances
are high that there is no other packages requiring these shared
libraries, making most of the benefit of shared libraries moot.

Now, if Haskell was as popular as C and the user _would_ run several
different processes at once that could share the shared library, this
would be interesting. At the moment, I do not see how dynamically built
Haskell programs are in the interest of our user.

> I was left with the impression that we were going to have this back in
> 2010 just as soon as squeeze got out the door...  :)

It seems that noone cared enough about that, but any help is welcome.
Two things to do: Patch haskell-devscripts to build the -dyn ways, and
manually adding the additional package stance to the debian/contol files
(if it is to be decided that the -dyn libraries should reside in
packages of their own. If we decide to include them in the regular
packages, this is not needed.)

Greetings,
Joachim

--

-- 
Joachim "nomeata" Breitner
Debian Developer
  nomeata <at> debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: nomeata <at> joachim-breitner.de | http://people.debian.org/~nomeata
Tyson Whitehead | 28 Nov 17:35 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

On November 28, 2012 04:45:57 Joachim Breitner wrote:
> Am Dienstag, den 27.11.2012, 21:57 -0500 schrieb Tyson Whitehead:
> > I was so excited for a bit thinking that this would finally mean that
> > Debian would move to a dynamic system.  Every haskell binary being 10s
> > of MBs (e.g., pandoc = 25MB executable) makes it look kind of bad.
> 
> its not like dynamic libraries make the bytes disappear – the
> (non-Haskell-developer) user who wants to use pandoc still has to
> install all these bytes, but now they just come split in a dozen of
> packages.

My point was more trying to get at the idea that maybe we don't need a 
separate copy of most of the bytes in each application.

> Or gix-annex, a more and more popular Haskell application: Building it
> requires 94 Haskell library packages. Now imagine this to be dynamically
> built: Now installing git-annex will require 94 strage sounding packages
> that the user most likely has no idea what they are about, and chances
> are high that there is no other packages requiring these shared
> libraries, making most of the benefit of shared libraries moot.
> 
> Now, if Haskell was as popular as C and the user _would_ run several
> different processes at once that could share the shared library, this
> would be interesting. At the moment, I do not see how dynamically built
> Haskell programs are in the interest of our user.

I guess this is really a question of how many haskell programs are there being 
used out there.  From the looks of popcon results, there isn't a whole lot of 
take up on anything at moment apart from ghc, xmond, and pandoc.

> > I was left with the impression that we were going to have this back in
> > 2010 just as soon as squeeze got out the door...  :)
> 
> It seems that noone cared enough about that, but any help is welcome.
> Two things to do: Patch haskell-devscripts to build the -dyn ways, and
> manually adding the additional package stance to the debian/contol files
> (if it is to be decided that the -dyn libraries should reside in
> packages of their own. If we decide to include them in the regular
> packages, this is not needed.)

Fair enough.

If I was update my 2010 patch so it worked again at some point in the upcoming 
year (I don't have the time to do this at the moment), would there be a 
reasonable chance it would seem worthwhile to include it at this point?

Please feel free to say no here if that is the case.  I realize that maybe in 
a few years, when there are even more haskell applications, we can revisit the 
again, and possibly then it will make more sense.

Cheers!  -Tyson

PS:  I don't mean to be critical here.  You've done a lot of work supporting 
haskell under Debian, and it's all volunteer.  I really appreciate that.

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users <at> haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Joachim Breitner | 28 Nov 18:30 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

Hi,

Am Mittwoch, den 28.11.2012, 11:35 -0500 schrieb Tyson Whitehead:
> > > I was left with the impression that we were going to have this back in
> > > 2010 just as soon as squeeze got out the door...  :)
> > 
> > It seems that noone cared enough about that, but any help is welcome.
> > Two things to do: Patch haskell-devscripts to build the -dyn ways, and
> > manually adding the additional package stance to the debian/contol files
> > (if it is to be decided that the -dyn libraries should reside in
> > packages of their own. If we decide to include them in the regular
> > packages, this is not needed.)
> 
> Fair enough.
> 
> If I was update my 2010 patch so it worked again at some point in the upcoming 
> year (I don't have the time to do this at the moment), would there be a 
> reasonable chance it would seem worthwhile to include it at this point?
> 
> Please feel free to say no here if that is the case.  I realize that maybe in 
> a few years, when there are even more haskell applications, we can revisit the 
> again, and possibly then it will make more sense.

there was even a patch? Sorry then, this is not how contributors should
be treated.

Given that dynamic linking is becoming the default elsewhere, we should
definitely provide it with the next release. So yes, please do look
again at the issue.

We currently have one package providing dynamic libraries, and that is
ghc-dyn. It contains lots of .dyn_hi and some .so files. Which of these
are required at runtime? What is the space ratio between the .dyn_hi
(which presumably are only required at build time) and the .so files? Do
we want separate packages that contain only the files required at
runtime? What should be the filename (just libghc-foo?)? Where do
the .dyn_hi files go? If a user only build dynamically (only staticaly,
both), what packages does she needs?

Make liberal use of package-version-hash-base virtual package names
(e.g. one set of names for the build time requirements with dynamic
packages and one set of names for whats required at runtime) to ensure
that we can revise these decisions without breaking everything.

But this is getting too specific for glasgow-haskell-users, I guess.
replies on this issue please only to d-haskell.

Greetings,
Joachim

--

-- 
Joachim "nomeata" Breitner
Debian Developer
  nomeata <at> debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: nomeata <at> joachim-breitner.de | http://people.debian.org/~nomeata
Tyson Whitehead | 28 Nov 19:43 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

On November 28, 2012 12:30:52 Joachim Breitner wrote:
> Am Mittwoch, den 28.11.2012, 11:35 -0500 schrieb Tyson Whitehead:
> > If I was update my 2010 patch so it worked again at some point in the
> > upcoming year (I don't have the time to do this at the moment), would
> > there be a reasonable chance it would seem worthwhile to include it at
> > this point?
> > 
> > Please feel free to say no here if that is the case.  I realize that
> > maybe in a few years, when there are even more haskell applications, we
> > can revisit the again, and possibly then it will make more sense.
> 
> there was even a patch? Sorry then, this is not how contributors should
> be treated.

Hey, no problem.  Like I said, I'm just glad we got some people interested in 
making Haskell a success under Debian.  Thanks for all your work.

The patch was attached to my original email.  It was built on top of the 0.7.5 
release of the haskell-devscripts.

http://lists.debian.org/debian-haskell/2010/03/msg00120.html

> Given that dynamic linking is becoming the default elsewhere, we should
> definitely provide it with the next release. So yes, please do look
> again at the issue.

Hopefully I'll have sometime in the new year to dust it back off again.

It won't be right away though, so, if someone else wants to take a stab at it, 
please do go ahead.

WRT the rest of your points/questions, I haven't looked at it since I wrote 
the patch, so I have to plead ignorant.

Cheers!  -Tyson

--

-- 
To UNSUBSCRIBE, email to debian-haskell-request <at> lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster <at> lists.debian.org
Archive: http://lists.debian.org/201211281343.40992.twhitehead <at> gmail.com

Bryan O'Sullivan | 30 Nov 18:07 2012

Re: Dynamic libraries by default and GHC 7.8

On Wed, Nov 28, 2012 at 1:45 AM, Joachim Breitner <nomeata <at> debian.org> wrote:

At the moment, I do not see how dynamically built
Haskell programs are in the interest of our user.

They do offer the prospect of fixing some annoying bugs for free, by offloading them to existing, working system infrastructure. For instance, we can't call C++ code from ghci right now, because ghci's loader doesn't invoke static initializers that C++ libraries tend to use heavily.

The "save some bytes" argument seems like a red herring, but it's far from the only thing to pay attention to.
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users <at> haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Jens Petersen | 28 Nov 08:14 2012

Re: Dynamic libraries by default and GHC 7.8

On 28 November 2012 08:28, Joachim Breitner <nomeata <at> debian.org> wrote:
> Open question: What should GHC on Debian do when building binaries,
> given that all libraries are likely available in both ways – shared or
> static. Shared means that all locally built binaries (e.g. xmonad!) will
> suddenly break when the user upgrades its Haskell packages, as the
> package management is ignorant of unpackaged, locally built programs.
> I’d feel more comfortable if that could not happen.

Right, I tried patching Fedora's xmonad for a while to use dynamic linking
(it made Mod-q almost instant! :-) but finally reverted it not to confuse people
linking their .xmonad/ to user libraries, at least until the time
ghc/Cabal support dyn by default...

Jens

--

-- 
To UNSUBSCRIBE, email to debian-haskell-request <at> lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster <at> lists.debian.org
Archive: http://lists.debian.org/CAEDGBq+CMkeyDJ_RXSUuwdxc2Q+NsdAumjF4UZGRM=+MjbC5gA <at> mail.gmail.com

Joachim Breitner | 28 Nov 10:37 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

Am Mittwoch, den 28.11.2012, 16:14 +0900 schrieb Jens Petersen:
> On 28 November 2012 08:28, Joachim Breitner <nomeata <at> debian.org> wrote:
> > Open question: What should GHC on Debian do when building binaries,
> > given that all libraries are likely available in both ways – shared or
> > static. Shared means that all locally built binaries (e.g. xmonad!) will
> > suddenly break when the user upgrades its Haskell packages, as the
> > package management is ignorant of unpackaged, locally built programs.
> > I’d feel more comfortable if that could not happen.
> 
> Right, I tried patching Fedora's xmonad for a while to use dynamic linking
> (it made Mod-q almost instant! :-) but finally reverted it not to confuse people
> linking their .xmonad/ to user libraries, at least until the time
> ghc/Cabal support dyn by default...

but would that not mean that when they upgrade a library used by xmonad,
their xmonad binary would stop working and they could not even log in
any more? It is that kind of breakage that is worrying me.

Greetings,
Joachim

--

-- 
Joachim "nomeata" Breitner
Debian Developer
  nomeata <at> debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: nomeata <at> joachim-breitner.de | http://people.debian.org/~nomeata
Simon Marlow | 28 Nov 10:09 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

On 27/11/12 23:28, Joachim Breitner wrote:
> Hi,
>
> Am Dienstag, den 27.11.2012, 14:52 +0000 schrieb Ian Lynagh:
>> The various issues are described in a wiki page here:
>>      http://hackage.haskell.org/trac/ghc/wiki/DynamicByDefault
>>
>> If you have a few minutes to read it then we'd be glad to hear your
>> feedback, to help us in making our decisions
>
> here comes the obligatory butting in by the Debian Haskell Group:
>
> Given the current sensitivity of the ABI hashes we really do not want to
> have Programs written in Haskell have a runtime dependency on all the
> included Haskell libraries. So I believe we should still link Haskell
> programs statically in Debian.
>
> Hence, Debian will continue to provide its libraries built the static
> way.
>
> Building them also in the dynamic way for the sake of GHCi users seems
> possible.

So let me try to articulate the options, because I think there are some 
dependencies that aren't obvious here.  It's not a straightforward 
choice between -dynamic/-static being the default, because of the GHCi 
interaction.

Here are the 3 options:

(1) (the current situation) GHCi is statically linked, and -static is
     the default.  Uses the RTS linker.

(2) (the proposal, at least for some platforms) GHCi is dynamically
     linked, and -dynamic is the default.  Does not use the RTS linker.

(3) GHCi is dynamically linked, but -static is the default.  Does not
     use the RTS linker.  Packages must be installed with -dynamic,
     otherwise they cannot be loaded into GHCi, and only objects
     compiled with -dynamic can be loaded into GHCi.

You seem to be saying that Debian would do (3), but we hadn't considered 
that as a viable option because of the extra hoops that GHCi users would 
have to jump through.  We consider it a prerequisite that GHCi continues 
to work without requiring any extra flags.

Cheers,
	Simon

> Open question: What should GHC on Debian do when building binaries,
> given that all libraries are likely available in both ways – shared or
> static. Shared means that all locally built binaries (e.g. xmonad!) will
> suddenly break when the user upgrades its Haskell packages, as the
> package management is ignorant of unpackaged, locally built programs.
> I’d feel more comfortable if that could not happen.
>
> Other open question: Should we put the dynamic libraries in the normal
> libghc-*-dev package? Con: Package size doubles (and xmonad users are
> already shocked by the size of stuff they need to install). Pro: It
> cannot happen that I can build Foo.hs statically, but not load it in
> GHCi, or vice-versa.
>
> I still find it unfortunate that once cannot use the .so for static
> linking as well, but that is a problem beyond the scope of GHC.
>
> Greetings,
> Joachim
>
>
>
> _______________________________________________
> Glasgow-haskell-users mailing list
> Glasgow-haskell-users <at> haskell.org
> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
>
Ian Lynagh | 28 Nov 13:41 2012

Re: Dynamic libraries by default and GHC 7.8

On Wed, Nov 28, 2012 at 09:09:58AM +0000, Simon Marlow wrote:
> On 27/11/12 23:28, Joachim Breitner wrote:
> >
> >Hence, Debian will continue to provide its libraries built the static
> >way.
> 
> So let me try to articulate the options, because I think there are
> some dependencies that aren't obvious here.  It's not a
> straightforward choice between -dynamic/-static being the default,
> because of the GHCi interaction.
> 
> Here are the 3 options:
> 
> (1) (the current situation) GHCi is statically linked, and -static is
>     the default.  Uses the RTS linker.
> 
> (2) (the proposal, at least for some platforms) GHCi is dynamically
>     linked, and -dynamic is the default.  Does not use the RTS linker.
> 
> (3) GHCi is dynamically linked, but -static is the default.  Does not
>     use the RTS linker.  Packages must be installed with -dynamic,
>     otherwise they cannot be loaded into GHCi, and only objects
>     compiled with -dynamic can be loaded into GHCi.
> 
> You seem to be saying that Debian would do (3), but we hadn't
> considered that as a viable option because of the extra hoops that
> GHCi users would have to jump through.  We consider it a
> prerequisite that GHCi continues to work without requiring any extra
> flags.

I think what Joachim means is that the binaries /in Debian packages/
will be explicitly linked with -static, but the open question below is
about whether to make -static or -dynamic the default for GHC.

> >Open question: What should GHC on Debian do when building binaries,
> >given that all libraries are likely available in both ways – shared or
> >static. Shared means that all locally built binaries (e.g. xmonad!) will
> >suddenly break when the user upgrades its Haskell packages, as the
> >package management is ignorant of unpackaged, locally built programs.
> >I’d feel more comfortable if that could not happen.

Thanks
Ian

_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users <at> haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Ian Lynagh | 28 Nov 13:39 2012

Re: Dynamic libraries by default and GHC 7.8

On Wed, Nov 28, 2012 at 12:28:31AM +0100, Joachim Breitner wrote:
> 
> here comes the obligatory butting in by the Debian Haskell Group:
> 
> Given the current sensitivity of the ABI hashes we really do not want to
> have Programs written in Haskell have a runtime dependency on all the
> included Haskell libraries. So I believe we should still link Haskell
> programs statically in Debian.

Note that you'll have to link ghc dynamically or ghci won't work. This
won't cause dependency problems, though, as ghc comes with all the
libraries that it needs.

Thanks
Ian
Stephen Paul Weber | 28 Nov 17:04 2012
Picon

Re: Dynamic libraries by default and GHC 7.8

Somebody signing messages as Joachim Breitner wrote:
>Given the current sensitivity of the ABI hashes we really do not want to
>have Programs written in Haskell have a runtime dependency on all the
>included Haskell libraries. So I believe we should still link Haskell
>programs statically in Debian.

This is also a concern I have.

>Hence, Debian will continue to provide its libraries built the static
>way.

Right.

>Building them also in the dynamic way for the sake of GHCi users seems
>possible.

Perhaps Debian could just ship a GHCi that uses the RTS linker, as now?  The 
change is to be made for "some platforms", we could opt to have Debian not 
be such a platform.

--

-- 
Stephen Paul Weber,  <at> singpolyma
See <http://singpolyma.net> for how I prefer to be contacted
edition right joseph
_______________________________________________
Glasgow-haskell-users mailing list
Glasgow-haskell-users <at> haskell.org
http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
Ian Lynagh | 28 Nov 18:04 2012

Re: Dynamic libraries by default and GHC 7.8

On Wed, Nov 28, 2012 at 11:04:44AM -0500, Stephen Paul Weber wrote:
> 
> >Building them also in the dynamic way for the sake of GHCi users seems
> >possible.
> 
> Perhaps Debian could just ship a GHCi that uses the RTS linker, as
> now?  The change is to be made for "some platforms", we could opt to
> have Debian not be such a platform.

This isn't a good long-term plan. Even if we don't actively remove
support for other platforms (which we plan to do), they'll probably
bitrot, and the current bugs won't be fixed.

Thanks
Ian

Gmane