David Naylor | 19 May 17:55 2011
Picon

PyPy compile memory requirements

Hi,

I'm sure you all know that translating pypy requires a large amount of RAM.

It has been my experience that GCC also uses a large amount of RAM when
compiling, about 4G.  Is it possible to get python/pypy to stop once
it has written
the source files and then to run that build process separately, thus avoiding
having both pypy and gcc in memory at the same time?

This is on FreeBSD/amd64:
# gcc -v
Using built-in specs.
Target: amd64-undermydesk-freebsd
Configured with: FreeBSD/amd64 system compiler
Thread model: posix
gcc version 4.2.2 20070831 prerelease [FreeBSD]

I've observed similar behavior using other versions of gcc.  Clang does not
exhibit this problem.

Regards,

David
Maciej Fijalkowski | 19 May 18:00 2011
Picon

Re: PyPy compile memory requirements

On Thu, May 19, 2011 at 5:55 PM, David Naylor <naylor.b.david <at> gmail.com> wrote:
> Hi,
>
> I'm sure you all know that translating pypy requires a large amount of RAM.
>
> It has been my experience that GCC also uses a large amount of RAM when
> compiling, about 4G.  Is it possible to get python/pypy to stop once
> it has written
> the source files and then to run that build process separately, thus avoiding
> having both pypy and gcc in memory at the same time?

This is probably due to some GCC bug (I remember the same thing
occuring on ubuntu a while ago).

If the only goal is to avoid doing those 2 at the same time, do
translate.py --source and then go to the /tmp/usession-*/testing_1
directory and type make

Note that it should nicely work with swap (since the python process is
not running any more when gcc is)

>
> This is on FreeBSD/amd64:
> # gcc -v
> Using built-in specs.
> Target: amd64-undermydesk-freebsd
> Configured with: FreeBSD/amd64 system compiler
> Thread model: posix
> gcc version 4.2.2 20070831 prerelease [FreeBSD]
>
(Continue reading)

Armin Rigo | 19 May 18:34 2011

Re: PyPy compile memory requirements

Hi,

On Thu, May 19, 2011 at 6:00 PM, Maciej Fijalkowski <fijall <at> gmail.com> wrote:
> If the only goal is to avoid doing those 2 at the same time, do
> translate.py --source and then go to the /tmp/usession-*/testing_1
> directory and type make

We should decide if it would make sense or not to just do
os.execv("make"), which would have the same effect.  The only drawback
I can think of would be that you wouldn't get the nice summary and
timings at the end of translation --- which we can maybe re-obtain
anyway by hacking a bit :-)

A bientôt,

Armin.
Ronny Pfannschmidt | 19 May 20:30 2011
Picon
Picon

Re: PyPy compile memory requirements

On Thu, 2011-05-19 at 18:34 +0200, Armin Rigo wrote:
> Hi,
> 
> On Thu, May 19, 2011 at 6:00 PM, Maciej Fijalkowski <fijall <at> gmail.com> wrote:
> > If the only goal is to avoid doing those 2 at the same time, do
> > translate.py --source and then go to the /tmp/usession-*/testing_1
> > directory and type make
> 
> We should decide if it would make sense or not to just do
> os.execv("make"), which would have the same effect.  The only drawback
> I can think of would be that you wouldn't get the nice summary and
> timings at the end of translation --- which we can maybe re-obtain
> anyway by hacking a bit :-)
why not run the translation in a subprocess to begin with?
> 
> 
> A bientôt,
> 
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev <at> python.org
> http://mail.python.org/mailman/listinfo/pypy-dev

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
(Continue reading)

Armin Rigo | 20 May 10:07 2011

Re: PyPy compile memory requirements

Hi Ronny,

On Thu, May 19, 2011 at 8:30 PM, Ronny Pfannschmidt
<Ronny.Pfannschmidt <at> gmx.de> wrote:
> why not run the translation in a subprocess to begin with?

It doesn't change much, except that it would break half of the
non-standard use cases of translate.py, like --fork-before.

Armin.

Gmane