headers
Vladimir Perić | 8 Jun 2011 13:07
Picon

2to3 speed

Hello,

I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The goal is to use a single code base and then run 2to3 automatically when building for Python 3. Unfortunately, I've hit a small problem: running 2to3 takes a _long_ time, somewhere around 10 mins. This is a problem for users, but a particularly big one for me as it makes development last much longer (the change (Py2) code, run 2to3, test cycle takes a long time). Is there a way to speed up 2to3 execution? I do realize we've got quite a large code base. Will running 2to3 with some fixers disabled help with running time (ie. if I know I've fixed reduce and callable before, I might as well disable the fixers)? Any other advice to making it run faster?

Thanks a lot!

[0] http://sympy.org/

--
Vladimir Perić

_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting
Permalink | Reply |
headers
Lennart Regebro | 8 Jun 2011 13:20
Picon
Gravatar

Re: 2to3 speed

On Wed, Jun 8, 2011 at 13:07, Vladimir Perić <vlada.peric <at> gmail.com> wrote:
> Hello,
>
> I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The
> goal is to use a single code base and then run 2to3 automatically when
> building for Python 3. Unfortunately, I've hit a small problem: running 2to3
> takes a _long_ time, somewhere around 10 mins. This is a problem for users,
> but a particularly big one for me as it makes development last much longer
> (the change (Py2) code, run 2to3, test cycle takes a long time). Is there a
> way to speed up 2to3 execution? I do realize we've got quite a large code
> base. Will running 2to3 with some fixers disabled help with running time
> (ie. if I know I've fixed reduce and callable before, I might as well
> disable the fixers)? Any other advice to making it run faster?

The major way to speed up 2to3 during development is to only run it on
the files that has changed, by comparing timestamps. Running it on one
file usually only takes a couple of seconds. More explanation here:
http://python3porting.com/strategies.html#converting-to-python-3-with-2to3

Code examples for how to do this exists in distutils build_2to3
command, if I remember correctly, but since SymPy is already using
distutils I would recommend using the build_2to3 command distutils
has. Or in fact, I'd recommend using Distribute as it's test command
already will run the tests in the build directory and hence
automatically use 2to3 if you have set it up to do that. See
http://packages.python.org/distribute/python3.html for more on that.

You can speed it up a bit more by only including fixers that you are
using, but for a complex software that is likely to be most of them
anyway.

//Lennart
_______________________________________________
Python-porting mailing list
Python-porting <at> python.org
http://mail.python.org/mailman/listinfo/python-porting
Permalink | Reply |
headers
Benjamin Peterson | 8 Jun 2011 16:01
Favicon
Gravatar

Re: 2to3 speed

2011/6/8 Vladimir Perić <vlada.peric <at> gmail.com>:
> Hello,
>
> I'm trying to port SymPy[0] (symbolic mathematics library) to Python 3. The
> goal is to use a single code base and then run 2to3 automatically when
> building for Python 3. Unfortunately, I've hit a small problem: running 2to3
> takes a _long_ time, somewhere around 10 mins. This is a problem for users,
> but a particularly big one for me as it makes development last much longer
> (the change (Py2) code, run 2to3, test cycle takes a long time). Is there a
> way to speed up 2to3 execution? I do realize we've got quite a large code
> base. Will running 2to3 with some fixers disabled help with running time
> (ie. if I know I've fixed reduce and callable before, I might as well
> disable the fixers)? Any other advice to making it run faster?

What version of 2to3 are you using? 2to3 got a nice performance boost
over the summer which should have made it into Python 3.2.

--

-- 
Regards,
Benjamin
_______________________________________________
Python-porting mailing list
Python-porting <at> python.org
http://mail.python.org/mailman/listinfo/python-porting
Permalink | Reply |
headers
Vladimir Perić | 10 Jun 2011 23:56
Picon

Re: 2to3 speed

Thanks to both of you! I am currently using tox for testing/development, but I think I understood from your explanation how to make it all work under Python 3 as well. I've also downloaded the 3.2 version of it and it's about 40% faster, so that was a great help as well!

I have encountered an interesting issue, though - one of the libraries SymPy bundles is written to support Python 3 in a single code base (without 2to3). Unfortunately, some of the fixes 2to3 makes bungle this up. Now, editing the library is not a good solution, I'd rather keep it as is - is it possible to tell to 2to3 to skip a part of the code base? And, of course, can I do this from distutils/distribute?


--
Vladimir Perić

_______________________________________________
Python-porting mailing list
Python-porting@...
http://mail.python.org/mailman/listinfo/python-porting
Permalink | Reply |
headers
Lennart Regebro | 11 Jun 2011 00:36
Picon
Gravatar

Re: 2to3 speed

On Fri, Jun 10, 2011 at 23:56, Vladimir Perić <vlada.peric <at> gmail.com> wrote:
> Thanks to both of you! I am currently using tox for testing/development, but
> I think I understood from your explanation how to make it all work under
> Python 3 as well. I've also downloaded the 3.2 version of it and it's about
> 40% faster, so that was a great help as well!
>
> I have encountered an interesting issue, though - one of the libraries SymPy
> bundles is written to support Python 3 in a single code base (without 2to3).
> Unfortunately, some of the fixes 2to3 makes bungle this up. Now, editing the
> library is not a good solution, I'd rather keep it as is - is it possible to
> tell to 2to3 to skip a part of the code base? And, of course, can I do this
> from distutils/distribute?

If those are included in package itself, then there isn't. If they are
separate packages it will work with no changes.
_______________________________________________
Python-porting mailing list
Python-porting <at> python.org
http://mail.python.org/mailman/listinfo/python-porting
Permalink | Reply |

Gmane