Stefan Behnel | 9 Oct 19:35 2011
Picon

[Cython] PyCon-DE wrap-up by Kay Hayen

Hi,

Kay Hayen wrote a blog post about his view of the first PyCon-DE, including 
a bit on the discussions I had with him about Nuitka.

http://www.nuitka.net/blog/2011/10/pycon-de-2011-my-report/

It was interesting to see that Nuitka actually comes from the other side, 
meaning that it tries to be a pure Python compiler, but should at some 
point start to support (Python) type hints for the compiler. Cython made 
static types a language feature from the very beginning and is now fixing 
up the Python compatibility. So both systems will eventually become rather 
similar in what they achieve, with Cython being essentially a superset of 
the feature set of Nuitka due to its additional focus on talking to 
external libraries efficiently and supporting things like parallel loops or 
the PEP-3118 buffer interface.

One of the impressions I took out of the technical discussions with Kay is 
that there isn't really a good reason why Cython should refuse to duplicate 
some of the inner mechanics of CPython for optimisation purposes. Nuitka 
appears to be somewhat more aggressive here, partly because Kay doesn't 
currently care all that much about portability (e.g. to Python 3).

I was previously very opposed to that (you may remember my opposition to 
the list.pop() optimisation), but now I think that we have to fix up the 
generated code for each new major CPython release anyway, so it won't make 
a difference if we have to rework some more of the code because a bit of 
those inner workings changed. They sure won't change for released CPython 
versions anymore, and many implementation details are unlikely enough to 
change for years to come. It's good to continue to be considerate about 
(Continue reading)

mark florisson | 9 Oct 19:57 2011
Picon

Re: [Cython] PyCon-DE wrap-up by Kay Hayen

On 9 October 2011 18:35, Stefan Behnel <stefan_ml@...> wrote:
> Hi,
>
> Kay Hayen wrote a blog post about his view of the first PyCon-DE, including
> a bit on the discussions I had with him about Nuitka.
>
> http://www.nuitka.net/blog/2011/10/pycon-de-2011-my-report/
>
> It was interesting to see that Nuitka actually comes from the other side,
> meaning that it tries to be a pure Python compiler, but should at some point
> start to support (Python) type hints for the compiler. Cython made static
> types a language feature from the very beginning and is now fixing up the
> Python compatibility. So both systems will eventually become rather similar
> in what they achieve, with Cython being essentially a superset of the
> feature set of Nuitka due to its additional focus on talking to external
> libraries efficiently and supporting things like parallel loops or the
> PEP-3118 buffer interface.
>
> One of the impressions I took out of the technical discussions with Kay is
> that there isn't really a good reason why Cython should refuse to duplicate
> some of the inner mechanics of CPython for optimisation purposes. Nuitka
> appears to be somewhat more aggressive here, partly because Kay doesn't
> currently care all that much about portability (e.g. to Python 3).

Interesting. What kind of (significant) optimizations could be made by
duplicating code? Do you want to duplicate entire functions or do you
want to inline parts of those?

I actually think we should not get too tied to CPython, e.g. what if
PyPy gets a CPython compatible API, or possibly a subset like PEP 384?
(Continue reading)

Stefan Behnel | 10 Oct 10:38 2011
Picon

Re: [Cython] PyCon-DE wrap-up by Kay Hayen

mark florisson, 09.10.2011 19:57:
> On 9 October 2011 18:35, Stefan Behnel wrote:
>> One of the impressions I took out of the technical discussions with Kay is
>> that there isn't really a good reason why Cython should refuse to duplicate
>> some of the inner mechanics of CPython for optimisation purposes. Nuitka
>> appears to be somewhat more aggressive here, partly because Kay doesn't
>> currently care all that much about portability (e.g. to Python 3).
>
> Interesting. What kind of (significant) optimizations could be made by
> duplicating code? Do you want to duplicate entire functions or do you
> want to inline parts of those?

I was mainly referring to things like direct access to type/object struct 
fields and little things like that. They can make a difference especially 
in loops, compared to calling into a generic C-API function. For example, 
we could have our own interned implementation of PyDict_Next(). I'm not 
very impressed by the performance of that C-API function - repeated calls 
to GetItem can be faster than looping over a dict with PyDict_Next()!

That being said, I wasn't referring to any specific changes. It was more of 
a general remark about the invisible line that we currently draw in Cython.

> I actually think we should not get too tied to CPython, e.g. what if
> PyPy gets a CPython compatible API

It already implements a part of the C-API:

http://morepypy.blogspot.com/2010/04/using-cpython-extension-modules-with.html

However, if we really want to support it at that level, there's likely more 
(Continue reading)

mark florisson | 10 Oct 21:59 2011
Picon

Re: [Cython] PyCon-DE wrap-up by Kay Hayen

On 10 October 2011 09:38, Stefan Behnel <stefan_ml@...> wrote:
> mark florisson, 09.10.2011 19:57:
>>
>> On 9 October 2011 18:35, Stefan Behnel wrote:
>>>
>>> One of the impressions I took out of the technical discussions with Kay
>>> is
>>> that there isn't really a good reason why Cython should refuse to
>>> duplicate
>>> some of the inner mechanics of CPython for optimisation purposes. Nuitka
>>> appears to be somewhat more aggressive here, partly because Kay doesn't
>>> currently care all that much about portability (e.g. to Python 3).
>>
>> Interesting. What kind of (significant) optimizations could be made by
>> duplicating code? Do you want to duplicate entire functions or do you
>> want to inline parts of those?
>
> I was mainly referring to things like direct access to type/object struct
> fields and little things like that.

Ah, I see. I suppose that if you do everything through Cython-specific
macros it will be easy to change it at any time and it will make it
easy to experiment with performance as well.

> They can make a difference especially in
> loops, compared to calling into a generic C-API function. For example, we
> could have our own interned implementation of PyDict_Next(). I'm not very
> impressed by the performance of that C-API function - repeated calls to
> GetItem can be faster than looping over a dict with PyDict_Next()!
>
(Continue reading)

Robert Bradshaw | 11 Oct 08:11 2011

Re: [Cython] PyCon-DE wrap-up by Kay Hayen

Thanks for the update and link. Sounds like PyCon-DE went well.

On Mon, Oct 10, 2011 at 1:38 AM, Stefan Behnel <stefan_ml@...> wrote:
> mark florisson, 09.10.2011 19:57:
>>
>> On 9 October 2011 18:35, Stefan Behnel wrote:
>>>
>>> One of the impressions I took out of the technical discussions with Kay
>>> is
>>> that there isn't really a good reason why Cython should refuse to
>>> duplicate
>>> some of the inner mechanics of CPython for optimisation purposes. Nuitka
>>> appears to be somewhat more aggressive here, partly because Kay doesn't
>>> currently care all that much about portability (e.g. to Python 3).
>>
>> Interesting. What kind of (significant) optimizations could be made by
>> duplicating code? Do you want to duplicate entire functions or do you
>> want to inline parts of those?
>
> I was mainly referring to things like direct access to type/object struct
> fields and little things like that. They can make a difference especially in
> loops, compared to calling into a generic C-API function. For example, we
> could have our own interned implementation of PyDict_Next(). I'm not very
> impressed by the performance of that C-API function - repeated calls to
> GetItem can be faster than looping over a dict with PyDict_Next()!
>
> That being said, I wasn't referring to any specific changes. It was more of
> a general remark about the invisible line that we currently draw in Cython.

CPython, especially the internals, is a slow enough moving target that
(Continue reading)

Stefan Behnel | 12 Oct 21:52 2011
Picon

Re: [Cython] PyCon-DE wrap-up by Kay Hayen

Robert Bradshaw, 11.10.2011 08:11:
> Thanks for the update and link. Sounds like PyCon-DE went well.

More than that - here's my take on it:

http://blog.behnel.de/index.php?p=188

Stefan

Gmane