headers
Robin Becker | 21 Apr 2009 15:48
Favicon

[Cython] inherit from list

I have some older code that used to run with pyrex without complaint, but now 
the following

cdef extern from "listobject.h":
	ctypedef class __builtin__.list [object PyListObject]:
		pass

causes a warning
> warning: C:\code\users\robin\_tex_wrap.pyx:144:1: list already a builtin Cython type

however, I cannot remove the above code as that causes an error at my class 
declaration

> cdef class ObjectList(list):

what's the correct cython way to inherit from a list or other builtin? I have 
looked in the extension_types section, but am no wiser.
--
Robin Becker
Permalink | Reply |
headers
Stefan Behnel | 21 Apr 2009 18:32
Picon
Favicon

Re: [Cython] inherit from list

Robin Becker wrote:
> I have some older code that used to run with pyrex without complaint, but
> now the following
>
> cdef extern from "listobject.h":
> 	ctypedef class __builtin__.list [object PyListObject]:
> 		pass
>
> causes a warning
>> warning: C:\code\users\robin\_tex_wrap.pyx:144:1: list already a builtin
>> Cython type

Yes, that warning is currently a bit misleading in this specific case, as
there is not yet a way to achieve inheritance of builtin types without the
above declaration. It's meant more for code that uses the declared type in
other places than supertypes, where Cython now handles these types better
on its own.

This is what ticket #258 is supposed to mean.

http://trac.cython.org/cython_trac/ticket/258

See also ticket #170:

http://trac.cython.org/cython_trac/ticket/170

If you use the "list" type only for subtypes, just ignore the warning. It
will become an error once Cython can deal with this case directly, so you
will notice when it's time to remove the declaration.
(Continue reading)

Permalink | Reply |
headers
Robin Becker | 21 Apr 2009 18:40
Favicon

Re: [Cython] inherit from list

.........
> 
> Yes, that warning is currently a bit misleading in this specific case, as
> there is not yet a way to achieve inheritance of builtin types without the
> above declaration. It's meant more for code that uses the declared type in
> other places than supertypes, where Cython now handles these types better
> on its own.
> 
> This is what ticket #258 is supposed to mean.
> 
> http://trac.cython.org/cython_trac/ticket/258
> 
> See also ticket #170:
> 
> http://trac.cython.org/cython_trac/ticket/170
> 
> If you use the "list" type only for subtypes, just ignore the warning. It
> will become an error once Cython can deal with this case directly, so you
> will notice when it's time to remove the declaration.
> 
> Stefan
......

OK thanks. I noticed that all seems to work. Is the intent to allow inheriting 
from builtin types to be handled automatically as it is in python where list is 
both a constructor and a type?
--

-- 
Robin Becker
Permalink | Reply |
headers
Greg Ewing | 22 Apr 2009 02:34
Picon
Picon
Favicon

Re: [Cython] inherit from list

Robin Becker wrote:

> OK thanks. I noticed that all seems to work. Is the intent to allow inheriting 
> from builtin types to be handled automatically as it is in python where list is 
> both a constructor and a type?

FWIW, the current version of Pyrex handles inheriting
from the builtin list type okay, without the need for
any declarations.

--

-- 
Greg
Permalink | Reply |
headers
Robin Becker | 22 Apr 2009 13:28
Favicon

Re: [Cython] inherit from list

Greg Ewing wrote:
> Robin Becker wrote:
> 
>> OK thanks. I noticed that all seems to work. Is the intent to allow inheriting 
>> from builtin types to be handled automatically as it is in python where list is 
>> both a constructor and a type?
> 
> FWIW, the current version of Pyrex handles inheriting
> from the builtin list type okay, without the need for
> any declarations.
> 
yes this code is a bit out of date.

--

-- 
Robin Becker
Permalink | Reply |
headers
Stefan Behnel | 22 Apr 2009 14:59
Picon
Favicon

Re: [Cython] inherit from list

Robin Becker wrote:
> Greg Ewing wrote:
>> FWIW, the current version of Pyrex handles inheriting
>> from the builtin list type okay, without the need for
>> any declarations.
>>
> yes this code is a bit out of date.

Not out of date (in the sense that it hasn't been worked on), but lacking
a port. This Pyrex feature dates back to March 2008 (Pyrex 0.9.7, hg rev.
26-37). It won't be trivial to port, and from a quick glance, it looks a
bit like a clever hack. It will likely interfere with Cython's builtin
caching feature, although that might become redundant in many cases.
True/False handling is another thing that works different in Cython now,
so that must be left out.

I also wonder how exception subtyping is handled for Python versions
before 2.5. I assume this is left to the user, so that

    cdef class MyExc(Exception): pass

will require Py2.5+, whereas

    class MyExc(Exception): pass

should still work in Py2.3.

It's too bad that Pyrex's test suite doesn't really test anything. It will
be a bit of work to add suitable test cases to Cython.
(Continue reading)

Permalink | Reply |
headers
Greg Ewing | 23 Apr 2009 02:59
Picon
Picon
Favicon

Re: [Cython] inherit from list

Stefan Behnel wrote:

> I also wonder how exception subtyping is handled for Python versions
> before 2.5. I assume this is left to the user, so that
> 
>     cdef class MyExc(Exception): pass
> 
> will require Py2.5+, whereas
> 
>     class MyExc(Exception): pass
> 
> should still work in Py2.3.

Yes. Do you think it can/should be handled differently?

--

-- 
Greg
Permalink | Reply |
headers
Stefan Behnel | 23 Apr 2009 07:13
Picon
Favicon

Re: [Cython] inherit from list


Greg Ewing wrote:
> Stefan Behnel wrote:
> 
>> I also wonder how exception subtyping is handled for Python versions
>> before 2.5. I assume this is left to the user, so that
>>
>>     cdef class MyExc(Exception): pass
>>
>> will require Py2.5+, whereas
>>
>>     class MyExc(Exception): pass
>>
>> should still work in Py2.3.
> 
> Yes. Do you think it can/should be handled differently?

No, that's what I expected. Cython doesn't currently handle exception types
any special, so it's not a problem here (unless users declare the exception
types themselves).

I think the behaviour of Pyrex makes sense here. After all, it's a user's
decision if speed is more important than backwards portability for exceptions.

I would expect a compile error in that case, though, like Cython does in a
couple of other cases. Something like a conditional

"""
#if PY_VERSION_HEX < 0x0205000000
#error This module requires new style exceptions, which are only \
(Continue reading)

Permalink | Reply |
headers
Lisandro Dalcin | 23 Apr 2009 03:37
Picon
Gravatar

Re: [Cython] inherit from list

On Wed, Apr 22, 2009 at 9:59 PM, Greg Ewing <greg.ewing@...> wrote:
> Stefan Behnel wrote:
>
>> I also wonder how exception subtyping is handled for Python versions
>> before 2.5. I assume this is left to the user, so that
>>
>>     cdef class MyExc(Exception): pass
>>
>> will require Py2.5+, whereas
>>
>>     class MyExc(Exception): pass
>>
>> should still work in Py2.3.
>
> Yes. Do you think it can/should be handled differently?
>
> --
> Greg
> _______________________________________________
> Cython-dev mailing list
> Cython-dev@...
> http://codespeak.net/mailman/listinfo/cython-dev
>

AFAIK, you cannot handle it differently... old-style, aka classic
classes are actually instances (of types.ClassType), so a cdef class
class (implicitly new-style) cannot inherit from an instance, very
much like "class A([]): pass" ends-up failing in core Py2.

--

--
(Continue reading)

Permalink | Reply |
headers
Robin Becker | 22 Apr 2009 15:39
Favicon

Re: [Cython] inherit from list

Stefan Behnel wrote:
> Robin Becker wrote:
>> Greg Ewing wrote:
>>> FWIW, the current version of Pyrex handles inheriting
>>> from the builtin list type okay, without the need for
>>> any declarations.
>>>
>> yes this code is a bit out of date.
> 

> Not out of date (in the sense that it hasn't been worked on), but lacking
> a port. This Pyrex feature dates back to March 2008 (Pyrex 0.9.7, hg rev.
> 26-37). It won't be trivial to port, and from a quick glance, it looks a
> bit like a clever hack. It will likely interfere with Cython's builtin
> caching feature, although that might become redundant in many cases.
> True/False handling is another thing that works different in Cython now,
> so that must be left out.
> 

you mistake me :) I was referring to my old pyrex compatible code; I wrote it 
about two years ago and pyrex has advanced since then so apparently I don't need 
the declarations with modern pyrex.

So far as I'm aware cython is advancing wave upon wave and will eventually win 
great victories.

.........

--

-- 
Robin Becker
(Continue reading)

Permalink | Reply |
headers
Stefan Behnel | 22 Apr 2009 07:43
Picon
Favicon

Re: [Cython] inherit from list


Greg Ewing wrote:
> Robin Becker wrote:
> 
>> OK thanks. I noticed that all seems to work. Is the intent to allow inheriting 
>> from builtin types to be handled automatically as it is in python where list is 
>> both a constructor and a type?
> 
> FWIW, the current version of Pyrex handles inheriting
> from the builtin list type okay, without the need for
> any declarations.

So, when you say

    cdef class mylist(list):
        pass

this will work? I assume this also works for all other (non-PyVarObject)
builtin types?

Will Pyrex handle the above type in any special way because it knows that
mylist is of type 'list', or will it ignore the builtin type specific
optimisations for it?

Stefan
Permalink | Reply |
headers
Greg Ewing | 23 Apr 2009 02:07
Picon
Picon
Favicon

Re: [Cython] inherit from list

Stefan Behnel wrote:

>     cdef class mylist(list):
>         pass
> 
> this will work?

I haven't tested it extensively, but the above certainly
compiles, and I don't know of any reason why it shouldn't
work. Pyrex just preloads the symbol table as if you had
written an external declaration for type 'list', so the
end result should be the same.

> I assume this also works for all other (non-PyVarObject)
> builtin types?

It should work for the ones Pyrex knows about. You will
need to provide your own declarations for any others.

> Will Pyrex handle the above type in any special way because it knows that
> mylist is of type 'list'

It *should*, but trying it just now, there seems to be
a bug... I'll see about fixing it.

--

-- 
Greg
Permalink | Reply |
headers
Robert Bradshaw | 22 Apr 2009 08:30
Favicon

Re: [Cython] inherit from list

On Apr 21, 2009, at 10:43 PM, Stefan Behnel wrote:

> Greg Ewing wrote:
>> Robin Becker wrote:
>>
>>> OK thanks. I noticed that all seems to work. Is the intent to  
>>> allow inheriting
>>> from builtin types to be handled automatically as it is in python  
>>> where list is
>>> both a constructor and a type?
>>
>> FWIW, the current version of Pyrex handles inheriting
>> from the builtin list type okay, without the need for
>> any declarations.
>
> So, when you say
>
>     cdef class mylist(list):
>         pass
>
> this will work? I assume this also works for all other (non- 
> PyVarObject)
> builtin types?
>
> Will Pyrex handle the above type in any special way because it  
> knows that
> mylist is of type 'list', or will it ignore the builtin type specific
> optimisations for it?

You have to ignore the type-specific optimizations, because mylist
(Continue reading)

Permalink | Reply |
headers
Greg Ewing | 23 Apr 2009 02:20
Picon
Picon
Favicon

Re: [Cython] inherit from list

Robert Bradshaw wrote:

> You have to ignore the type-specific optimizations, because mylist  
> (or a subclass thereof) might override the default behavior.

That's somewhat tricky in Pyrex, because it makes its
optimisation decisions based on the static type of the
object. So if you pass your list subclass to something
declared as taking a list, the list optimisastions will
get done on it anyway.

I'm not sure what's the best thing to do here -- only
do the optimisations for something declared as exactly
a list, or for any subclass of list.

--

-- 
Greg
Permalink | Reply |
headers
Robert Bradshaw | 23 Apr 2009 02:35
Favicon

Re: [Cython] inherit from list

On Apr 22, 2009, at 5:20 PM, Greg Ewing wrote:

> Robert Bradshaw wrote:
>
>> You have to ignore the type-specific optimizations, because mylist
>> (or a subclass thereof) might override the default behavior.
>
> That's somewhat tricky in Pyrex, because it makes its
> optimisation decisions based on the static type of the
> object. So if you pass your list subclass to something
> declared as taking a list, the list optimisastions will
> get done on it anyway.
>
> I'm not sure what's the best thing to do here -- only
> do the optimisations for something declared as exactly
> a list, or for any subclass of list.

In Cython we only allow None or exactly a list to be assigned to a  
cdef list variable, not subclass thereof. We discussed this when we  
introduced it, and I am still convinced it is the right thing to do,  
but perhaps this makes subclassing messier.

- Robert
Permalink | Reply |
headers
Lisandro Dalcin | 23 Apr 2009 02:31
Picon
Gravatar

Re: [Cython] inherit from list

On Wed, Apr 22, 2009 at 9:20 PM, Greg Ewing <greg.ewing@...> wrote:
> Robert Bradshaw wrote:
>
>> You have to ignore the type-specific optimizations, because mylist
>> (or a subclass thereof) might override the default behavior.
>
> I'm not sure what's the best thing to do here -- only
> do the optimisations for something declared as exactly
> a list, or for any subclass of list.
>

If you dive into core CPython sources, I would not be surprised to
find places where PyList_Check() is done and PyList_{Set|Get}Item() is
next called.

--

-- 
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
Permalink | Reply |
headers
Greg Ewing | 23 Apr 2009 03:37
Picon
Picon
Favicon

Re: [Cython] inherit from list

Lisandro Dalcin wrote:

> If you dive into core CPython sources, I would not be surprised to
> find places where PyList_Check() is done and PyList_{Set|Get}Item() is
> next called.

But that's a dynamic check, which might be all right
for Cython, but it's not the way Pyrex does things.

--

-- 
Greg
Permalink | Reply |
headers
Lisandro Dalcin | 23 Apr 2009 03:43
Picon
Gravatar

Re: [Cython] inherit from list

On Wed, Apr 22, 2009 at 10:37 PM, Greg Ewing
<greg.ewing@...> wrote:
> Lisandro Dalcin wrote:
>
>> If you dive into core CPython sources, I would not be surprised to
>> find places where PyList_Check() is done and PyList_{Set|Get}Item() is
>> next called.
>
> But that's a dynamic check, which might be all right
> for Cython, but it's not the way Pyrex does things.
>

Did you noticed I'm talking about core CPython, not Cython?

Anyway, IMHO, Cython/Pyrex should optimize ONLY if PyList_CheckExact()
returns 1... Just to avoid surprises...

--

-- 
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
Permalink | Reply |
headers
Robert Bradshaw | 23 Apr 2009 05:55
Favicon

Re: [Cython] inherit from list

On Apr 22, 2009, at 6:43 PM, Lisandro Dalcin wrote:

> On Wed, Apr 22, 2009 at 10:37 PM, Greg Ewing
> <greg.ewing@...> wrote:
>> Lisandro Dalcin wrote:
>>
>>> If you dive into core CPython sources, I would not be surprised to
>>> find places where PyList_Check() is done and PyList_{Set|Get}Item 
>>> () is
>>> next called.
>>
>> But that's a dynamic check, which might be all right
>> for Cython, but it's not the way Pyrex does things.

Cython checks the type on assignment, not every time it gets accessed  
(just like extension types).

> Did you noticed I'm talking about core CPython, not Cython?
>
> Anyway, IMHO, Cython/Pyrex should optimize ONLY if PyList_CheckExact()
> returns 1... Just to avoid surprises...
>
>
>
> -- 
> Lisandro Dalcín
> ---------------
> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
(Continue reading)

Permalink | Reply |
headers
Stefan Behnel | 22 Apr 2009 08:57
Picon
Favicon

Re: [Cython] inherit from list

Robert Bradshaw wrote:
> On Apr 21, 2009, at 10:43 PM, Stefan Behnel wrote:
>>     cdef class mylist(list):
>>         pass
>>
>> Will Pyrex handle the above type in any special way because it
>> knows that mylist is of type 'list', or will it ignore the
>> builtin type specific optimisations for it?
>
> You have to ignore the type-specific optimizations, because mylist
> (or a subclass thereof) might override the default behavior.

As would Cython, currently (so that's the answer I'd expect). That's a
different topic now, but if the subtype does not override __getitem__(),
most list optimisations would work just fine, for example. Same for the
append() optimisation if the type does not override append() or
__getattr(ibute)__().

Now that we have a transform for builtin type method optimisations anyway,
we could check the type definition itself, instead of restricting it to
the plain builtin types.

One reason more to fix Cython here so that we can make builtin type
redefinition a real error.

Stefan
Permalink | Reply |
headers
Stefan Behnel | 21 Apr 2009 19:04
Picon
Favicon

Re: [Cython] inherit from list

Robin Becker wrote:
> Is the intent to allow inheriting
> from builtin types to be handled automatically as it is in python where
> list is both a constructor and a type?

Yes, that's what ticket #258 is supposed to track (although the
description is a bit short to really get the idea).

Stefan
Permalink | Reply |

Gmane