headers
Dag Sverre Seljebotn | 4 Jul 23:46
Picon
Picon

[Cython] Buffer must know exact C types

Currently, wherever a C type is used in Cython, it is enough to specify 
sort-of what kind of class of type it is... the following will work all 
right:

cdef extern ...:
     ctypedef int int8
     ctypedef int int16
     ctypedef int int32
     ctypedef int int64

However, when types are used in buffers:

cdef object[int16, 3] x

one must translate the type (dtype) into a character representation (as 
listed in http://docs.python.org/lib/module-struct.html). For instance, 
signed short is "h" and signed int is "i"; and one can imagine int16 to 
be any of these (which one being decided by the included header-file; 
NumPy comes with a long list defining intXX for a lot of cases; you can 
choose whether to use a C-dependant type like "npy_longlong" or a fixed 
type like "npy_int64").

I see two options:
a) Start requiring exact ctypedefs in Cython; at least to get correct 
buffer behaviour. However, for NumPy this would require a lot of 
"#ifdefs" Cython-side. (NumPy comes with two options for each type,

b) "Somehow" get the C compiler to map the C-compiler-resolved type to 
the char. With C++ this would be easy:
(Continue reading)

Permalink | Reply |
headers
Stefan Behnel | 5 Jul 07:11
Picon
Favicon

Re: [Cython] Buffer must know exact C types

Hi,

Dag Sverre Seljebotn wrote:
> Currently, wherever a C type is used in Cython, it is enough to specify 
> sort-of what kind of class of type it is... the following will work all 
> right:
> 
> cdef extern ...:
>      ctypedef int int8
>      ctypedef int int16
>      ctypedef int int32
>      ctypedef int int64
> 
> 
> However, when types are used in buffers:
> 
> cdef object[int16, 3] x
> 
> one must translate the type (dtype) into a character representation (as 
> listed in http://docs.python.org/lib/module-struct.html).

What about going the opposite way and either a) replacing the type definition
in buffers by the type character entirely (not sure, but this might not work
if you require a specific type name to appear in the C code), or b) by
requiring users to provide both the type and the character in a suitable
buffer declaration syntax?

Stefan
(Continue reading)

Permalink | Reply |
headers
Dag Sverre Seljebotn | 5 Jul 11:09
Picon
Picon

Re: [Cython] Buffer must know exact C types

Stefan Behnel wrote:
> Hi,
> 
> Dag Sverre Seljebotn wrote:
>> Currently, wherever a C type is used in Cython, it is enough to specify 
>> sort-of what kind of class of type it is... the following will work all 
>> right:
>>
>> cdef extern ...:
>>      ctypedef int int8
>>      ctypedef int int16
>>      ctypedef int int32
>>      ctypedef int int64
>>
>>
>> However, when types are used in buffers:
>>
>> cdef object[int16, 3] x
>>
>> one must translate the type (dtype) into a character representation (as 
>> listed in http://docs.python.org/lib/module-struct.html).
> 
> What about going the opposite way and either a) replacing the type definition
> in buffers by the type character entirely (not sure, but this might not work
> if you require a specific type name to appear in the C code), or b) by
> requiring users to provide both the type and the character in a suitable
> buffer declaration syntax?

Thanks, new ideas like that I was looking for!
(Continue reading)

Permalink | Reply |
headers
Greg Ewing | 6 Jul 02:18
Picon
Picon
Favicon

Re: [Cython] Buffer must know exact C types

How does one write portable code in C to do this?

If it's possible to write portably in C, then there
must be a way to have Cython generate that portable
C from some portable representation in Cython source.

--

-- 
Greg
Permalink | Reply |
headers
Dag Sverre Seljebotn | 6 Jul 11:00
Picon
Picon

Re: [Cython] Buffer must know exact C types

Greg wrote:
> How does one write portable code in C to do this?
>
> If it's possible to write portably in C, then there
> must be a way to have Cython generate that portable
> C from some portable representation in Cython source.

The problem (which is now solved) arises because Cython ctypedefs are not
required to be exact, only approximate. When writing C code, your typedefs
are exact and you can always know (at least in principle) the exact type
you are dealing with.

So if you have something like this:

#ifdef A
typedef short foo
#else
typedef int foo
#endif

...then you can always throw in "#define FOO_LETTER 'h'" or "#define
FOO_LETTER 'i'" inside those #ifdefs. However, if it says this in Cython:

ctypedef int foo

then you don't "really" know the exact type of foo (and this feature will
be relied on for numpy.pxd to provide "int16", "int64" etc., and I
wouldn't like to remove it and thus require duplication of all such
#ifdefs in the pxd files).
(Continue reading)

Permalink | Reply |
headers
Robert Bradshaw | 5 Jul 20:02
Favicon

Re: [Cython] Buffer must know exact C types

On Jul 5, 2008, at 2:09 AM, Dag Sverre Seljebotn wrote:

> Stefan Behnel wrote:
>> Hi,
>>
>> Dag Sverre Seljebotn wrote:
>>> Currently, wherever a C type is used in Cython, it is enough to  
>>> specify
>>> sort-of what kind of class of type it is... the following will  
>>> work all
>>> right:
>>>
>>> cdef extern ...:
>>>      ctypedef int int8
>>>      ctypedef int int16
>>>      ctypedef int int32
>>>      ctypedef int int64
>>>
>>>
>>> However, when types are used in buffers:
>>>
>>> cdef object[int16, 3] x
>>>
>>> one must translate the type (dtype) into a character  
>>> representation (as
>>> listed in http://docs.python.org/lib/module-struct.html).
>>
>> What about going the opposite way and either a) replacing the type  
>> definition
>> in buffers by the type character entirely (not sure, but this
(Continue reading)

Permalink | Reply |
headers
Dag Sverre Seljebotn | 5 Jul 20:50
Picon
Picon

Re: [Cython] Buffer must know exact C types

Robert Bradshaw wrote:
> If I understand correctly, the issue here is how to map cython/c  
> types to the "format" string of the buffer. There are three things to  
> match:
> 
> Numeric type (int/float)--this can be detected at compile time by  
> (<type>1)/(<type>2) == 0 vs the buffer format string
> Signed--this can be detected at compile time by (<type>-1) < 0 vs the  
> buffer format string
> Size--this can be detected at compile time by sizeof() vs the  
> "itemsize" field of the buffer, or calcsize from the struct module if  
> you need.

Very nice!

Or simply sizof(type) vs sizeof(int).

> All other types are required to match exactly (or something like  
> that, some thought would have to be put into how to handle objects).  
> For hybrid types I think it would be enough to just look at the total  
> size after comparing the structs for (int/float/other and signed/ 
> unsigned). Then you could use numpy.uint and numpy.unit32, etc.

OK, writeup for clarity in the discussion, this is what needs to happen:

- A format string is recieved run-time when acquiring a buffer
- We have a compile-time assumed type for the buffer
- They need to be compared for sanity-checking

And this is how I see it happen (which is quite different from what I
(Continue reading)

Permalink | Reply |
headers
Robert Bradshaw | 6 Jul 02:55
Favicon

Re: [Cython] Buffer must know exact C types

On Jul 5, 2008, at 11:50 AM, Dag Sverre Seljebotn wrote:

> Robert Bradshaw wrote:
>> If I understand correctly, the issue here is how to map cython/c
>> types to the "format" string of the buffer. There are three things to
>> match:
>>
>> Numeric type (int/float)--this can be detected at compile time by
>> (<type>1)/(<type>2) == 0 vs the buffer format string
>> Signed--this can be detected at compile time by (<type>-1) < 0 vs the
>> buffer format string
>> Size--this can be detected at compile time by sizeof() vs the
>> "itemsize" field of the buffer, or calcsize from the struct module if
>> you need.
>
> Very nice!
>
> Or simply sizof(type) vs sizeof(int).
>
>> All other types are required to match exactly (or something like
>> that, some thought would have to be put into how to handle objects).
>> For hybrid types I think it would be enough to just look at the total
>> size after comparing the structs for (int/float/other and signed/
>> unsigned). Then you could use numpy.uint and numpy.unit32, etc.
>
>
> OK, writeup for clarity in the discussion, this is what needs to  
> happen:
>
> - A format string is recieved run-time when acquiring a buffer
(Continue reading)

Permalink | Reply |
headers
Dag Sverre Seljebotn | 4 Jul 23:49
Picon
Picon

Re: [Cython] Buffer must know exact C types

Dag Sverre Seljebotn wrote:

> int main() {
>    int t;
>    cout << TypeChar<typeof(t)>::value;
> }
> 
> however I'm not sure if any tricks are possible in C to get something 
> like this?
> 

To be clear, C++ would then also allow this:

typedef int foo;
int main() {
   foo t;
   cout << TypeChar<typeof(t)>::value;
}

and so, one wouldn't need an "exact" foo ctypedef in Cython, one would 
just use the template to map the type to the character when needed...

--

-- 
Dag Sverre
Permalink | Reply |

Gmane