Christian Heimes | 17 Jul 2012 17:38
Picon

[Cython] 2d buffer interface with aligned data

Hello,

I'm the author of https://bitbucket.org/tiran/smc.freeimage , a Cython
wrapper of the FreeImage and LCMS libraries. FreeImage supports a broad
variety of image formats and pixel formats from standard RGB up to RGBAF
and complex numbers.

Now I like to add support for buffer interface to my code to support
zero-copy access of pixel data from NumPy. The wiki describes several
ways to access objects through the new and old Python buffer interface.
However I'm unable to find an example how to *implement* the buffer
interface.

Memory alignment makes the buffer interface more complicated, too.
FreeImage stores the raw pixels non-contiguously. The lines are memory
aligned (usually 32bit). For example a standard RGB image with 1 byte
per pixel and size of 2*3 pixels has a memory layout of

   BGRBGRxxBGRBGRxxBGRBGRxx

on a little-endian machine. 'xx' are the two alignment bytes.

Here is an excerpt from the FreeImage manual on how to access the raw
pixel data. The above example has width: 2, height: 3, pitch: 8 and bpp: 3.

unsigned width = FreeImage_GetWidth(dib);   // 2
unsigned height = FreeImage_GetHeight(dib); // 3
unsigned pitch = FreeImage_GetPitch(dib);   // 8
unsigned bpp = FreeImage_GetBPP(dib);       // 3

(Continue reading)

Dag Sverre Seljebotn | 17 Jul 2012 18:55
Picon
Picon
Gravatar

Re: [Cython] 2d buffer interface with aligned data

On 07/17/2012 05:38 PM, Christian Heimes wrote:
> Hello,
>
> I'm the author of https://bitbucket.org/tiran/smc.freeimage , a Cython
> wrapper of the FreeImage and LCMS libraries. FreeImage supports a broad
> variety of image formats and pixel formats from standard RGB up to RGBAF
> and complex numbers.
>
> Now I like to add support for buffer interface to my code to support
> zero-copy access of pixel data from NumPy. The wiki describes several
> ways to access objects through the new and old Python buffer interface.
> However I'm unable to find an example how to *implement* the buffer
> interface.

Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in 
your cdef class (don't know if it's documented but you can see example 
in tests/run/buffer.pyx).

>
> Memory alignment makes the buffer interface more complicated, too.
> FreeImage stores the raw pixels non-contiguously. The lines are memory
> aligned (usually 32bit). For example a standard RGB image with 1 byte
> per pixel and size of 2*3 pixels has a memory layout of
>
>     BGRBGRxxBGRBGRxxBGRBGRxx
>
> on a little-endian machine. 'xx' are the two alignment bytes.

This is easily supported; above you would let

(Continue reading)

Dag Sverre Seljebotn | 17 Jul 2012 18:56
Picon
Picon
Gravatar

Re: [Cython] 2d buffer interface with aligned data

On 07/17/2012 06:55 PM, Dag Sverre Seljebotn wrote:
> On 07/17/2012 05:38 PM, Christian Heimes wrote:
>> Hello,
>>
>> I'm the author of https://bitbucket.org/tiran/smc.freeimage , a Cython
>> wrapper of the FreeImage and LCMS libraries. FreeImage supports a broad
>> variety of image formats and pixel formats from standard RGB up to RGBAF
>> and complex numbers.
>>
>> Now I like to add support for buffer interface to my code to support
>> zero-copy access of pixel data from NumPy. The wiki describes several
>> ways to access objects through the new and old Python buffer interface.
>> However I'm unable to find an example how to *implement* the buffer
>> interface.
>
> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
> your cdef class (don't know if it's documented but you can see example
> in tests/run/buffer.pyx).
>
>
>>
>> Memory alignment makes the buffer interface more complicated, too.
>> FreeImage stores the raw pixels non-contiguously. The lines are memory
>> aligned (usually 32bit). For example a standard RGB image with 1 byte
>> per pixel and size of 2*3 pixels has a memory layout of
>>
>> BGRBGRxxBGRBGRxxBGRBGRxx
>>
>> on a little-endian machine. 'xx' are the two alignment bytes.
>
(Continue reading)

Christian Heimes | 17 Jul 2012 22:55
Picon

Re: [Cython] 2d buffer interface with aligned data

Am 17.07.2012 18:55, schrieb Dag Sverre Seljebotn:
> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
> your cdef class (don't know if it's documented but you can see example
> in tests/run/buffer.pyx).

The new buffer interface from PEP 3118 is only available for Python 2.6
and newer. I was hoping for some abstraction layer in Cython. Well, I
don't have to support Python 2.5 and older. Thanks for the hint!

> This is easily supported; above you would let
> 
> ndim = 2
> strides = [8, 1]
> shape = [2, 3]
> itemsize = 1
> 
> The alignment bytes are skipped simply because shape[0] * itemsize <
> strides[0].

Either I'm doing something wrong or I found a Cython bug. I've attached
two files. The output is unexpected and looks like something is
accessing uninitialized memory:

format BBB
itemsize 3
ndim 2
readonly True
shape (140704200676960L, 140704199917920L)
strides (2L, 140704196619665L)
suboffsets None
(Continue reading)

Nathaniel Smith | 18 Jul 2012 01:53
Picon
Favicon

Re: [Cython] 2d buffer interface with aligned data

On Tue, Jul 17, 2012 at 9:55 PM, Christian Heimes <lists@...> wrote:
> Am 17.07.2012 18:55, schrieb Dag Sverre Seljebotn:
>> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
>> your cdef class (don't know if it's documented but you can see example
>> in tests/run/buffer.pyx).
>
> The new buffer interface from PEP 3118 is only available for Python 2.6
> and newer. I was hoping for some abstraction layer in Cython. Well, I
> don't have to support Python 2.5 and older. Thanks for the hint!

If you're worried about supporting older Python versions then the
simplest thing is probably to just implement the __array_interface__
or __array_struct__ interface. They're pretty trivial, and while not
as standards-compliant and Correct as PEP3118, they'll work just fine
with numpy across any version of Python. (Of course you can also
implement both PEP3118 and __array_struct__ together.)

http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

-n
Stefan Behnel | 18 Jul 2012 07:18
Picon
Favicon

Re: [Cython] 2d buffer interface with aligned data

Christian Heimes, 17.07.2012 22:55:
> Am 17.07.2012 18:55, schrieb Dag Sverre Seljebotn:
>> Read PEP 3118. Then implement __getbuffer__ and __releasebuffer__ in
>> your cdef class (don't know if it's documented but you can see example
>> in tests/run/buffer.pyx).
> 
> The new buffer interface from PEP 3118 is only available for Python 2.6
> and newer. I was hoping for some abstraction layer in Cython. Well, I
> don't have to support Python 2.5 and older. Thanks for the hint!

Cython supports the buffer interface also in older Python versions as long
as you implement your buffer methods in a .pxd file as part of the
extension type declaration and cimport from that in the modules where you
use it. It's a pure Cython feature though, other extensions (and Python
code, obviously) won't see it.

BTW, note that your question is better suited for the Cython users mailing
list than the core developer mailing list.

Stefan
Christian Heimes | 18 Jul 2012 11:55
Picon

Re: [Cython] 2d buffer interface with aligned data

Am 18.07.2012 07:18, schrieb Stefan Behnel:
> BTW, note that your question is better suited for the Cython users mailing
> list than the core developer mailing list.

I'm sorry. I didn't know Cython had a user list, too. Thanks Stefan!

Christian
Christian Heimes | 18 Jul 2012 11:55
Picon

Re: [Cython] 2d buffer interface with aligned data

Am 17.07.2012 22:55, schrieb Christian Heimes:
> Either I'm doing something wrong or I found a Cython bug. I've attached
> two files. The output is unexpected and looks like something is
> accessing uninitialized memory:

For the record:
It's my fault. The shade and stripes Py_ssize_t* arrays can't be local
variables. I've to malloc() two arrays and free() them in
__releasebuffer__(). Cython is so easy to use, it makes me forget all
this little annoying C things.

Christian

Dag Sverre Seljebotn | 18 Jul 2012 12:24
Picon
Picon
Gravatar

Re: [Cython] 2d buffer interface with aligned data

If the image will always be 2D, you can add

cdef Py_ssize_t shape[2]

To your cdef class and assign self.shape to the Py_buffer. This is a bit more efficient.

Dag
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Christian Heimes <lists-RPk2yecm54SELgA04lAiVw@public.gmane.org> wrote:
Am 17.07.2012 22:55, schrieb Christian Heimes:
> Either I'm doing something wrong or I found a Cython bug. I've attached
> two files. The output is unexpected and looks like something is
> accessing uninitialized memory:

For the record:
It's my fault. The shade and stripes Py_ssize_t* arrays can't be local
variables. I've to malloc() two arrays and free() them in
__releasebuffer__(). Cython is so easy to use, it makes me forget all
this little annoying C things.

Christian


cython-devel mailing list
cython-devel-+ZN9ApsXKcEdnm+yROfE0A@public.gmane.org
http://mail.python.org/mailman/listinfo/cython-devel
<div>If the image will always be 2D, you can add<br><br>
cdef Py_ssize_t shape[2]<br><br>
To your cdef class and assign self.shape to the Py_buffer. This is a bit more efficient.<br><br>
Dag<br>
-- <br>
Sent from my Android phone with K-9 Mail. Please excuse my brevity.<br><br><div class="gmail_quote">Christian Heimes &lt;lists@...&gt; wrote:<blockquote class="gmail_quote">
Am 17.07.2012 22:55, schrieb Christian Heimes:<br>&gt; Either I'm doing something wrong or I found a Cython bug. I've attached<br>&gt; two files. The output is unexpected and looks like something is<br>&gt; accessing uninitialized memory:<br><br>For the record:<br>It's my fault. The shade and stripes Py_ssize_t* arrays can't be local<br>variables. I've to malloc() two arrays and free() them in<br>__releasebuffer__(). Cython is so easy to use, it makes me forget all<br>this little annoying C things.<br><br>Christian<br><br><br>cython-devel mailing list<br>cython-devel@...<br><a href="http://mail.python.org/mailman/listinfo/cython-devel">http://mail.python.org/mailman/listinfo/cython-devel</a><br>
</blockquote>
</div>
</div>

Gmane