Questions, help, and general non-development discussion related to Cython. ()

Robert Bradshaw | 7 Apr 2010 19:36

Re: C++ Wrapper Confusion

On Apr 7, 2010, at 10:06 AM, SevenThunders wrote:

> I am trying to wrap an existing C++ class in cython.  I've had some
> success in the past wrapping a C library,  however I find the state of
> the C++ documentation rather confusing.  I believe there have been
> some prior posts on this issue.  My first question is what wrapping
> syntax is actually suported in Cython 0.12.1?  The online
> documentation for version 0.12 has the 'older' syntax here,
> http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html .
>
> Some earlier posts suggest that this is deprecated and that we should
> use the rather terse discussion here:
> http://wiki.cython.org/WrappingCPlusPlus

The new syntax is not available in Cython 0.12.1, but will be in  
Cython 0.13 which should be out soon (yes, soon has dragged on for a  
month or two more than we would have liked, but I think the issues  
uncovered by Sage have been resolved (Craig?) and as has been  
discussed, if we need to postpone closures due to a refcounting bug,  
so be it).

> Although the effort to provide C++ support is greatly appreciated,
> there are a few things missing from this documentation.

Though it has been getting better, there is still *lot* of things  
missing from this documentation.

> There is no discussion about how to define the extension class

See http://docs.cython.org/src/tutorial/cdef_classes.html

(Continue reading)

headers

SevenThunders | 7 Apr 2010 22:19

Re: C++ Wrapper Confusion

OK well that makes sense then.  I erroneously got the impression that
there was stealth support
for the new syntax.

I downloaded a development version and attempted to use it to start
wrapping a somewhat complex C++ class (that is fortunately no longer
templated).  I pointed my PYTHONPATH to the install directory and the
linux PATH variable to the underlying bin directory.  I ran into some
issues,  some of which I could overcome, but the final issue is
baffling to me.

First I found that cython doesn't seem to handle type declarations of
the form std::string.  Or at least whenever I used it as a type
signature in a function call it complained of a syntax error.

I then found that it doesn't like the ~ character in the destructor
declaration.

Next,  when wrapping my extension class,  it's single data attribute
is the pointer to the underlying C++ class.  But how do you
dereference that pointer in cython code?  After all a lot of my
methods take  (const gpumat &) as the type signature.  Well it seems
you have to drop the const declaration and then if you erroneously try
to pass *gptr as an argument, it treats it as a star argument within
python syntax,  which I suppose is a kind of unbracket or list
extraction operator  (reasonable I suppose).  So I think you have to
do something like self.gptr[0]  as an argument.

I then found that cuda's dim3 struct type had to be declared
explicitly for some reason.

(Continue reading)

headers

Robert Bradshaw | 7 Apr 2010 22:51

Re: Re: C++ Wrapper Confusion

On Apr 7, 2010, at 1:19 PM, SevenThunders wrote:

> OK well that makes sense then.  I erroneously got the impression that
> there was stealth support
> for the new syntax.
>
> I downloaded a development version and attempted to use it to start
> wrapping a somewhat complex C++ class (that is fortunately no longer
> templated).  I pointed my PYTHONPATH to the install directory and the
> linux PATH variable to the underlying bin directory.  I ran into some
> issues,  some of which I could overcome, but the final issue is
> baffling to me.
>
> First I found that cython doesn't seem to handle type declarations of
> the form std::string.  Or at least whenever I used it as a type
> signature in a function call it complained of a syntax error.

You can't use C++ :: to denote namespaces, as :: has a completely  
meaning in Python. Instead, use

cdef extern from "string" namespace std:
     cdef cppclass string

(On that note, one thing that we don't yet provide is any nice Python  
<-> C++ string conversion.)

> I then found that it doesn't like the ~ character in the destructor
> declaration.

Use del (which is they Python syntax). Again, the ~ operator already

(Continue reading)

headers

SevenThunders | 8 Apr 2010 01:06

Re: C++ Wrapper Confusion

Is this simple enough?

--------------badmat.pyx ---------------
import numpy as np
cimport numpy as np
import sys

cdef extern from "gpumat.h" :
    ctypedef struct dim3:
        int x, y, z
    ctypedef enum MatrixType :
        Realmatrix = 0
        Complexmatrix=1
        Cuppermatrix = 2
    ctypedef struct gpumat
    void fromgpu(gpumat &gin, float *hval)
    cdef cppclass gpumat:
        int nrows
        int ncols
        int ld
        MatrixType mtype
        float *val
        dim3 block
        dim3 grid
        dim3 residual
        gpumat()

-----------------------------------------
The default constructor line gpumat()  causes cython to complain.

(Continue reading)

headers

Robert Bradshaw | 8 Apr 2010 01:30

Re: Re: C++ Wrapper Confusion

On Apr 7, 2010, at 4:06 PM, SevenThunders wrote:

> Is this simple enough?
>
> --------------badmat.pyx ---------------
> import numpy as np
> cimport numpy as np
> import sys
>
>
>
> cdef extern from "gpumat.h" :
>    ctypedef struct dim3:
>        int x, y, z
>    ctypedef enum MatrixType :
>        Realmatrix = 0
>        Complexmatrix=1
>        Cuppermatrix = 2
>    ctypedef struct gpumat
>    void fromgpu(gpumat &gin, float *hval)
>    cdef cppclass gpumat:
>        int nrows
>        int ncols
>        int ld
>        MatrixType mtype
>        float *val
>        dim3 block
>        dim3 grid
>        dim3 residual
>        gpumat()

(Continue reading)

headers

SevenThunders | 8 Apr 2010 02:25

Re: C++ Wrapper Confusion

There is often no simple way to get around the need for forward
declarations,  though I am far from a C++ expert,  hacker mainly.  For
example my existing gpumat class has cupper as a subclass.  Several
methods return a cupper type inside gpumat,  ie the QR decomposition
method qr().   Several methods in cupper return or require knowledge
of gpumat.  The easy way around this is to forward declare cupper.

Is there another way to forward declare classes or C++ methods in
Cython?  I know the :: notation won't fly.

As for my numpy problems,  I think they occur because it can't find
the numpy include files.  I probably need to add them to the setup.py
distutils mechanism.  So now to figure out where my distro has put my
python installation files...

On Apr 7, 7:30 pm, Robert Bradshaw <rober... <at> math.washington.edu>
wrote:
> On Apr 7, 2010, at 4:06 PM, SevenThunders wrote:
>
>
>
> > Is this simple enough?
>
> > --------------badmat.pyx ---------------
> > import numpy as np
> > cimport numpy as np
> > import sys
>
> > cdef extern from "gpumat.h" :
> >    ctypedef struct dim3:

(Continue reading)

headers

Robert Bradshaw | 8 Apr 2010 06:53

Re: Re: C++ Wrapper Confusion

On Apr 7, 2010, at 5:25 PM, SevenThunders wrote:

> There is often no simple way to get around the need for forward
> declarations,  though I am far from a C++ expert,  hacker mainly.  For
> example my existing gpumat class has cupper as a subclass.  Several
> methods return a cupper type inside gpumat,  ie the QR decomposition
> method qr().   Several methods in cupper return or require knowledge
> of gpumat.  The easy way around this is to forward declare cupper.

Forward declarations should work, and they work for structs, but  
something seems to be (sometimes?) broken for cpp classes. I'd fix  
this if I had time, but I simply don't right now.

> Is there another way to forward declare classes or C++ methods in
> Cython?  I know the :: notation won't fly.
>
> As for my numpy problems,  I think they occur because it can't find
> the numpy include files.  I probably need to add them to the setup.py
> distutils mechanism.

That would do it.

> So now to figure out where my distro has put my
> python installation files...

That's where the numpy.get_include() method comes in handy, use it  
right in your setup.py.

- Robert

(Continue reading)

headers

SevenThunders | 8 Apr 2010 19:38

Re: C++ Wrapper Confusion

Just from the nature of the bug it almost seems as though some
confusion occurs between a forward declared class and it's
constructor.

Unfortunately I now have another show stopper bug that has prevented
me from moving forward with this.
http://groups.google.com/group/cython-users/browse_thread/thread/817a6a8add296d11

I filed a bug report here:
http://trac.cython.org/cython_trac/ticket/524

There seems to be a whole class of bugs similar to this.  If I could
figure a work around I might be able to move forward.  Perhaps it's  a
syntax error on my part...

On Apr 8, 12:53 am, Robert Bradshaw <rober... <at> math.washington.edu>
wrote:
> On Apr 7, 2010, at 5:25 PM, SevenThunders wrote:
>
> > There is often no simple way to get around the need for forward
> > declarations,  though I am far from a C++ expert,  hacker mainly.  For
> > example my existing gpumat class has cupper as a subclass.  Several
> > methods return a cupper type inside gpumat,  ie the QR decomposition
> > method qr().   Several methods in cupper return or require knowledge
> > of gpumat.  The easy way around this is to forward declare cupper.
>
> Forward declarations should work, and they work for structs, but  
> something seems to be (sometimes?) broken for cpp classes. I'd fix  
> this if I had time, but I simply don't right now.
>

(Continue reading)

headers

Robert Bradshaw | 9 Apr 2010 04:18

Re: Re: C++ Wrapper Confusion

On Apr 8, 2010, at 10:38 AM, SevenThunders wrote:

> Just from the nature of the bug it almost seems as though some
> confusion occurs between a forward declared class and it's
> constructor.
>
> Unfortunately I now have another show stopper bug that has prevented
> me from moving forward with this.
> http://groups.google.com/group/cython-users/browse_thread/thread/817a6a8add296d11
>
> I filed a bug report here:
> http://trac.cython.org/cython_trac/ticket/524
>
> There seems to be a whole class of bugs similar to this.  If I could
> figure a work around I might be able to move forward.  Perhaps it's  a
> syntax error on my part...

Looks like
badmat(unsigned i, unsigned j, unsigned nld)
was used but not defined. We *should* be raising an error here.

> On Apr 8, 12:53 am, Robert Bradshaw <rober... <at> math.washington.edu>
> wrote:
>> On Apr 7, 2010, at 5:25 PM, SevenThunders wrote:
>>
>>> There is often no simple way to get around the need for forward
>>> declarations,  though I am far from a C++ expert,  hacker mainly.   
>>> For
>>> example my existing gpumat class has cupper as a subclass.  Several
>>> methods return a cupper type inside gpumat,  ie the QR decomposition

(Continue reading)

headers

Robert Bradshaw | 9 Apr 2010 04:55

Re: Re: C++ Wrapper Confusion

On Apr 8, 2010, at 7:18 PM, Robert Bradshaw wrote:

> On Apr 8, 2010, at 10:38 AM, SevenThunders wrote:
>
>> Just from the nature of the bug it almost seems as though some
>> confusion occurs between a forward declared class and it's
>> constructor.
>>
>> Unfortunately I now have another show stopper bug that has prevented
>> me from moving forward with this.
>> http://groups.google.com/group/cython-users/browse_thread/thread/817a6a8add296d11
>>
>> I filed a bug report here:
>> http://trac.cython.org/cython_trac/ticket/524
>>
>> There seems to be a whole class of bugs similar to this.  If I could
>> figure a work around I might be able to move forward.  Perhaps  
>> it's  a
>> syntax error on my part...
>
> Looks like
> badmat(unsigned i, unsigned j, unsigned nld)
> was used but not defined. We *should* be raising an error here.

http://hg.cython.org/cython-devel/rev/78e6dc767358

Looks like error reporting for C++ wrapping isn't being tested at all : 
(.

- Robert

(Continue reading)

headers

SevenThunders | 10 Apr 2010 01:39

Re: C++ Wrapper Confusion

I should have noticed that when I hastily put together my test case.
Unfortunately for me though my original .pyx file did not have that
error,  but it still generates an error that I can not reproduce right
now with a simple test case.  It might require my somewhat complex and
proprietary include file gpumat.h.  The include file compiles under
gcc just fine.

Here is the .pyx file I'm using so far
-------------- gpumat.pyx ---------------------------------
import numpy as np
cimport numpy as np
import sys

cdef extern from "gpumat.h" :
    ctypedef struct dim3:
        int x, y, z
    ctypedef enum MatrixType :
        Realmatrix = 0
        Complexmatrix=1
        Cuppermatrix = 2
    cdef cppclass gpumat:
        int nrows
        int ncols
        int ld
        MatrixType mtype
        float *val
        dim3 block
        dim3 grid
        dim3 residual
        gpumat()

(Continue reading)

headers

Lisandro Dalcin | 10 Apr 2010 03:56

Re: Re: C++ Wrapper Confusion

On 9 April 2010 20:39, SevenThunders <mattcbro <at> earthlink.net> wrote:
> pygpumat.cpp: In function ‘PyObject* __Pyx_Import(PyObject*,
> PyObject*)’:
> pygpumat.cpp:4346: error: expected unqualified-id before ‘=’ token
> pygpumat.cpp:4352: error: expected primary-expression before ‘=’ token
>
> The section of the .cpp file with the error looks like this:
>
> ------------------pygpumat.cpp -------------------------
> ...
> static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list) {
>    PyObject *__import__ = 0; // line 4346,  the first offending line
>    PyObject *empty_list = 0;

Could you try to apply the patch below to Cython sources and give a
try? We should not use __import__ for a local variable name 

diff -r b442fb71c1de Cython/Compiler/ExprNodes.py
--- a/Cython/Compiler/ExprNodes.py	Sat Mar 20 18:07:43 2010 +0100
+++ b/Cython/Compiler/ExprNodes.py	Fri Apr 09 22:55:08 2010 -0300
 <at>  <at>  -6287,14 +6287,14  <at>  <at> 
 """,
 impl = """
 static PyObject *__Pyx_Import(PyObject *name, PyObject *from_list) {
-    PyObject *__import__ = 0;
+    PyObject *py__import__ = 0;
     PyObject *empty_list = 0;
     PyObject *module = 0;
     PyObject *global_dict = 0;
     PyObject *empty_dict = 0;

(Continue reading)

headers

SevenThunders | 12 Apr 2010 22:27

Re: C++ Wrapper Confusion

The good news is that the patch allows the cythoned code to be
compiled.  The patch appears to comment out a bunch of code,  if I did
it correctly.  The bad news is that my code segfaults.  It segfaults
prior to doing anything interesting like running a kernel on my gpu
device.  Rather it appears to fail when I attempt to assign a NULL to
a float pointer.  In particular for the class that wraps the float
pointer,  defined in cython like this:
---------------------------------
# Python wrapper for a float pointer
cdef class FloatPtr :
    cdef float *ptr
    cdef setf(FloatPtr self, float *fp) :
        self.ptr = fp
    cdef float *getf(FloatPtr self) :
        return self.ptr
-----------------------------------

It segfaults when I make this cython call:
------------------------
cdef FloatPtr f
f.setf(NULL)

-----------------------------
Did I do something wrong here?

On Apr 9, 9:56 pm, Lisandro Dalcin <dalc... <at> gmail.com> wrote:
> On 9 April 2010 20:39, SevenThunders <mattc... <at> earthlink.net> wrote:
>
> > pygpumat.cpp: In function ‘PyObject* __Pyx_Import(PyObject*,
> > PyObject*)’:

(Continue reading)

headers

Robert Bradshaw | 12 Apr 2010 23:01

Re: Re: C++ Wrapper Confusion

On Apr 12, 2010, at 1:27 PM, SevenThunders wrote:

> The good news is that the patch allows the cythoned code to be
> compiled.  The patch appears to comment out a bunch of code,  if I did
> it correctly.

Excellent.

> The bad news is that my code segfaults.  It segfaults
> prior to doing anything interesting like running a kernel on my gpu
> device.  Rather it appears to fail when I attempt to assign a NULL to
> a float pointer.  In particular for the class that wraps the float
> pointer,  defined in cython like this:
> ---------------------------------
> # Python wrapper for a float pointer
> cdef class FloatPtr :
>    cdef float *ptr
>    cdef setf(FloatPtr self, float *fp) :
>        self.ptr = fp
>    cdef float *getf(FloatPtr self) :
>        return self.ptr
> -----------------------------------
>
> It segfaults when I make this cython call:
> ------------------------
> cdef FloatPtr f
> f.setf(NULL)
>
> -----------------------------
> Did I do something wrong here?

(Continue reading)

headers

Lisandro Dalcin | 12 Apr 2010 23:04

Re: Re: C++ Wrapper Confusion

On 12 April 2010 18:01, Robert Bradshaw <robertwb <at> math.washington.edu> wrote:
> On Apr 12, 2010, at 1:27 PM, SevenThunders wrote:
>
>> The good news is that the patch allows the cythoned code to be
>> compiled.  The patch appears to comment out a bunch of code,  if I did
>> it correctly.
>
> Excellent.
>

FYI, http://hg.cython.org/cython-devel/rev/fb26a68ec463

-- 
Lisandro Dalcin
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

--

-- 
To unsubscribe, reply using "remove me" as the subject.

headers

SevenThunders | 13 Apr 2010 20:38

Re: C++ Wrapper Confusion


On Apr 12, 5:04 pm, Lisandro Dalcin <dalc... <at> gmail.com> wrote:
> On 12 April 2010 18:01, Robert Bradshaw <rober... <at> math.washington.edu> wrote:
>
> > On Apr 12, 2010, at 1:27 PM, SevenThunders wrote:
>
> >> The good news is that the patch allows the cythoned code to be
> >> compiled.  The patch appears to comment out a bunch of code,  if I did
> >> it correctly.
>
> > Excellent.
>
> FYI,http://hg.cython.org/cython-devel/rev/fb26a68ec463
>

Was this link supposed to have the corrected ExprNodes.py?  I
downloaded a later snapshot from mercurial and got the same error,
and the error persists even when using the
raw file copy from the link you provided.  I was trying to install it
on another computer.

I'll have to get my hands on my patched version on another machine and
see what the differences are.
> --
> Lisandro Dalcin
> ---------------
> Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
> Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
> Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
> PTLC - Güemes 3450, (3000) Santa Fe, Argentina

(Continue reading)

headers

Lisandro Dalcin | 15 Apr 2010 05:41

Re: Re: C++ Wrapper Confusion

On 13 April 2010 15:38, SevenThunders <mattcbro <at> earthlink.net> wrote:
>
>
> On Apr 12, 5:04 pm, Lisandro Dalcin <dalc... <at> gmail.com> wrote:
>> On 12 April 2010 18:01, Robert Bradshaw <rober... <at> math.washington.edu> wrote:
>>
>> > On Apr 12, 2010, at 1:27 PM, SevenThunders wrote:
>>
>> >> The good news is that the patch allows the cythoned code to be
>> >> compiled.  The patch appears to comment out a bunch of code,  if I did
>> >> it correctly.
>>
>> > Excellent.
>>
>> FYI,http://hg.cython.org/cython-devel/rev/fb26a68ec463
>>
>
> Was this link supposed to have the corrected ExprNodes.py?  I
> downloaded a later snapshot from mercurial and got the same error,
> and the error persists even when using the
> raw file copy from the link you provided.  I was trying to install it
> on another computer.
>

Sorry, at this point I do not know the status of this... Does a
checkout of cython-devel repo works or not for your?

--

-- 
Lisandro Dalcin
---------------

(Continue reading)

headers

SevenThunders | 16 Apr 2010 20:11

Re: C++ Wrapper Confusion

Sorry about that Lisandro.  I think the problem was on my side.  I
forgot to touch my source code and it was picking up the old .cpp
file. It seems to be working now.  Thanks again for getting a patch in
so quickly.

If given some time on Monday I might try to see if it will run under
windows as well,  and I plan to start wrapping some more complex stuff
from my source,  including some operator overloads and so forth.

On Apr 14, 11:41 pm, Lisandro Dalcin <dalc... <at> gmail.com> wrote:
> On 13 April 2010 15:38, SevenThunders <mattc... <at> earthlink.net> wrote:
>
>
>
>
>
> > On Apr 12, 5:04 pm, Lisandro Dalcin <dalc... <at> gmail.com> wrote:
> >> On 12 April 2010 18:01, Robert Bradshaw <rober... <at> math.washington.edu> wrote:
>
> >> > On Apr 12, 2010, at 1:27 PM, SevenThunders wrote:
>
> >> >> The good news is that the patch allows the cythoned code to be
> >> >> compiled.  The patch appears to comment out a bunch of code,  if I did
> >> >> it correctly.
>
> >> > Excellent.
>
> >> FYI,http://hg.cython.org/cython-devel/rev/fb26a68ec463
>
> > Was this link supposed to have the corrected ExprNodes.py?  I

(Continue reading)

headers

SevenThunders | 13 Apr 2010 06:01

Re: C++ Wrapper Confusion

Thanks to all for the help and the incredibly rapid fix time.  Cython
is a very impressive tool IMHO.  My simple wrapper start is actually
working Hoorah!

I just have a couple of comments with regards to some pitfalls I ran
into in case it may help someone else.
Prior to carefully reading Robert's fix to my attempt to wrap a
pointer  I decided that my problem was a lack of a __cinit__
initializer in FloatPtr().  So I did this:

------------------------
# Python wrapper for a float pointer
cdef class FloatPtr :
    cdef float *ptr
    def __cinit__(self) :
        print "Initializing pointer to NULL"
        self.ptr = NULL
    cdef setf(FloatPtr self, float *fp) :
        print "Setting pointer to %ld" % <long int>fp
        self.ptr = fp
    cdef float *getf(FloatPtr self) :
        return self.ptr

-----------------------

This compiled and ran for a ways without segfaulting, but in fact I
was not initializing my pointer to null
when I did this.
---------------------------
    cdef FloatPtr f = FloatPtr

(Continue reading)

headers

Lisandro Dalcin | 13 Apr 2010 16:36

Re: Re: C++ Wrapper Confusion

On 13 April 2010 01:01, SevenThunders <mattcbro <at> earthlink.net> wrote:
>
> My suggestion is to create an automated way to wrap pointers in a fake
> Python class,  similar in functionality to the FloatPtr class above,
> but with some syntactic sugar.

Do you have any syntax in mind?

>  In particular I don't want to write
> gptr[0].nrows,  I want to write gptr.nrows,  which would be possible
> if gptr were a 'class'.

cdef XXX *gptr = self.gptr
gptr.rows ...
gptr.cols ....

Just a single extra line (though you would need it in every [c][p]def method)...

PS: This works because Cython converts gptr.nrows in Cython code to
gptr->rows in the generated C code.

--

-- 
Lisandro Dalcin
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

(Continue reading)

headers

Robert Bradshaw | 13 Apr 2010 20:52

Re: Re: C++ Wrapper Confusion

On Apr 12, 2010, at 9:01 PM, SevenThunders wrote:

> Thanks to all for the help and the incredibly rapid fix time.  Cython
> is a very impressive tool IMHO.  My simple wrapper start is actually
> working Hoorah!
>
> I just have a couple of comments with regards to some pitfalls I ran
> into in case it may help someone else.
> Prior to carefully reading Robert's fix to my attempt to wrap a
> pointer  I decided that my problem was a lack of a __cinit__
> initializer in FloatPtr().  So I did this:
>
> ------------------------
> # Python wrapper for a float pointer
> cdef class FloatPtr :
>    cdef float *ptr
>    def __cinit__(self) :
>        print "Initializing pointer to NULL"
>        self.ptr = NULL
>    cdef setf(FloatPtr self, float *fp) :
>        print "Setting pointer to %ld" % <long int>fp
>        self.ptr = fp
>    cdef float *getf(FloatPtr self) :
>        return self.ptr
>
> -----------------------
>
> This compiled and ran for a ways without segfaulting, but in fact I
> was not initializing my pointer to null
> when I did this.

(Continue reading)

headers

SevenThunders | 14 Apr 2010 00:09

Re: C++ Wrapper Confusion

My code wraps an interface to some GPU accelerated kernels that use
CUDA.  The GPU memory has pointers that are
'wrapped' as float pointers on the host CPU.  Thus those float
pointers can not be used in numpy or
by anything else except for the underlying gpu device.

I was talking about a general capability,  an automated way to wrap
pointers of any kind.  Probably C++ class pointers would be the
primary use.

If I can already use self.gptr.nrows  as is,  without doing
self.gptr[0].nrows,  then I suppose the syntactic sugar
issue is already addressed,  assuming that that interpretation will
remain in future versions of cython.

However I can imagine a lot of utility for a generic pointer wrapper
depending on how far you want to take it.
Wouldn't it be nice at the Python level if you could index them,
compare them,  use them for iterators,  set them, resize them, and
at the cython level extend them.  If you automatically wrapped a class
pointer inside one of these and then gave the programmer the ability
to extend them,  it might be a bit easier to write the extension
class.  One could then incrementally add capabilities as cython
progresses.  For example a nice initial
capability would be to automatically wrap declared constructors and
provide a delete method to be called when the reference count goes to
zero
as the current the current __dealloc__ attribute does.

If I understand what you are saying,  when someone declares

(Continue reading)

headers

Robert Bradshaw | 14 Apr 2010 01:56

Re: Re: C++ Wrapper Confusion

On Apr 13, 2010, at 3:09 PM, SevenThunders wrote:

> My code wraps an interface to some GPU accelerated kernels that use
> CUDA.  The GPU memory has pointers that are
> 'wrapped' as float pointers on the host CPU.  Thus those float
> pointers can not be used in numpy or
> by anything else except for the underlying gpu device.
>
> I was talking about a general capability,  an automated way to wrap
> pointers of any kind.  Probably C++ class pointers would be the
> primary use.

I think you're conflating two different ideas here--the fact that C/C+ 
+ uses pointers for both arrays and indirection is (to me at least)  
more of implementation detail, though, of course, this being C the  
boundaries are quite fuzzy.

I also think it would be useful to have a native array type, something  
that behaves like a Python list but is backed by a float* or int* or  
whatever. Ideally, this could be used to eliminate the need for users  
to manually use malloc/free in 90% of the use cases. (For your use  
case, of course, you would be setting the GPU pointer manually.)  
Memory ownership can get messy. We already have NumPy and http://docs.python.org/c-api/cobject.html 
  , and of course bytes/str for char*, but I think there's a case to  
be made for something in-between. In this case I'm not sure how much  
use it would be to "extend" such an object--it would mostly be used  
implicitly when converting to/from Python. We also try to avoid adding  
a lot of magic syntax, as that is one of the things that makes C++ so  
hard to learn (and read!) compared to, say, Python. Type parameters  
for cdef classes would also go a long way towards a solution to this

(Continue reading)

headers

SevenThunders | 15 Apr 2010 05:08

Re: C++ Wrapper Confusion


>
> I think you're conflating two different ideas here--the fact that C/C+
> + uses pointers for both arrays and indirection is (to me at least)  
> more of implementation detail, though, of course, this being C the  
> boundaries are quite fuzzy.

I am conflating two ideas here.  But of course as you point out so
does C++.
>
> I also think it would be useful to have a native array type, something  
> that behaves like a Python list but is backed by a float* or int* or  
> whatever. Ideally, this could be used to eliminate the need for users  
> to manually use malloc/free in 90% of the use cases. (For your use  
> case, of course, you would be setting the GPU pointer manually.)  
> Memory ownership can get messy. We already have NumPy andhttp://docs.python.org/c-api/cobject.html
>   , and of course bytes/str for char*, but I think there's a case to  
> be made for something in-between. In this case I'm not sure how much  
> use it would be to "extend" such an object--it would mostly be used  
> implicitly when converting to/from Python.

I agree that's where you want to be, but I was thinking of a kind of
spiral development technique for getting there,  in the form of some
kind of minimal auto-wrap functionality.  The first obvious case is to
just auto-wrap the pointer type.  If one already has a PyCobject type,
then perhaps that's the infrastructure to use.

If you just provide minimal functionality however,  then you want the
user to be able to fill in the functionality as needed.

(Continue reading)

headers

SevenThunders | 14 Apr 2010 01:30

Re: C++ Wrapper Confusion

My apologies if this is a repeat from what I sent earlier.  Google
groups seemed to eat my prior response.
The float pointer I'm wrapping is what's called a 'device pointer'  in
CUDA.  It has no meaning outside of the
GPU device that it was created for.

If I understanding your meaning correctly if I declare
cdef Foo *fptr

for a cppclass Foo,  I can then freely use  ptr.attr  in lieu of ptr-
>attr or ptr[0].attr?  That would solve the syntax sugar issue.

What I was alluding too earler was an idea for automating minimal
functionality in some kind of pointer wrapper.  What if we want to
carry around a Foo pointer in Python even if it's
to just pass into another function?  I'm assuming that we can't do
cpdef Foo *fptr

since *ptr is not Python syntax or more precisely pointer to Foo is
not a defined python type.  What if cython supported a generic pointer
class Ptr with some 'magic' syntax.  Say something like

cpdef *fptr = Foo(a1,a2,a3)

where a1, a2, a3 are constructor arguments.  In this case  the pointer
nature of fptr tells Cython we want to create an inherited class Foo
that extends some base pointer wrapper class (say Ptr).
Maybe we indicate this by
cpdef *fptr = Ptr(Foo(a1,a2,a3)).  In the latter syntax  Ptr would
return a class Foo that inherits a base pointer class with some

(Continue reading)