wlavrijsen | 23 Apr 07:08 2012

cppyy: C++ bindings for PyPy

Hi,

as detailed in a couple of blog posts in the past (*), for some time now we
have been working on making C++ bindings through the Reflex package available
on PyPy, in the form of the "cppyy" module. Software is never done, and that
is true also in this case, but it has reached a level of maturity at which it
can be said to be usable. Initial docs are now up on:

    http://doc.pypy.org/en/latest/cppyy.html

with instructions on how to setup the reflex-support branch and how to test
it out. The documentation will be updated with more advanced and detailed
topics in the next couple of weeks or so.

There's a sizable (non-PyPy) set of unit tests that still need to be worked
through, thus development will steadily continue at its current pace. Still,
if you find that cppyy almost works for you except for a particular feature,
feel free to ask for it to be prioritized.

The biggest obstacle for most people (that are not in the field of HEP) will
be the current set of dependencies. Although the dependency set for cppyy is
really only Reflex, which could be distributed separately, for the CPython
equivalent code, the dependency set is a large portion of the ROOT class
library. The medium-term plan then is to use Cling, which is based on CLang
from llvm (http://root.cern.ch/drupal/content/cling) as the back-end. For
this, there will be a PyCling on CPython, and all as stand-alone projects to
remove the large dependency set.

More interesting though, having Cling on the back of the bindings allows
far more advanced features, such as dynamic setup of callbacks and cross
(Continue reading)

Maciej Fijalkowski | 23 Apr 08:28 2012
Picon

Re: cppyy: C++ bindings for PyPy

On Mon, Apr 23, 2012 at 7:08 AM, <wlavrijsen <at> lbl.gov> wrote:

Hi,

as detailed in a couple of blog posts in the past (*), for some time now we
have been working on making C++ bindings through the Reflex package available
on PyPy, in the form of the "cppyy" module. Software is never done, and that
is true also in this case, but it has reached a level of maturity at which it
can be said to be usable. Initial docs are now up on:

  http://doc.pypy.org/en/latest/cppyy.html

with instructions on how to setup the reflex-support branch and how to test
it out. The documentation will be updated with more advanced and detailed
topics in the next couple of weeks or so.

There's a sizable (non-PyPy) set of unit tests that still need to be worked
through, thus development will steadily continue at its current pace. Still,
if you find that cppyy almost works for you except for a particular feature,
feel free to ask for it to be prioritized.

The biggest obstacle for most people (that are not in the field of HEP) will
be the current set of dependencies. Although the dependency set for cppyy is
really only Reflex, which could be distributed separately, for the CPython
equivalent code, the dependency set is a large portion of the ROOT class
library. The medium-term plan then is to use Cling, which is based on CLang
from llvm (http://root.cern.ch/drupal/content/cling) as the back-end. For
this, there will be a PyCling on CPython, and all as stand-alone projects to
remove the large dependency set.

More interesting though, having Cling on the back of the bindings allows
far more advanced features, such as dynamic setup of callbacks and cross
language inheritance to put Python code into C++ frameworks. This has been
prototyped successfully in Clings predecessor (CINT), but would be very hard
to do in Reflex.

Of course, an important reason for pushing this code out to the community
somewhat early, is that it allows anyone so interested to start hacking on
it and help shape it!

Best regards,
          Wim

(*) http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.html
   http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries.html
--
WLavrijsen <at> lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev

Quick question - if it's mature, why not merge it to default? I presume it should be turned off, since there is a sizeable dependency, but still having it in default can be good.

Cheers,
fijal
_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
wlavrijsen | 23 Apr 08:42 2012

Re: cppyy: C++ bindings for PyPy

Hi Maciej,

> Quick question - if it's mature, why not merge it to default? I presume it
> should be turned off, since there is a sizeable dependency, but still
> having it in default can be good.

the dependency is the main issue: for it to build, it requires headers and
libs from Reflex. Completely off (i.e. the module not picked up) could work:
there are only a few changes that fall outside of the cppyy module proper
(mainly access to the raw internals of arrays and a rule for .cxx in the
generated Makefile). However, I'm not sure what the advantage would be, as
the branch is regularly kept up to date with merges from default?

Best regards,
            Wim
--

-- 
WLavrijsen <at> lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
Maciej Fijalkowski | 23 Apr 08:45 2012
Picon

Re: cppyy: C++ bindings for PyPy

On Mon, Apr 23, 2012 at 8:42 AM, <wlavrijsen <at> lbl.gov> wrote:

Hi Maciej,


Quick question - if it's mature, why not merge it to default? I presume it
should be turned off, since there is a sizeable dependency, but still
having it in default can be good.

the dependency is the main issue: for it to build, it requires headers and
libs from Reflex. Completely off (i.e. the module not picked up) could work:
there are only a few changes that fall outside of the cppyy module proper
(mainly access to the raw internals of arrays and a rule for .cxx in the
generated Makefile). However, I'm not sure what the advantage would be, as
the branch is regularly kept up to date with merges from default?

Best regards,
          Wim

It'll at the very least run tests nightly so we'll make sure we don't break it (we can install reflex on tannit). It's easier to maintain and I prefer done features on trunk
_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
wlavrijsen | 23 Apr 08:50 2012

Re: cppyy: C++ bindings for PyPy

Hi Maciej,

> It'll at the very least run tests nightly so we'll make sure we don't break
> it (we can install reflex on tannit).

that, to me, sounds great!

Best regards,
            Wim
--

-- 
WLavrijsen <at> lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
Alexander Pyattaev | 23 Apr 10:33 2012
Picon
Picon

Re: cppyy: C++ bindings for PyPy

An important question from userland - would it be benefitial to switch from SWIG to the new interface? When can that be done?

 

Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though.

 

Still, the point is that a new C++ interface is a cool thing to have, especially if it is native to pypy. Cant wait to test it out on a huge c++ extension

 

PS: I would have gladly tested an alpha-beta quality version of the lib (i have some unittests for SWIG bindings, so they should work with new lib also i suppose), but I can not build pypy from sources, not enuf RAM =) Maybe someone could send me a build with new feature for testing?

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
Amaury Forgeot d'Arc | 23 Apr 10:52 2012
Picon

Re: cppyy: C++ bindings for PyPy

2012/4/23 Alexander Pyattaev <alexander.pyattaev <at> tut.fi>
Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though.

Do you have a sample code for this issue?

--
Amaury Forgeot d'Arc
_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
Alex Pyattaev | 23 Apr 11:26 2012
Picon

Re: cppyy: C++ bindings for PyPy

Well, I have never been able to isolate it, and the project where it is triggered is rather close-sourced.

 

long story short - make up a SWIG object inside a function, pass it as a pointer to the swig-wrapped function, then call gc.collect(). After that swig calls object's destructor and if the C code is using it, youre screwd.

 

I am not sure if the bug persists on a small scale, however, it only shows during long simulation runs where thousands of objects like that are circulated.

If it ever happens on a smaller case I'll let you know.

Alex

maanantai 23 huhtikuu 2012 10:52:04 Amaury Forgeot d'Arc kirjoitti:

2012/4/23 Alexander Pyattaev <alexander.pyattaev <at> tut.fi>

Right now the main problem with SWIG is the object ownership, i.e. with pypy one has to set thisown=False for all objects to avoid crashes related to GC code. The problem occurs only for long-living objects though.


Do you have a sample code for this issue?


--
Amaury Forgeot d'Arc



_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
wlavrijsen | 23 Apr 16:52 2012

Re: cppyy: C++ bindings for PyPy

Hi Alexander,

> An important question from userland - would it be benefitial to switch from
> SWIG to the new interface? When can that be done?

there are no many major differences between SWIG and this on the Python side
(nothing that can't be captured in a compatibility layer). I'm thinking of
such things like cvar that is needed in SWIG, but not in cppyy, and the use
of templates which need to be named in the .i's in SWIG but are have a more
"instantiable" syntax in cppyy (for when the Cling backend becomes available).
Also, Reflex can parse more code than can SWIG, so you'd have to be careful
in which direction you use things.

> Right now the main problem with SWIG is the object ownership, i.e. with pypy
> one has to set thisown=False for all objects to avoid crashes related to GC
> code. The problem occurs only for long-living objects though.

The GC and cppyy work fine for those objects clearly owned by PyPy. Of course,
if the C++ side takes ownership, that would need to be patched up "by hand" or
with a rule (if the C++ API is consistent).

> PS: I would have gladly tested an alpha-beta quality version of the lib (i 
> have some unittests for SWIG bindings, so they should work with new lib also
> i suppose), but I can not build pypy from sources, not enuf RAM =) Maybe
> someone could send me a build with new feature for testing?

What distro (in particular, 32b or 64b and which version of libssl)? I've
never distributed a binary version of PyPy, but could give it a try.

Best regards,
            Wim
--

-- 
WLavrijsen <at> lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
Alex Pyattaev | 23 Apr 20:42 2012
Picon

Re: cppyy: C++ bindings for PyPy

Distro is latest gentoo, amd64 with sse2 and sse3 (core i5).

libssl is in 2 versions:

dev-libs/openssl-0.9.8u:0.9.8

dev-libs/openssl-1.0.0h:0

so pick one that suits you better. Anyway, default binary pypy works fine.

 

Currently i have only one typemap in my *.i file for swig, and it allows PyObject* pointers to go through to c++ side. Otherwise no special features are used, so migration should be pretty painless.

 

And of course if there are bugs they would most likely arise, especially if they are in memory management, because the c++ code is executed millions of times during the program run.

 

BR,

Alex

PS: sorry for using 2 emails - one is work other home, screwd company policy.

 

maanantai 23 huhtikuu 2012 07:52:49 wlavrijsen <at> lbl.gov kirjoitti:

> Hi Alexander,

>

> > An important question from userland - would it be benefitial to switch

> > from

> > SWIG to the new interface? When can that be done?

>

> there are no many major differences between SWIG and this on the Python side

> (nothing that can't be captured in a compatibility layer). I'm thinking of

> such things like cvar that is needed in SWIG, but not in cppyy, and the use

> of templates which need to be named in the .i's in SWIG but are have a more

> "instantiable" syntax in cppyy (for when the Cling backend becomes

> available). Also, Reflex can parse more code than can SWIG, so you'd have

> to be careful in which direction you use things.

>

> > Right now the main problem with SWIG is the object ownership, i.e. with

> > pypy one has to set thisown=False for all objects to avoid crashes

> > related to GC code. The problem occurs only for long-living objects

> > though.

>

> The GC and cppyy work fine for those objects clearly owned by PyPy. Of

> course, if the C++ side takes ownership, that would need to be patched up

> "by hand" or with a rule (if the C++ API is consistent).

>

> > PS: I would have gladly tested an alpha-beta quality version of the lib (i

> > have some unittests for SWIG bindings, so they should work with new lib

> > also i suppose), but I can not build pypy from sources, not enuf RAM =)

> > Maybe someone could send me a build with new feature for testing?

>

> What distro (in particular, 32b or 64b and which version of libssl)? I've

> never distributed a binary version of PyPy, but could give it a try.

>

> Best regards,

> Wim

_______________________________________________
pypy-dev mailing list
pypy-dev <at> python.org
http://mail.python.org/mailman/listinfo/pypy-dev
wlavrijsen | 23 Apr 21:20 2012

Re: cppyy: C++ bindings for PyPy

Hi Alex,

> Distro is latest gentoo, amd64 with sse2 and sse3 (core i5).

there's an older version of (Py)ROOT distributed with gentoo:

    http://packages.gentoo.org/package/sci-physics/root

Do you want to use that package, or pick up a more recent version (doesn't
matter for Reflex, I think, as it has been mostly stable)? The pypy binary
below is against 5.32, as I had it available.

> libssl is in 2 versions:
> dev-libs/openssl-0.9.8u:0.9.8
> dev-libs/openssl-1.0.0h:0
> so pick one that suits you better.

Okay; the machine that I had a binary ready for and is probably closest (I'm
not sure whether I can find an amd64 box, but Intel binary should do), has
0.9.8e, so that should work.

First attempt, build against ROOT 5.32 (latest stable version):

   http://cern.ch/wlav/pypy-reflex-support-042312.tar.bz2

with md5sum:

   c6ae683605658fa43e0089a99c82c49b  pypy-reflex-support-042312.tar.bz2

> Anyway, default binary pypy works fine.

Different machines, different people, etc. :) This will be a little trial and
error, I'm afraid, since I haven't distributed PyPy in binary before (for CERN,
I simply installed binaries on the global file system, which makes it available
to most institutes, by building it on a typical worker node).

Btw., if there'll be a number of iterations necessary, then we can take this
off the list.

> Currently i have only one typemap in my *.i file for swig, and it allows
> PyObject* pointers to go through to c++ side.

Passing PyObject* is in PyROOT, not yet in cppyy, as I wasn't sure about its
representation (I'll probably have to pass it through cpyext first to build
an actual PyObject* with the guaranteed expected layout). It's on the TODO
list, though.

> so migration should be pretty painless.

Also depends on the specific C++ features used of course. Actually, what I
hope to gain from this exercise is more that cppyy gives proper and clear
diagnostics if features are missing, rather than just crash.

Furthermore, any answers to questions you may have can be turned into docs,
so that's useful as well.

> And of course if there are bugs they would most likely arise, especially if
> they are in memory management, because the c++ code is executed millions of
> times during the program run.

What we do in the unit tests, is to call the gc explicitly and then see whether
cleanup was done as expected (by having an instance counter on the C++ side).

Best regards,
            Wim
--

-- 
WLavrijsen <at> lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
Leonard Ritter | 23 Apr 11:17 2012

Re: cppyy: C++ bindings for PyPy

(sorry, that was also supposed to go to the ML)

Awesome work, keep it up!

But: I showed the reflex library you're using to friends and they both
suggested clang to parse C++ headers instead. What do you think about
that?

Cheers,
Leonard

On Mon, Apr 23, 2012 at 7:08 AM,  <wlavrijsen <at> lbl.gov> wrote:
> Hi,
>
> as detailed in a couple of blog posts in the past (*), for some time now we
> have been working on making C++ bindings through the Reflex package
> available
> on PyPy, in the form of the "cppyy" module. Software is never done, and that
> is true also in this case, but it has reached a level of maturity at which
> it
> can be said to be usable. Initial docs are now up on:
>
>   http://doc.pypy.org/en/latest/cppyy.html
>
> with instructions on how to setup the reflex-support branch and how to test
> it out. The documentation will be updated with more advanced and detailed
> topics in the next couple of weeks or so.
>
> There's a sizable (non-PyPy) set of unit tests that still need to be worked
> through, thus development will steadily continue at its current pace. Still,
> if you find that cppyy almost works for you except for a particular feature,
> feel free to ask for it to be prioritized.
>
> The biggest obstacle for most people (that are not in the field of HEP) will
> be the current set of dependencies. Although the dependency set for cppyy is
> really only Reflex, which could be distributed separately, for the CPython
> equivalent code, the dependency set is a large portion of the ROOT class
> library. The medium-term plan then is to use Cling, which is based on CLang
> from llvm (http://root.cern.ch/drupal/content/cling) as the back-end. For
> this, there will be a PyCling on CPython, and all as stand-alone projects to
> remove the large dependency set.
>
> More interesting though, having Cling on the back of the bindings allows
> far more advanced features, such as dynamic setup of callbacks and cross
> language inheritance to put Python code into C++ frameworks. This has been
> prototyped successfully in Clings predecessor (CINT), but would be very hard
> to do in Reflex.
>
> Of course, an important reason for pushing this code out to the community
> somewhat early, is that it allows anyone so interested to start hacking on
> it and help shape it!
>
> Best regards,
>           Wim
>
> (*)
> http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.html
>
>  http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries.html
> --
> WLavrijsen <at> lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
> _______________________________________________
> pypy-dev mailing list
> pypy-dev <at> python.org
> http://mail.python.org/mailman/listinfo/pypy-dev
Leonard Ritter | 23 Apr 11:21 2012

Re: cppyy: C++ bindings for PyPy

Addendum: apparently, clang even provides python bindings.

http://eli.thegreenplace.net/2011/07/03/parsing-c-in-python-with-clang/

On Mon, Apr 23, 2012 at 11:17 AM, Leonard Ritter <paniq <at> paniq.org> wrote:
> (sorry, that was also supposed to go to the ML)
>
> Awesome work, keep it up!
>
> But: I showed the reflex library you're using to friends and they both
> suggested clang to parse C++ headers instead. What do you think about
> that?
>
> Cheers,
> Leonard
>
> On Mon, Apr 23, 2012 at 7:08 AM,  <wlavrijsen <at> lbl.gov> wrote:
>> Hi,
>>
>> as detailed in a couple of blog posts in the past (*), for some time now we
>> have been working on making C++ bindings through the Reflex package
>> available
>> on PyPy, in the form of the "cppyy" module. Software is never done, and that
>> is true also in this case, but it has reached a level of maturity at which
>> it
>> can be said to be usable. Initial docs are now up on:
>>
>>   http://doc.pypy.org/en/latest/cppyy.html
>>
>> with instructions on how to setup the reflex-support branch and how to test
>> it out. The documentation will be updated with more advanced and detailed
>> topics in the next couple of weeks or so.
>>
>> There's a sizable (non-PyPy) set of unit tests that still need to be worked
>> through, thus development will steadily continue at its current pace. Still,
>> if you find that cppyy almost works for you except for a particular feature,
>> feel free to ask for it to be prioritized.
>>
>> The biggest obstacle for most people (that are not in the field of HEP) will
>> be the current set of dependencies. Although the dependency set for cppyy is
>> really only Reflex, which could be distributed separately, for the CPython
>> equivalent code, the dependency set is a large portion of the ROOT class
>> library. The medium-term plan then is to use Cling, which is based on CLang
>> from llvm (http://root.cern.ch/drupal/content/cling) as the back-end. For
>> this, there will be a PyCling on CPython, and all as stand-alone projects to
>> remove the large dependency set.
>>
>> More interesting though, having Cling on the back of the bindings allows
>> far more advanced features, such as dynamic setup of callbacks and cross
>> language inheritance to put Python code into C++ frameworks. This has been
>> prototyped successfully in Clings predecessor (CINT), but would be very hard
>> to do in Reflex.
>>
>> Of course, an important reason for pushing this code out to the community
>> somewhat early, is that it allows anyone so interested to start hacking on
>> it and help shape it!
>>
>> Best regards,
>>           Wim
>>
>> (*)
>> http://morepypy.blogspot.com/2011/08/wrapping-c-libraries-with-reflection.html
>>
>>  http://morepypy.blogspot.com/2010/07/cern-sprint-report-wrapping-c-libraries.html
>> --
>> WLavrijsen <at> lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev <at> python.org
>> http://mail.python.org/mailman/listinfo/pypy-dev
wlavrijsen | 23 Apr 16:19 2012

Re: cppyy: C++ bindings for PyPy

Leonard,

> Addendum: apparently, clang even provides python bindings.
> http://eli.thegreenplace.net/2011/07/03/parsing-c-in-python-with-clang/

yes, but AFAIK it's C only and in any case, those are bindings to CLang,
rather than bindings to the code parsed from CLang. You'll run into the
same problems in their use from PyPy just as any other extension library,
and you'd still have to build up the bindings to user code from those
bindings to CLang (which is where the major work resides).

> But: I showed the reflex library you're using to friends and they both
> suggested clang to parse C++ headers instead. What do you think about
> that?

As said in the mail and the docs: the medium term is to use Cling, which
is based on CLang. The difference is the C++ interactivity, which is a
better match for Python to allow build-up of callbacks and cross-language
inheritance.

We started out with Reflex b/c that was a known quantity. Also, for large
C++ libraries across many different projects (as we have to deal with),
there is nothing in CLang that allows you to cleanly pre-package reflection
info like is done with shared libraries in Reflex. For Cling, a method is
being developed based on pre-compiled headers.

Best regards,
            Wim
--

-- 
WLavrijsen <at> lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

Gmane