Carl Witty | 5 Aug 06:01 2010
Picon

[Cython] working on Cython for IronPython

I'm working with Enthought on a branch of Cython to target .NET for
IronPython.  I'm maintaining a repository at
http://bitbucket.org/cwitty/cython-for-ironpython (I just started
writing code, so there's not much interesting there yet).

I'm hopeful that eventually this code will be merged back into Cython;
I don't want to fork Cython.

I'm starting by essentially duplicating all of the generate_* methods
into generate_dotnet_* methods, and modifying the duplicated methods
to generate C++/CLI code.  My hope is that once I have a working
compiler, I can refactor to merge most of the methods back together
(eliminating the duplicate code) while still not being very intrusive
to the original codebase.

Carl
Dag Sverre Seljebotn | 5 Aug 09:08 2010
Picon
Picon

Re: [Cython] working on Cython for IronPython

On 08/05/2010 06:01 AM, Carl Witty wrote:
> I'm working with Enthought on a branch of Cython to target .NET for
> IronPython.  I'm maintaining a repository at
> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
> writing code, so there's not much interesting there yet).
>    
Great, I think this is fantastic news.
> I'm hopeful that eventually this code will be merged back into Cython;
> I don't want to fork Cython.
>    
I hope this stays in Cython as well.
> I'm starting by essentially duplicating all of the generate_* methods
> into generate_dotnet_* methods, and modifying the duplicated methods
> to generate C++/CLI code.  My hope is that once I have a working
>    
Ah, this is very interesting -- when I've spoken with Enthought on the 
matter we've largely ignored C++ for this and were talking about a C# 
backend.
> compiler, I can refactor to merge most of the methods back together
> (eliminating the duplicate code) while still not being very intrusive
> to the original codebase.
>    

Have you considered writing a transform instead of adding more methods 
to the nodes? E.g. something like Cython/CodeWriter.py for the .NET 
backend? That would be even less intrusive...ideally, one would just 
configure the pipeline differently depending on what the backend is.

(I once suggested that the CPython generation was also moved into a 
seperate generator class like this as well; of course, that's a lot of 
(Continue reading)

Carl Witty | 5 Aug 18:58 2010
Picon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
<dagss@...> wrote:
> On 08/05/2010 06:01 AM, Carl Witty wrote:
>> I'm starting by essentially duplicating all of the generate_* methods
>> into generate_dotnet_* methods, and modifying the duplicated methods
>> to generate C++/CLI code.  My hope is that once I have a working
>>
> Ah, this is very interesting -- when I've spoken with Enthought on the
> matter we've largely ignored C++ for this and were talking about a C#
> backend.

I convinced the Enthought people that a C++/CLI backend was the best
first step.  For a variety of reasons (mostly the fact that Cython's
"extern from" describes the API, not the ABI, but also catching C++
exceptions and turning them into Python exceptions) a C# backend would
have to generate C or C++ wrappers for every operation on "extern
from" values/types (function call, global variable read, global
variable write, struct member read, struct member write, ...).  On the
other hand, a C++/CLI backend can handle "extern from" in exactly the
same way that the current C/C++ backend does.

Once the C++/CLI backend works, a C# backend is a possible next step.
All of the code generation for C# to interface with IronPython would
be exactly the same as C++/CLI (modulo minor syntactic issues that can
easily be localized).  Even if the ultimate goal is a C# backend, I
think that very little of the work on a C++/CLI backend would be
wasted.

>> compiler, I can refactor to merge most of the methods back together
>> (eliminating the duplicate code) while still not being very intrusive
(Continue reading)

Robert Bradshaw | 6 Aug 05:26 2010

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 9:58 AM, Carl Witty <carl.witty@...> wrote:
> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
> <dagss@...> wrote:
>> On 08/05/2010 06:01 AM, Carl Witty wrote:
>>> I'm starting by essentially duplicating all of the generate_* methods
>>> into generate_dotnet_* methods, and modifying the duplicated methods
>>> to generate C++/CLI code.  My hope is that once I have a working
>>>
>> Ah, this is very interesting -- when I've spoken with Enthought on the
>> matter we've largely ignored C++ for this and were talking about a C#
>> backend.
>
> I convinced the Enthought people that a C++/CLI backend was the best
> first step.  For a variety of reasons (mostly the fact that Cython's
> "extern from" describes the API, not the ABI, but also catching C++
> exceptions and turning them into Python exceptions) a C# backend would
> have to generate C or C++ wrappers for every operation on "extern
> from" values/types (function call, global variable read, global
> variable write, struct member read, struct member write, ...).  On the
> other hand, a C++/CLI backend can handle "extern from" in exactly the
> same way that the current C/C++ backend does.
>
> Once the C++/CLI backend works, a C# backend is a possible next step.
> All of the code generation for C# to interface with IronPython would
> be exactly the same as C++/CLI (modulo minor syntactic issues that can
> easily be localized).  Even if the ultimate goal is a C# backend, I
> think that very little of the work on a C++/CLI backend would be
> wasted.

The ultimate goal is really just Cython working (well) for IronPython,
(Continue reading)

Carl Witty | 6 Aug 06:55 2010
Picon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 8:26 PM, Robert Bradshaw
<robertwb@...> wrote:
> On Thu, Aug 5, 2010 at 9:58 AM, Carl Witty <carl.witty@...> wrote:
>> Once the C++/CLI backend works, a C# backend is a possible next step.
>> All of the code generation for C# to interface with IronPython would
>> be exactly the same as C++/CLI (modulo minor syntactic issues that can
>> easily be localized).  Even if the ultimate goal is a C# backend, I
>> think that very little of the work on a C++/CLI backend would be
>> wasted.
>
> The ultimate goal is really just Cython working (well) for IronPython,
> right? How essential is C# generation (vs. C++/CLI) to that goal?

For IronPython running under Microsoft's .NET implementation for
Windows, as far as I know there are no problems with C++/CLI.  For
IronPython running under Mono on non-Windows platforms, C++/CLI
doesn't help at all; there are no implementations of C++/CLI on
non-Windows platforms and no effort toward making an implementation.
So the purpose of a C# backend would be to support Mono.

Carl
Robert Bradshaw | 6 Aug 07:14 2010

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 9:55 PM, Carl Witty <carl.witty@...> wrote:
> On Thu, Aug 5, 2010 at 8:26 PM, Robert Bradshaw
> <robertwb@...> wrote:
>> On Thu, Aug 5, 2010 at 9:58 AM, Carl Witty <carl.witty@...> wrote:
>>> Once the C++/CLI backend works, a C# backend is a possible next step.
>>> All of the code generation for C# to interface with IronPython would
>>> be exactly the same as C++/CLI (modulo minor syntactic issues that can
>>> easily be localized).  Even if the ultimate goal is a C# backend, I
>>> think that very little of the work on a C++/CLI backend would be
>>> wasted.
>>
>> The ultimate goal is really just Cython working (well) for IronPython,
>> right? How essential is C# generation (vs. C++/CLI) to that goal?
>
> For IronPython running under Microsoft's .NET implementation for
> Windows, as far as I know there are no problems with C++/CLI.  For
> IronPython running under Mono on non-Windows platforms, C++/CLI
> doesn't help at all; there are no implementations of C++/CLI on
> non-Windows platforms and no effort toward making an implementation.
> So the purpose of a C# backend would be to support Mono.

Ah. As much as I support the Mono project, I would imagine that most
Mono users who wanted to use Python would just use CPython directly
(though I wouldn't rule out any usecase of IronPython on Mono). Of
course this is just pure speculation, I don't follow the Mono project
very closely. If it comes out naturally once the C++/CLI is going well
that would be a huge plus.

- Robert
(Continue reading)

Dag Sverre Seljebotn | 6 Aug 11:31 2010
Picon
Picon

Re: [Cython] working on Cython for IronPython

Robert Bradshaw wrote:
> On Thu, Aug 5, 2010 at 9:55 PM, Carl Witty <carl.witty@...> wrote:
>   
>> On Thu, Aug 5, 2010 at 8:26 PM, Robert Bradshaw
>> <robertwb@...> wrote:
>>     
>>> On Thu, Aug 5, 2010 at 9:58 AM, Carl Witty <carl.witty@...> wrote:
>>>       
>>>> Once the C++/CLI backend works, a C# backend is a possible next step.
>>>> All of the code generation for C# to interface with IronPython would
>>>> be exactly the same as C++/CLI (modulo minor syntactic issues that can
>>>> easily be localized).  Even if the ultimate goal is a C# backend, I
>>>> think that very little of the work on a C++/CLI backend would be
>>>> wasted.
>>>>         
>>> The ultimate goal is really just Cython working (well) for IronPython,
>>> right? How essential is C# generation (vs. C++/CLI) to that goal?
>>>       
>> For IronPython running under Microsoft's .NET implementation for
>> Windows, as far as I know there are no problems with C++/CLI.  For
>> IronPython running under Mono on non-Windows platforms, C++/CLI
>> doesn't help at all; there are no implementations of C++/CLI on
>> non-Windows platforms and no effort toward making an implementation.
>> So the purpose of a C# backend would be to support Mono.
>>     
>
> Ah. As much as I support the Mono project, I would imagine that most
> Mono users who wanted to use Python would just use CPython directly
> (though I wouldn't rule out any usecase of IronPython on Mono). Of
> course this is just pure speculation, I don't follow the Mono project
(Continue reading)

Stefan Behnel | 8 Aug 11:15 2010
Picon

Re: [Cython] working on Cython for IronPython

Dag Sverre Seljebotn, 06.08.2010 11:31:
> Mono support could be important *long-term*, because it allows all the
> non-Windows users to not break things on the .NET side when developing
> and merging patches. One can include Mono builds in build farms, whether
> it is Cython, NumPy, SciPy etc.
>
> For that purpose Wine could work as well though if .NET is well
> supported in Wine. Well, given some common scientific Python build farm
> infrastructure (where Sage and Enthought pools together?), even
> VirtualBox would be workable. But at the end of the day, Mono will
> always be more convenient to set up for Linux devs.

Actually, .NET not being portable pretty much rules out any use case I 
might have for it. Mono is the only reason for me to consider IronPython a 
viable alternative to CPython.

> Also, though I don't really know, it might be that Cython+C# would beat
> Cython+CPython performance wise for non-typed Python stuff. And no GIL!

Remember that the GIL makes your life easier. Cython is about writing 
extension modules, and very, very many of those only work *because* of the 
GIL. Here's an example:

http://code.activestate.com/recipes/577336-fast-re-entrant-optimistic-lock-implemented-in-cyt/

In the Cython wrappers I wrote so far, a *lot* of critical code truly 
relies on the GIL.

So, one part of porting Cython to .NET would be to add a global lock for 
the code (at least when threading is used, which may be tricky to 
(Continue reading)

Dag Sverre Seljebotn | 8 Aug 21:53 2010
Picon
Picon

Re: [Cython] working on Cython for IronPython

On 08/08/2010 11:15 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 06.08.2010 11:31:
>    
>> Mono support could be important *long-term*, because it allows all the
>> non-Windows users to not break things on the .NET side when developing
>> and merging patches. One can include Mono builds in build farms, whether
>> it is Cython, NumPy, SciPy etc.
>>
>> For that purpose Wine could work as well though if .NET is well
>> supported in Wine. Well, given some common scientific Python build farm
>> infrastructure (where Sage and Enthought pools together?), even
>> VirtualBox would be workable. But at the end of the day, Mono will
>> always be more convenient to set up for Linux devs.
>>      
> Actually, .NET not being portable pretty much rules out any use case I
> might have for it. Mono is the only reason for me to consider IronPython a
> viable alternative to CPython.
>    

I think the main point of the .NET Cython port is to allow libraries to 
also target .NET. I.e. I don't think many of the *current* CPython users 
will really use it themselves (Mono or not), it is all about reaching 
out to *new* users who are currently using .NET .

The usefulness of this depends, of course, very much on the library in 
question. For SciPy and NumPy there is definitely a demand (IronClad was 
developed to run NumPy IIRC).

Dag Sverre
(Continue reading)

Carl Witty | 8 Aug 22:11 2010
Picon

Re: [Cython] working on Cython for IronPython

On Sun, Aug 8, 2010 at 2:15 AM, Stefan Behnel <stefan_ml@...> wrote:
> Remember that the GIL makes your life easier. Cython is about writing
> extension modules, and very, very many of those only work *because* of the
> GIL. Here's an example:
>
> http://code.activestate.com/recipes/577336-fast-re-entrant-optimistic-lock-implemented-in-cyt/
>
> In the Cython wrappers I wrote so far, a *lot* of critical code truly
> relies on the GIL.
>
> So, one part of porting Cython to .NET would be to add a global lock for
> the code (at least when threading is used, which may be tricky to
> determine). Obviously, this could become a compiler option for the .NET
> backend, but the lock would still have to be turned off explicitly in order
> to prevent breaking Cython code by default.

OK, in GIL-compatibility mode I can add a global lock for all Cython
code (which would then be a global lock for all cdef data).  This
still won't be exactly equivalent to CPython, though; since IronPython
has no GIL, I can't block pure-Python code from running simultaneously
with Cython code.  So Cython code that assumes (for example) that it
holds a lock on all Python dictionaries (including module
dictionaries), by virtue of holding the GIL, could break.

This would be good enough for the "Fast, re-entrant, optimistic lock"
you linked to, since it's manipulating cdef data (although it wouldn't
be as fast any more, since it's adding an extra lock/unlock on every
method call).

Does your code assume that the Cython GIL locks Python data
(Continue reading)

Stefan Behnel | 8 Aug 22:31 2010
Picon

Re: [Cython] working on Cython for IronPython

Carl Witty, 08.08.2010 22:11:
> On Sun, Aug 8, 2010 at 2:15 AM, Stefan Behnel wrote:
>> Remember that the GIL makes your life easier. Cython is about writing
>> extension modules, and very, very many of those only work *because* of the
>> GIL. Here's an example:
>>
>> http://code.activestate.com/recipes/577336-fast-re-entrant-optimistic-lock-implemented-in-cyt/
>>
>> In the Cython wrappers I wrote so far, a *lot* of critical code truly
>> relies on the GIL.
>>
>> So, one part of porting Cython to .NET would be to add a global lock for
>> the code (at least when threading is used, which may be tricky to
>> determine). Obviously, this could become a compiler option for the .NET
>> backend, but the lock would still have to be turned off explicitly in order
>> to prevent breaking Cython code by default.
>
> OK, in GIL-compatibility mode I can add a global lock for all Cython
> code (which would then be a global lock for all cdef data).  This
> still won't be exactly equivalent to CPython, though; since IronPython
> has no GIL, I can't block pure-Python code from running simultaneously
> with Cython code.  So Cython code that assumes (for example) that it
> holds a lock on all Python dictionaries (including module
> dictionaries), by virtue of holding the GIL, could break.

Right. I think the requirement here is more like a lock for Cython code, 
but global to the system. While it is true that the GIL prevents the 
execution of /any/ Python code while an extension holds it, I would expect 
that the most important property is that it prevents other threads from 
executing parts of the extension itself.
(Continue reading)

Sturla Molden | 9 Aug 19:42 2010
Picon

Re: [Cython] working on Cython for IronPython

> Dag Sverre Seljebotn, 06.08.2010 11:31:

> Remember that the GIL makes your life easier. Cython is about writing
> extension modules, and very, very many of those only work *because* of the
> GIL.

One of the main virtues of CPython is that extension code (compiled C,
C++, Fortran) is atomic with respect to the interpreter, unless the GIL is
released.

Sturla

David Cournapeau | 11 Aug 06:56 2010
Picon

Re: [Cython] working on Cython for IronPython

On Fri, Aug 6, 2010 at 1:55 PM, Carl Witty <carl.witty <at> gmail.com> wrote:

> For IronPython running under Microsoft's .NET implementation for
> Windows, as far as I know there are no problems with C++/CLI.  For
> IronPython running under Mono on non-Windows platforms, C++/CLI
> doesn't help at all; there are no implementations of C++/CLI on
> non-Windows platforms and no effort toward making an implementation.
> So the purpose of a C# backend would be to support Mono.

I would think that it would also help for a jython backend, as there
is no equivalent to C++/CLI in the java land AFAIK, and changing from
C# to java back in cython should be much easier.

For people who are not so familiar with C++/Cli, could you expand a
bit more on the choice of C++/CLI vs C# as a first step ? The "extern"
part sounds logical (a VM like the CLR or the JVM has strong strong
ABI constraints), but the exception part not so much. Also, what does
C++/Cli buys you instead of pure C or C++ (through p/invoke) ? CLI is
gc aware, right ? I am also curious about the differences between
C++/CLI vs p/invoke (the latter having an equivalent in the java),

cheers,

Davi
_______________________________________________
Cython-dev mailing list
Cython-dev <at> codespeak.net
http://codespeak.net/mailman/listinfo/cython-dev
Robert Bradshaw | 11 Aug 07:58 2010

Re: [Cython] working on Cython for IronPython

On Tue, Aug 10, 2010 at 9:56 PM, David Cournapeau <cournape@...> wrote:
> On Fri, Aug 6, 2010 at 1:55 PM, Carl Witty <carl.witty@...> wrote:
>
>> For IronPython running under Microsoft's .NET implementation for
>> Windows, as far as I know there are no problems with C++/CLI.  For
>> IronPython running under Mono on non-Windows platforms, C++/CLI
>> doesn't help at all; there are no implementations of C++/CLI on
>> non-Windows platforms and no effort toward making an implementation.
>> So the purpose of a C# backend would be to support Mono.
>
> I would think that it would also help for a jython backend, as there
> is no equivalent to C++/CLI in the java land AFAIK, and changing from
> C# to java back in cython should be much easier.
>
> For people who are not so familiar with C++/Cli, could you expand a
> bit more on the choice of C++/CLI vs C# as a first step ? The "extern"
> part sounds logical (a VM like the CLR or the JVM has strong strong
> ABI constraints), but the exception part not so much. Also, what does
> C++/Cli buys you instead of pure C or C++ (through p/invoke) ? CLI is
> gc aware, right ? I am also curious about the differences between
> C++/CLI vs p/invoke (the latter having an equivalent in the java),

I'd imagine that the primary goal of using CLI is that IronPython
objects live in the CLI environment, and the goal is to play nice with
that. I'm not very familiar with all of this though, so would be
curious to hear more details myself.

- Robert
Carl Witty | 11 Aug 08:42 2010
Picon

Re: [Cython] working on Cython for IronPython

On Tue, Aug 10, 2010 at 9:56 PM, David Cournapeau <cournape@...> wrote:
> On Fri, Aug 6, 2010 at 1:55 PM, Carl Witty <carl.witty@...> wrote:
>
>> For IronPython running under Microsoft's .NET implementation for
>> Windows, as far as I know there are no problems with C++/CLI.  For
>> IronPython running under Mono on non-Windows platforms, C++/CLI
>> doesn't help at all; there are no implementations of C++/CLI on
>> non-Windows platforms and no effort toward making an implementation.
>> So the purpose of a C# backend would be to support Mono.
>
> I would think that it would also help for a jython backend, as there
> is no equivalent to C++/CLI in the java land AFAIK, and changing from
> C# to java back in cython should be much easier.

It certainly seems likely that a C# backend would be a much better
starting point for a Jython port than a C++/CLI backend.  (I'm not
familiar enough with Jython internals or with the JVM->C interface to
be more precise, though.)

> For people who are not so familiar with C++/Cli, could you expand a
> bit more on the choice of C++/CLI vs C# as a first step ? The "extern"
> part sounds logical (a VM like the CLR or the JVM has strong strong
> ABI constraints), but the exception part not so much.

Sure.  The issue with exceptions is almost trivial, but still important.

According to the Mono P/Invoke documentation, P/Invoke is for talking
to C functions.  If a function throws a C++ exception, it wasn't a C
function.  And in particular, under Mono, if you P/Invoke a function
that throws a C++ exception, very bad things happen; the exception
(Continue reading)

David Cournapeau | 12 Aug 05:08 2010
Picon

Re: [Cython] working on Cython for IronPython

On Wed, Aug 11, 2010 at 3:42 PM, Carl Witty <carl.witty <at> gmail.com> wrote:

>
> Sure.  The issue with exceptions is almost trivial, but still important.
>
> According to the Mono P/Invoke documentation, P/Invoke is for talking
> to C functions.  If a function throws a C++ exception, it wasn't a C
> function.  And in particular, under Mono, if you P/Invoke a function
> that throws a C++ exception, very bad things happen; the exception
> can't be caught by managed code, and probably your application
> crashes.

Ok. This issue is likely the same in the java world - running a C++
function without C++ runtime support cannot work, hence the need for a
wrapper, like it is usually done when you want to dlopen a C++
library.

>
> In my opinion, the major advantage of C++/CLI over C# with P/Invoke is
> that C++/CLI has a working C++ header file parser.  This means that we
> can keep the current Cython semantics, where users describe to Cython
> the API (rather than the ABI) of the library they're trying to call;
> then Cython generates C, C++, or C++/CLI code that conforms to this
> API, and the compiler handles the rest.

So just to be sure I understand, using C++/CLI gives you a way to
generate the C++ wrapper needed to expose a C-like ABI to cython (and
python runtime) ? A bit like python boost does ?

It seems to me that having a "toolchain" to do this would also be
(Continue reading)

Carl Witty | 12 Aug 07:11 2010
Picon

Re: [Cython] working on Cython for IronPython

On Wed, Aug 11, 2010 at 8:08 PM, David Cournapeau <cournape@...> wrote:
> On Wed, Aug 11, 2010 at 3:42 PM, Carl Witty <carl.witty@...> wrote:
>> In my opinion, the major advantage of C++/CLI over C# with P/Invoke is
>> that C++/CLI has a working C++ header file parser.  This means that we
>> can keep the current Cython semantics, where users describe to Cython
>> the API (rather than the ABI) of the library they're trying to call;
>> then Cython generates C, C++, or C++/CLI code that conforms to this
>> API, and the compiler handles the rest.
>
> So just to be sure I understand, using C++/CLI gives you a way to
> generate the C++ wrapper needed to expose a C-like ABI to cython (and
> python runtime) ? A bit like python boost does ?

Well, not exactly.  Cython can already deal with a C++ API (better in
the not-yet-released 0.13 than in previous versions); I would say that
Cython itself is the tool that takes a C++ API and exposes a C ABI to
the Python runtime.  The advantage of C++/CLI is that it can directly
consume C and C++ APIs (as expected, since it's basically a superset
of C++).  With C++/CLI there isn't really a wrapper.  I identified two
reasons for a wrapper; one is to translate C or C++ APIs into known
ABIs, and the other is to catch and deal with C++ exceptions.  With
C++/CLI, neither is needed (just like we don't need a wrapper when
generating C++ to target CPython).

> It seems to me that having a "toolchain" to do this would also be
> useful for auto-generating .pxd from header files directly. May be
> enough to motivate me to continue the work I started there using clang
>
> It seems that fixing this would solve the issue for both java and C#
> output at the same time, which is a good point.
(Continue reading)

Dan Stromberg | 12 Aug 07:57 2010
Picon

Re: [Cython] working on Cython for IronPython


On Wed, Aug 11, 2010 at 10:11 PM, Carl Witty <carl.witty <at> gmail.com> wrote:
On Wed, Aug 11, 2010 at 8:08 PM, David Cournapeau <cournape-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Wed, Aug 11, 2010 at 3:42 PM, Carl Witty <carl.witty-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> In my opinion, the major advantage of C++/CLI over C# with P/Invoke is
>> that C++/CLI has a working C++ header file parser.  This means that we
>> can keep the current Cython semantics, where users describe to Cython
>> the API (rather than the ABI) of the library they're trying to call;
>> then Cython generates C, C++, or C++/CLI code that conforms to this
>> API, and the compiler handles the rest.
>
> So just to be sure I understand, using C++/CLI gives you a way to
> generate the C++ wrapper needed to expose a C-like ABI to cython (and
> python runtime) ? A bit like python boost does ?

Well, not exactly.  Cython can already deal with a C++ API (better in
the not-yet-released 0.13 than in previous versions); I would say that
Cython itself is the tool that takes a C++ API and exposes a C ABI to
the Python runtime.  The advantage of C++/CLI is that it can directly
consume C and C++ APIs (as expected, since it's basically a superset
of C++).  With C++/CLI there isn't really a wrapper.  I identified two
reasons for a wrapper; one is to translate C or C++ APIs into known
ABIs, and the other is to catch and deal with C++ exceptions.  With
C++/CLI, neither is needed (just like we don't need a wrapper when
generating C++ to target CPython).

> It seems to me that having a "toolchain" to do this would also be
> useful for auto-generating .pxd from header files directly. May be
> enough to motivate me to continue the work I started there using clang
>
> It seems that fixing this would solve the issue for both java and C#
> output at the same time, which is a good point.

You're talking about a toolchain to generate C-ABI wrappers for
arbitrary C++ code?  That would indeed be a good place to start for
JVM or C# Cython backends.  (Although there would still be a lot of
work involved; like I said, Cython currently targets C++ APIs
directly, so there would be a lot of changes to make it target your
wrappers instead.)

Carl
_______________________________________________
Cython-dev mailing list
Cython-dev-F/1GfIIGwJtbRRN4PJnoQQ@public.gmane.org
http://codespeak.net/mailman/listinfo/cython-dev

I don't know much about Cython internals, but if something could be done that would facilitate Cython interop with Jython, IronPython, Pypy, clpython, hotpy and maybe even pyjamas (perhaps not pyjamas because pyjamas is going to mostly be used for running things in a browser without a plugin), that would probably be a very good thing for Python in general.  Pypy is probably the most noteworthy Python implementation that seems to be getting left out of the discussion so far (but I admit, I've not read the whole thread).

One quick note about C++/CLI: IIRC, it's a bit like Cython itself.  ISTR that C++/CLI can call C/C++ ABI and .Net ABI callables, much as Cython can call CPython, C and C++ callables.

--
Dan Stromberg
<div>
<br><div class="gmail_quote">On Wed, Aug 11, 2010 at 10:11 PM, Carl Witty <span dir="ltr">&lt;<a href="mailto:carl.witty@...">carl.witty <at> gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote">
<div class="im">On Wed, Aug 11, 2010 at 8:08 PM, David Cournapeau &lt;<a href="mailto:cournape@...">cournape@...</a>&gt; wrote:<br>
&gt; On Wed, Aug 11, 2010 at 3:42 PM, Carl Witty &lt;<a href="mailto:carl.witty@...">carl.witty@...</a>&gt; wrote:<br>
</div>
<div class="im">&gt;&gt; In my opinion, the major advantage of C++/CLI over C# with P/Invoke is<br>
&gt;&gt; that C++/CLI has a working C++ header file parser. &nbsp;This means that we<br>
&gt;&gt; can keep the current Cython semantics, where users describe to Cython<br>
&gt;&gt; the API (rather than the ABI) of the library they're trying to call;<br>
&gt;&gt; then Cython generates C, C++, or C++/CLI code that conforms to this<br>
&gt;&gt; API, and the compiler handles the rest.<br>
&gt;<br>
&gt; So just to be sure I understand, using C++/CLI gives you a way to<br>
&gt; generate the C++ wrapper needed to expose a C-like ABI to cython (and<br>
&gt; python runtime) ? A bit like python boost does ?<br><br>
</div>Well, not exactly. &nbsp;Cython can already deal with a C++ API (better in<br>
the not-yet-released 0.13 than in previous versions); I would say that<br>
Cython itself is the tool that takes a C++ API and exposes a C ABI to<br>
the Python runtime. &nbsp;The advantage of C++/CLI is that it can directly<br>
consume C and C++ APIs (as expected, since it's basically a superset<br>
of C++). &nbsp;With C++/CLI there isn't really a wrapper. &nbsp;I identified two<br>
reasons for a wrapper; one is to translate C or C++ APIs into known<br>
ABIs, and the other is to catch and deal with C++ exceptions. &nbsp;With<br>
C++/CLI, neither is needed (just like we don't need a wrapper when<br>
generating C++ to target CPython).<br><div class="im">
<br>
&gt; It seems to me that having a "toolchain" to do this would also be<br>
&gt; useful for auto-generating .pxd from header files directly. May be<br>
&gt; enough to motivate me to continue the work I started there using clang<br>
&gt;<br>
&gt; It seems that fixing this would solve the issue for both java and C#<br>
&gt; output at the same time, which is a good point.<br><br>
</div>You're talking about a toolchain to generate C-ABI wrappers for<br>
arbitrary C++ code? &nbsp;That would indeed be a good place to start for<br>
JVM or C# Cython backends. &nbsp;(Although there would still be a lot of<br>
work involved; like I said, Cython currently targets C++ APIs<br>
directly, so there would be a lot of changes to make it target your<br>
wrappers instead.)<br><br>
Carl<br><div>
<div></div>
<div class="h5">_______________________________________________<br>
Cython-dev mailing list<br><a href="mailto:Cython-dev@...">Cython-dev@...</a><br><a href="http://codespeak.net/mailman/listinfo/cython-dev" target="_blank">http://codespeak.net/mailman/listinfo/cython-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>I don't know much about Cython internals, but if something could be done that would facilitate Cython interop with Jython, IronPython, Pypy, clpython, hotpy and maybe even pyjamas (perhaps not pyjamas because pyjamas is going to mostly be used for running things in a browser without a plugin), that would probably be a very good thing for Python in general. &nbsp;Pypy is probably the most noteworthy Python implementation that seems to be getting left out of the discussion so far (but I admit, I've not read the whole thread).<br clear="all"><br><div>One quick note about C++/CLI: IIRC, it's a bit like Cython itself. &nbsp;ISTR that C++/CLI can call C/C++ ABI and .Net ABI callables, much as Cython can call CPython, C and C++ callables.</div>
<div>
<br>-- <br>Dan Stromberg<br>
</div>
</div>
Stefan Behnel | 12 Aug 08:16 2010
Picon

Re: [Cython] working on Cython for IronPython

Dan Stromberg, 12.08.2010 07:57:
> I don't know much about Cython internals, but if something could be done
> that would facilitate Cython interop with Jython, IronPython, Pypy,
> clpython, hotpy and maybe even pyjamas (perhaps not pyjamas because pyjamas
> is going to mostly be used for running things in a browser without a
> plugin), that would probably be a very good thing for Python in general.

Yes, I think Cython would be the right tool to enable this. At least the 
platforms that have some kind of CPython C-API compatible layer should be 
relatively easy to target in a first run, and Cython could apply some 
adaptations in the C code to make them run better (e.g. avoid some special 
casing of builtins that proves to be hard to port or inefficient to emulate).

>   Pypy is probably the most noteworthy Python implementation that seems to be
> getting left out of the discussion so far

True. Although they do have a compatibility layer by now, I bet there is 
lots of stuff that could be done in the code that Cython generates to make 
extensions run faster/better/safer with PyPy.

I'll take the heresy to ask on their mailing list if they think there's 
anything Cython can do.

Stefan
Dag Sverre Seljebotn | 6 Aug 11:12 2010
Picon
Picon

Re: [Cython] working on Cython for IronPython

Carl Witty wrote:
> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
> <dagss@...> wrote:
>   
>> On 08/05/2010 06:01 AM, Carl Witty wrote:
>>     
>>> I'm starting by essentially duplicating all of the generate_* methods
>>> into generate_dotnet_* methods, and modifying the duplicated methods
>>> to generate C++/CLI code.  My hope is that once I have a working
>>>
>>>       
>> Ah, this is very interesting -- when I've spoken with Enthought on the
>> matter we've largely ignored C++ for this and were talking about a C#
>> backend.
>>     
>
> I convinced the Enthought people that a C++/CLI backend was the best
> first step.  For a variety of reasons (mostly the fact that Cython's
> "extern from" describes the API, not the ABI, but also catching C++
> exceptions and turning them into Python exceptions) a C# backend would
> have to generate C or C++ wrappers for every operation on "extern
> from" values/types (function call, global variable read, global
> variable write, struct member read, struct member write, ...).  On the
> other hand, a C++/CLI backend can handle "extern from" in exactly the
> same way that the current C/C++ backend does.
>   
Ahh. Yes, that makes everything much nicer indeed. (I know very little 
about .NET, but I did manage to identify the "extern from" as the 
critical non-obvious piece to get Cython running...)

My reason for asking is that I looked forward to all the goodies of C#: 
The "dynamic" type with call-site caching when calling methods on pure 
Python objects, and (down the line) builtin support for closures. (I 
don't know how much of that is available in C++/CLI?)

Speaking, hypothetically, about a C# backend, my two preferred options 
for _C_ code are either a) require that an ABI is specified in the 
Cython code (which is backwards-compatible with specifying API), or b) a 
two-stage build to generate C source that generates C# source (using 
offsetof and sizeof etc. to get hold of the ABI).

Interfacing with C++ would need run-time wrappers, anyway, though...

Anyway, I'm really glad that you have such a good handle on these 
things; I'm really happy Enthought asked you about this, they couldn't 
have got someone better for the task.

Dag Sverre
Carl Witty | 6 Aug 18:52 2010
Picon

Re: [Cython] working on Cython for IronPython

On Fri, Aug 6, 2010 at 2:12 AM, Dag Sverre Seljebotn
<dagss@...> wrote:
> My reason for asking is that I looked forward to all the goodies of C#:
> The "dynamic" type with call-site caching when calling methods on pure
> Python objects, and (down the line) builtin support for closures. (I
> don't know how much of that is available in C++/CLI?)

Unfortunately, as far as I can tell "dynamic" is useless for a Cython
backend, because it will use .NET semantics rather than Python
semantics for .NET types.

For example, IronPython represents a Python string as a .NET string --
not a wrapper or a subclass, just a plain .NET string.  Then there's a
bit of magic in the IronPython method lookup code, so that it will
notice if you're looking up a Python method on a .NET string and do
the right thing.  But the magic won't be there in the code generated
for C# "dynamic", so this would fail:

  dynamic s = "a random string";
  dynamic h = s.__hash__();

Cython-generated code will have the same call-site caching as
"dynamic"; we'll just have to generate the code ourselves.  In fact,
there is some of this already in the code I posted; for an untyped
addition, it will generate the following:

A declaration in a "module global variables" class:

  CallSite< Func< CallSite^, Object^, Object^, Object^ >^ >^
__site_op_add_16_14;

This executes once at module load time:

    __site_op_add_16_14 = CallSite< Func< CallSite^, Object^, Object^,
Object^ >^ >::Create(context->CreateBinaryOperationBinder(ExpressionType::Add));

And then to actually do the addition:

  __pyx_t_1 = __pyx_globals->__site_op_add_16_14->Target(__pyx_globals->__site_op_add_16_14,
__pyx_v_v1, __pyx_v_v2);

(The 16 and 14 are the line and column number of the addition
operation; I thought this would be nicer than some sort of counter.  I
encoded the operation type in the call site name so you can see what
the call is doing.)

I think it's true that C# supports closures and C++/CLI does not.  I
haven't thought much about the issues here, but I was assuming that
the IronPython backend would implement closures in the same way as the
CPython backend, rather than relying on closure support in the target
language.

> Speaking, hypothetically, about a C# backend, my two preferred options
> for _C_ code are either a) require that an ABI is specified in the
> Cython code (which is backwards-compatible with specifying API), or b) a
> two-stage build to generate C source that generates C# source (using
> offsetof and sizeof etc. to get hold of the ABI).

Option (a) is really annoying for libraries like libc, with a portable
API but wildly varying ABIs.  For example, to wrap the stat() system
call, you would need to deal with the fact that "struct stat" has
different members with different names on different OSes (and OS
versions), and the fact that errno might or might not be provided by

  #define errno (*_errno())

But maybe there are no other libraries like libc; I can't think of
other examples off the top of my head.  I guess that usually if you
want to portably wrap a library, it's because the library
implementation is portable and will have a similar ABI on different
platforms.  And wrapping libc is not so useful, since so much of it is
already provided by Python.

Option (b) is intriguing, and not something I'd considered.  It
doesn't help for "#define errno (*_errno())", but should make it
possible to not write wrappers for struct member access, which would
be a big efficiency advantage.  I'll definitely consider this if I do
end up working on a C# backend.

> Interfacing with C++ would need run-time wrappers, anyway, though...
>
> Anyway, I'm really glad that you have such a good handle on these
> things; I'm really happy Enthought asked you about this, they couldn't
> have got someone better for the task.

Thanks!

Carl
Dag Sverre Seljebotn | 6 Aug 19:05 2010
Picon
Picon

Re: [Cython] working on Cython for IronPython

Carl Witty wrote:
> A declaration in a "module global variables" class:
>
>   CallSite< Func< CallSite^, Object^, Object^, Object^ >^ >^
> __site_op_add_16_14;
>
> This executes once at module load time:
>
>     __site_op_add_16_14 = CallSite< Func< CallSite^, Object^, Object^,
> Object^ >^ >::Create(context->CreateBinaryOperationBinder(ExpressionType::Add));
>
> And then to actually do the addition:
>
>   __pyx_t_1 = __pyx_globals->__site_op_add_16_14->Target(__pyx_globals->__site_op_add_16_14,
> __pyx_v_v1, __pyx_v_v2);
>
> (The 16 and 14 are the line and column number of the addition
> operation; I thought this would be nicer than some sort of counter.  I
> encoded the operation type in the call site name so you can see what
> the call is doing.)
>   
Great, that looks really cool!

Dag Sverre
Robert Bradshaw | 5 Aug 10:27 2010

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
<dagss@...> wrote:
> On 08/05/2010 06:01 AM, Carl Witty wrote:
>> I'm working with Enthought on a branch of Cython to target .NET for
>> IronPython.  I'm maintaining a repository at
>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
>> writing code, so there's not much interesting there yet).
>>
> Great, I think this is fantastic news.

I'm excited about this too.

>> I'm hopeful that eventually this code will be merged back into Cython;
>> I don't want to fork Cython.
>>
> I hope this stays in Cython as well.
>> I'm starting by essentially duplicating all of the generate_* methods
>> into generate_dotnet_* methods, and modifying the duplicated methods
>> to generate C++/CLI code.  My hope is that once I have a working
>>
> Ah, this is very interesting -- when I've spoken with Enthought on the
> matter we've largely ignored C++ for this and were talking about a C#
> backend.
>> compiler, I can refactor to merge most of the methods back together
>> (eliminating the duplicate code) while still not being very intrusive
>> to the original codebase.
>>
>
> Have you considered writing a transform instead of adding more methods
> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
> backend? That would be even less intrusive...ideally, one would just
> configure the pipeline differently depending on what the backend is.

+1 The transform could go through and replace any nodes that need
special casing with special dot_net versions (that only need to know
how to generate their code). This would keep things nice and modular.
(The real question is how much code could be shared between the
CPython bindings and the IronPython ones--the raw C code generation
would probably be virtually identical.)

> (I once suggested that the CPython generation was also moved into a
> seperate generator class like this as well; of course, that's a lot of
> unnecessary work)
>
> I'm just not too sure about having "if dotnet:" all over the place
> (eventually). Of course, you should just do whatever gives results
> quickest at this point (and I very much trust your judgement), I'm just
> making sure the option is mentioned.

On this note, it would be cool if, eventually, Jython could be
supported through similar mechanisms.

- Robert
Carl Witty | 5 Aug 19:13 2010
Picon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 1:27 AM, Robert Bradshaw
<robertwb@...> wrote:
> +1 The transform could go through and replace any nodes that need
> special casing with special dot_net versions (that only need to know
> how to generate their code). This would keep things nice and modular.
> (The real question is how much code could be shared between the
> CPython bindings and the IronPython ones--the raw C code generation
> would probably be virtually identical.)

I do like this idea.

My tentative plan for merging the existing backend and the C++/CLI
backend back together is to move most of the C code fragments from
Nodes.py/ExprNodes.py into more methods on Code.py's CCodeWriter.
Then I would split the generic parts of CCodeWriter into a new base
class BaseCodeWriter, and add a new subclass CPPCLICodeWriter.  Then
(hopefully) a single conditional in ModuleNode.generate_c_code, where
it creates the rootwriter object, could handle most of the differences
between the backends.  For nodes where the code generation difference
is too large to be reasonably hidden in Code.py, your transform sounds
like a reasonable way to avoid "if dotnet:".

> On this note, it would be cool if, eventually, Jython could be
> supported through similar mechanisms.

That would be very cool.  (I don't know anything about Jython
internals or the JVM foreign function interface, so I don't know how
hard it would be.)

> - Robert

Carl
Robert Bradshaw | 6 Aug 05:30 2010

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 10:13 AM, Carl Witty <carl.witty@...> wrote:
> On Thu, Aug 5, 2010 at 1:27 AM, Robert Bradshaw
> <robertwb@...> wrote:
>> +1 The transform could go through and replace any nodes that need
>> special casing with special dot_net versions (that only need to know
>> how to generate their code). This would keep things nice and modular.
>> (The real question is how much code could be shared between the
>> CPython bindings and the IronPython ones--the raw C code generation
>> would probably be virtually identical.)
>
> I do like this idea.
>
> My tentative plan for merging the existing backend and the C++/CLI
> backend back together is to move most of the C code fragments from
> Nodes.py/ExprNodes.py into more methods on Code.py's CCodeWriter.
> Then I would split the generic parts of CCodeWriter into a new base
> class BaseCodeWriter, and add a new subclass CPPCLICodeWriter.  Then
> (hopefully) a single conditional in ModuleNode.generate_c_code, where
> it creates the rootwriter object, could handle most of the differences
> between the backends.

That sounds like a very good move in the long run.

> For nodes where the code generation difference
> is too large to be reasonably hidden in Code.py, your transform sounds
> like a reasonable way to avoid "if dotnet:".
>
>> On this note, it would be cool if, eventually, Jython could be
>> supported through similar mechanisms.
>
> That would be very cool.  (I don't know anything about Jython
> internals or the JVM foreign function interface, so I don't know how
> hard it would be.)

I don't know either, but the very act of supporting any other backend
should shed light on this, even if the technical details are
different.

On Thu, Aug 5, 2010 at 10:32 AM, Carl Witty <carl.witty@...> wrote:
> On Thu, Aug 5, 2010 at 3:56 AM, Robert Bradshaw
> <robertwb@...> wrote:
>> My one concerns is if (taken to the extreem) having a visitor that
>> knows how to generate code for every type (and the requisite
>> internals) is preferable to encapsulating this knowledge on a
>> node-by-node basis. Perhaps.
>
> Ah yes... the perennial style debate: if you have a bunch of data
> types, and a bunch of operations on those types, is it better to group
> the code by type, or by operation?  (And of course there's no "right"
> answer to such a generic question.)
>
> AFAIK, other compilers (like gcc and LLVM) group code by operation --
> I think they would typically have a source file for each compiler
> pass, for example.  On the other hand, Cython probably has more node
> types (and semantically richer node types), and fewer operations, than
> those compilers, so the right answer for them isn't necessarily right
> for Cython...

Yes, it's far from clear. If we're moving this direction long-term, it
may be possible to use transforms to reduce things to a smaller subset
of simpler node types earlier on.

- Robert
Dag Sverre Seljebotn | 5 Aug 11:39 2010
Picon
Picon

Re: [Cython] working on Cython for IronPython

Robert Bradshaw wrote:
> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
> <dagss@...> wrote:
>   
>> On 08/05/2010 06:01 AM, Carl Witty wrote:
>>     
>>> I'm working with Enthought on a branch of Cython to target .NET for
>>> IronPython.  I'm maintaining a repository at
>>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
>>> writing code, so there's not much interesting there yet).
>>>
>>>       
>> Great, I think this is fantastic news.
>>     
>
> I'm excited about this too.
>
>   
>>> I'm hopeful that eventually this code will be merged back into Cython;
>>> I don't want to fork Cython.
>>>
>>>       
>> I hope this stays in Cython as well.
>>     
>>> I'm starting by essentially duplicating all of the generate_* methods
>>> into generate_dotnet_* methods, and modifying the duplicated methods
>>> to generate C++/CLI code.  My hope is that once I have a working
>>>
>>>       
>> Ah, this is very interesting -- when I've spoken with Enthought on the
>> matter we've largely ignored C++ for this and were talking about a C#
>> backend.
>>     
>>> compiler, I can refactor to merge most of the methods back together
>>> (eliminating the duplicate code) while still not being very intrusive
>>> to the original codebase.
>>>
>>>       
>> Have you considered writing a transform instead of adding more methods
>> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
>> backend? That would be even less intrusive...ideally, one would just
>> configure the pipeline differently depending on what the backend is.
>>     
>
> +1 The transform could go through and replace any nodes that need
> special casing with special dot_net versions (that only need to know
> how to generate their code). This would keep things nice and modular.
> (The real question is how much code could be shared between the
> CPython bindings and the IronPython ones--the raw C code generation
> would probably be virtually identical.)
>   
I was actually thinking that one had the "transform" (or rather "sink") 
just emit/serialize code directly when visiting a node. The "return 
value" could either be string fragments, or if that is too inefficient, 
nothing at all.

def visit_BinopNode(self, node):
    # Very naive, I know things are more complicated...
    return '%s %s %s' % (self.visit(node.operand1), node.operator, 
self.visit(node.operand2))

def visit_FuncNode(self, node):
    # Here we would return a StringIOTree instead (see 
Cython/StringIOTree.py),
    # for efficiency.

or, perhaps just:

def visit_BinopNode(self, node):
    self.visit(node.operand1)
    self.put(node.operator)
    self.visit(node.operand2)

So generate_... would never be called, on *any* node, when doing .NET 
output.

One could deal with shared CPython/.NET code by creating a superclass:

class CLikeCodeGenerator:
    # pull out stuff that is common between CPython and .NET here

class DotNetCodeGenerator(CLikeCodeGenerator):
    # .NET-specific nodes handled here

class CPythonCodeGenerator(CLikeCodeGenerator):
    def visit_Node(self, node):
        # for now, just call generate_... if the node wasn't handled
        # by CLikeCodeGenerator

This makes it easy to plug in Java as well without making Nodes.py and 
ExprNodes.py even larger and more tangled and more full of conditionals...

Dag Sverre
Dag Sverre Seljebotn | 5 Aug 11:43 2010
Picon
Picon

Re: [Cython] working on Cython for IronPython

Dag Sverre Seljebotn wrote:
> Robert Bradshaw wrote:
>   
>> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
>> <dagss@...> wrote:
>>   
>>     
>>> On 08/05/2010 06:01 AM, Carl Witty wrote:
>>>     
>>>       
>>>> I'm working with Enthought on a branch of Cython to target .NET for
>>>> IronPython.  I'm maintaining a repository at
>>>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
>>>> writing code, so there's not much interesting there yet).
>>>>
>>>>       
>>>>         
>>> Great, I think this is fantastic news.
>>>     
>>>       
>> I'm excited about this too.
>>
>>   
>>     
>>>> I'm hopeful that eventually this code will be merged back into Cython;
>>>> I don't want to fork Cython.
>>>>
>>>>       
>>>>         
>>> I hope this stays in Cython as well.
>>>     
>>>       
>>>> I'm starting by essentially duplicating all of the generate_* methods
>>>> into generate_dotnet_* methods, and modifying the duplicated methods
>>>> to generate C++/CLI code.  My hope is that once I have a working
>>>>
>>>>       
>>>>         
>>> Ah, this is very interesting -- when I've spoken with Enthought on the
>>> matter we've largely ignored C++ for this and were talking about a C#
>>> backend.
>>>     
>>>       
>>>> compiler, I can refactor to merge most of the methods back together
>>>> (eliminating the duplicate code) while still not being very intrusive
>>>> to the original codebase.
>>>>
>>>>       
>>>>         
>>> Have you considered writing a transform instead of adding more methods
>>> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
>>> backend? That would be even less intrusive...ideally, one would just
>>> configure the pipeline differently depending on what the backend is.
>>>     
>>>       
>> +1 The transform could go through and replace any nodes that need
>> special casing with special dot_net versions (that only need to know
>> how to generate their code). This would keep things nice and modular.
>> (The real question is how much code could be shared between the
>> CPython bindings and the IronPython ones--the raw C code generation
>> would probably be virtually identical.)
>>   
>>     
> I was actually thinking that one had the "transform" (or rather "sink") 
> just emit/serialize code directly when visiting a node. The "return 
> value" could either be string fragments, or if that is too inefficient, 
> nothing at all.
>
> def visit_BinopNode(self, node):
>     # Very naive, I know things are more complicated...
>     return '%s %s %s' % (self.visit(node.operand1), node.operator, 
> self.visit(node.operand2))
>
> def visit_FuncNode(self, node):
>     # Here we would return a StringIOTree instead (see 
> Cython/StringIOTree.py),
>     # for efficiency.
>
> or, perhaps just:
>
> def visit_BinopNode(self, node):
>     self.visit(node.operand1)
>     self.put(node.operator)
>     self.visit(node.operand2)
>
> So generate_... would never be called, on *any* node, when doing .NET 
> output.
>
> One could deal with shared CPython/.NET code by creating a superclass:
>   
Actually, in the first pass, one could implement DotNetCodeGenerator 
like this:

class DotNetCodeGenerator(CLikeCodeGenerator):
    # Default fallback to CPython generation
    def visit_Node(self, node):
        node.generate_...(self.code)

    def visit_ExprNode(self, node):
        # Slightly different logic I believe...
        node.generate_...(self.code)

    # Any .NET specific stuff
    def visit_FuncNode(self, node): ...

Dag Sverre

> class CLikeCodeGenerator:
>     # pull out stuff that is common between CPython and .NET here
>
> class DotNetCodeGenerator(CLikeCodeGenerator):
>     # .NET-specific nodes handled here
>
> class CPythonCodeGenerator(CLikeCodeGenerator):
>     def visit_Node(self, node):
>         # for now, just call generate_... if the node wasn't handled
>         # by CLikeCodeGenerator
>
>
> This makes it easy to plug in Java as well without making Nodes.py and 
> ExprNodes.py even larger and more tangled and more full of conditionals...
>
> Dag Sverre
> _______________________________________________
> Cython-dev mailing list
> Cython-dev@...
> http://codespeak.net/mailman/listinfo/cython-dev
>   

Carl Witty | 5 Aug 19:24 2010
Picon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 2:43 AM, Dag Sverre Seljebotn
<dagss@...> wrote:
> Actually, in the first pass, one could implement DotNetCodeGenerator
> like this:
>
> class DotNetCodeGenerator(CLikeCodeGenerator):
>    # Default fallback to CPython generation
>    def visit_Node(self, node):
>        node.generate_...(self.code)
>
>    def visit_ExprNode(self, node):
>        # Slightly different logic I believe...
>        node.generate_...(self.code)
>
>    # Any .NET specific stuff
>    def visit_FuncNode(self, node): ...

It looks like this would work if you're calling generate_result_code,
say.  Other generate_ methods, like generate_function_definitions, are
recursive, so if you ever called the CPython version that would be
used for all of the children of the node as well as the node itself --
there's no mechanism for the transform to get control back.

> Dag Sverre

Carl
Robert Bradshaw | 5 Aug 12:56 2010

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 2:39 AM, Dag Sverre Seljebotn
<dagss@...> wrote:
> Robert Bradshaw wrote:
>> On Thu, Aug 5, 2010 at 12:08 AM, Dag Sverre Seljebotn
>> <dagss@...> wrote:
>>
>>> On 08/05/2010 06:01 AM, Carl Witty wrote:
>>>
>>>> I'm working with Enthought on a branch of Cython to target .NET for
>>>> IronPython.  I'm maintaining a repository at
>>>> http://bitbucket.org/cwitty/cython-for-ironpython (I just started
>>>> writing code, so there's not much interesting there yet).
>>>>
>>>>
>>> Great, I think this is fantastic news.
>>>
>>
>> I'm excited about this too.
>>
>>
>>>> I'm hopeful that eventually this code will be merged back into Cython;
>>>> I don't want to fork Cython.
>>>>
>>>>
>>> I hope this stays in Cython as well.
>>>
>>>> I'm starting by essentially duplicating all of the generate_* methods
>>>> into generate_dotnet_* methods, and modifying the duplicated methods
>>>> to generate C++/CLI code.  My hope is that once I have a working
>>>>
>>>>
>>> Ah, this is very interesting -- when I've spoken with Enthought on the
>>> matter we've largely ignored C++ for this and were talking about a C#
>>> backend.
>>>
>>>> compiler, I can refactor to merge most of the methods back together
>>>> (eliminating the duplicate code) while still not being very intrusive
>>>> to the original codebase.
>>>>
>>>>
>>> Have you considered writing a transform instead of adding more methods
>>> to the nodes? E.g. something like Cython/CodeWriter.py for the .NET
>>> backend? That would be even less intrusive...ideally, one would just
>>> configure the pipeline differently depending on what the backend is.
>>>
>>
>> +1 The transform could go through and replace any nodes that need
>> special casing with special dot_net versions (that only need to know
>> how to generate their code). This would keep things nice and modular.
>> (The real question is how much code could be shared between the
>> CPython bindings and the IronPython ones--the raw C code generation
>> would probably be virtually identical.)
>>
> I was actually thinking that one had the "transform" (or rather "sink")
> just emit/serialize code directly when visiting a node. The "return
> value" could either be string fragments, or if that is too inefficient,
> nothing at all.
>
> def visit_BinopNode(self, node):
>    # Very naive, I know things are more complicated...
>    return '%s %s %s' % (self.visit(node.operand1), node.operator,
> self.visit(node.operand2))
>
> def visit_FuncNode(self, node):
>    # Here we would return a StringIOTree instead (see
> Cython/StringIOTree.py),
>    # for efficiency.
>
> or, perhaps just:
>
> def visit_BinopNode(self, node):
>    self.visit(node.operand1)
>    self.put(node.operator)
>    self.visit(node.operand2)
>
> So generate_... would never be called, on *any* node, when doing .NET
> output.
>
> One could deal with shared CPython/.NET code by creating a superclass:
>
> class CLikeCodeGenerator:
>    # pull out stuff that is common between CPython and .NET here
>
> class DotNetCodeGenerator(CLikeCodeGenerator):
>    # .NET-specific nodes handled here
>
> class CPythonCodeGenerator(CLikeCodeGenerator):
>    def visit_Node(self, node):
>        # for now, just call generate_... if the node wasn't handled
>        # by CLikeCodeGenerator
>
>
> This makes it easy to plug in Java as well without making Nodes.py and
> ExprNodes.py even larger and more tangled and more full of conditionals...

Ah, yes, I remember discussing this before. If that could be pulled
off, it would be nice and clean. The fact that memory management is
handled for you (I think) makes things much simpler.

My one concerns is if (taken to the extreem) having a visitor that
knows how to generate code for every type (and the requisite
internals) is preferable to encapsulating this knowledge on a
node-by-node basis. Perhaps.

- Robert
Carl Witty | 5 Aug 19:32 2010
Picon

Re: [Cython] working on Cython for IronPython

On Thu, Aug 5, 2010 at 3:56 AM, Robert Bradshaw
<robertwb@...> wrote:
> My one concerns is if (taken to the extreem) having a visitor that
> knows how to generate code for every type (and the requisite
> internals) is preferable to encapsulating this knowledge on a
> node-by-node basis. Perhaps.

Ah yes... the perennial style debate: if you have a bunch of data
types, and a bunch of operations on those types, is it better to group
the code by type, or by operation?  (And of course there's no "right"
answer to such a generic question.)

AFAIK, other compilers (like gcc and LLVM) group code by operation --
I think they would typically have a source file for each compiler
pass, for example.  On the other hand, Cython probably has more node
types (and semantically richer node types), and fewer operations, than
those compilers, so the right answer for them isn't necessarily right
for Cython...

> - Robert

Carl

Gmane