Dag Sverre Seljebotn | 2 Mar 11:20 2011
Picon
Picon

[Cython] Multiple modules in one compilation unit

This has been raised earlier, but I don't think there was such a 
demonstrative use-case as what I have now.

Fwrap is suppose to be able to wrap Fortran "modules", which is 
essentially a namespace mechanism. It makes sense to convert the 
namespace to Python by creating one Cython pyx file per Fortran module.

However, if you are wrapping a Fortran library this way, you suddenly 
have lots of opportunity to mess up the build:

  - If you build the Fortran code as a static library (rather 
common...), then each pyx file will have their own copy. This will link 
successfully but likely have a rather poor effect.

  - If you link each Cython file with only the corresponding Fortran 
file, things won't work (likely to get missing symbols from cross-module 
calls Fortran-side).

Yes, linking each Cython file to the same shared Fortran library should 
work. Still, this seems dangerous.

Options:
  a) Simply make sure to link with shared versions of Fortran libraries 
("this is a documentation problem")

  b) Use some other namespace mechanism in the same pyx (classes with 
static methods...)

  c) Somehow provide more than one module in the same compilation unit. 
Again, this requires the build to work correctly, but seems less 
(Continue reading)

Stefan Behnel | 2 Mar 11:48 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Dag Sverre Seljebotn, 02.03.2011 11:20:
> c) Somehow provide more than one module in the same compilation unit.
> Again, this requires the build to work correctly, but seems less dangerous,
> and also has the advantage of *allowing* static linking of the Fortran
> library, if one wants to.
>
> But is something like this possible? Could one implement
>
> cython -o assembly.c file1.pyx file2.pyx file3.pyx
>
> ...where assembly.so would contain three Python modules? (initfile1,
> initfile2, and so on...)

Can't currently work because the three modules would define the same static 
C names. This could be fixed as part of the PEP 3121 implementation:

http://trac.cython.org/cython_trac/ticket/173

Or it could be worked around by overriding the prefixes in Naming.py (which 
sounds ugly).

Generally speaking, I'm -1 on this idea. I don't see a real use case, and 
you're saying yourself that this isn't required to make your Fortran use 
case work either.

> - If you build the Fortran code as a static library (rather common...),
> then each pyx file will have their own copy. This will link successfully
> but likely have a rather poor effect.

So? lxml has two main modules, and if you build it statically against 
(Continue reading)

Dag Sverre Seljebotn | 2 Mar 11:54 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

On 03/02/2011 11:48 AM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 02.03.2011 11:20:
>> c) Somehow provide more than one module in the same compilation unit.
>> Again, this requires the build to work correctly, but seems less 
>> dangerous,
>> and also has the advantage of *allowing* static linking of the Fortran
>> library, if one wants to.
>>
>> But is something like this possible? Could one implement
>>
>> cython -o assembly.c file1.pyx file2.pyx file3.pyx
>>
>> ...where assembly.so would contain three Python modules? (initfile1,
>> initfile2, and so on...)
>
> Can't currently work because the three modules would define the same 
> static C names. This could be fixed as part of the PEP 3121 
> implementation:
>
> http://trac.cython.org/cython_trac/ticket/173
>
> Or it could be worked around by overriding the prefixes in Naming.py 
> (which sounds ugly).
>
> Generally speaking, I'm -1 on this idea. I don't see a real use case, 
> and you're saying yourself that this isn't required to make your 
> Fortran use case work either.

But assuming work is spent on Cython, it *could* work? I.e. there's not 
a problem with import mechanisms; Python assumes one module per .so or 
(Continue reading)

Stefan Behnel | 2 Mar 12:35 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Dag Sverre Seljebotn, 02.03.2011 11:54:
> On 03/02/2011 11:48 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 02.03.2011 11:20:
>>> c) Somehow provide more than one module in the same compilation unit.
>>> Again, this requires the build to work correctly, but seems less dangerous,
>>> and also has the advantage of *allowing* static linking of the Fortran
>>> library, if one wants to.
>>>
>>> But is something like this possible? Could one implement
>>>
>>> cython -o assembly.c file1.pyx file2.pyx file3.pyx
>>>
>>> ...where assembly.so would contain three Python modules? (initfile1,
>>> initfile2, and so on...)
>>
>> Can't currently work because the three modules would define the same
>> static C names. This could be fixed as part of the PEP 3121 implementation:
>>
>> http://trac.cython.org/cython_trac/ticket/173
>>
>> Or it could be worked around by overriding the prefixes in Naming.py
>> (which sounds ugly).
>>
>> Generally speaking, I'm -1 on this idea. I don't see a real use case, and
>> you're saying yourself that this isn't required to make your Fortran use
>> case work either.
>
> But assuming work is spent on Cython, it *could* work? I.e. there's not a
> problem with import mechanisms

(Continue reading)

Lisandro Dalcin | 2 Mar 16:11 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 2 March 2011 08:35, Stefan Behnel <stefan_ml@...> wrote:
> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>
>> Problem is that Fortran code often has...interesting...programming
>> practices. Global variables abound, and are often initialised between
>> modules. Imagine:
>>
>> settings_mod.set_alpha(0.34)
>> print compute_mod.get_alpha_squared()
>>
>> This behaves quite differently with two static versions rather than one...
>
> Then I'd suggest always linking dynamically.
>

And where are you going to put your fortran shared libraries? Dynamic
linking details varies wildly across platforms... I'm very much
understand Dag's use case and concerns, and I do think that some
research in all this is worth it.

--

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169
(Continue reading)

Dag Sverre Seljebotn | 2 Mar 16:37 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

On 03/02/2011 04:11 PM, Lisandro Dalcin wrote:
> On 2 March 2011 08:35, Stefan Behnel<stefan_ml@...>  wrote:
>> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>> Problem is that Fortran code often has...interesting...programming
>>> practices. Global variables abound, and are often initialised between
>>> modules. Imagine:
>>>
>>> settings_mod.set_alpha(0.34)
>>> print compute_mod.get_alpha_squared()
>>>
>>> This behaves quite differently with two static versions rather than one...
>> Then I'd suggest always linking dynamically.
>>
> And where are you going to put your fortran shared libraries? Dynamic
> linking details varies wildly across platforms... I'm very much
> understand Dag's use case and concerns, and I do think that some
> research in all this is worth it.

I'm not sure if there's much more to research at the moment -- Stefan 
says it is possible, and that's what I wanted to know at this stage. If 
I want it, I obviously need to implement it myself. (And if such a patch 
implements PEP 3121 and there's a demonstrated need for it with some 
users, I really can't see it getting rejected just out of it being in 
"poor taste").

I.e., I'm going to make Fwrap spit out multiple pyx files and worry 
about this later. If multiple .pyx in one .so was fundamentally 
impossible, I might have gone another route with Fwrap. That was all.

Thanks Stefan!,
(Continue reading)

Stefan Behnel | 2 Mar 17:01 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Dag Sverre Seljebotn, 02.03.2011 16:37:
> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote:
>> On 2 March 2011 08:35, Stefan Behnel<stefan_ml@...> wrote:
>>> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>>> Problem is that Fortran code often has...interesting...programming
>>>> practices. Global variables abound, and are often initialised between
>>>> modules. Imagine:
>>>>
>>>> settings_mod.set_alpha(0.34)
>>>> print compute_mod.get_alpha_squared()
>>>>
>>>> This behaves quite differently with two static versions rather than one...
>>> Then I'd suggest always linking dynamically.
>>>
>> And where are you going to put your fortran shared libraries? Dynamic
>> linking details varies wildly across platforms... I'm very much
>> understand Dag's use case and concerns, and I do think that some
>> research in all this is worth it.
>
> I'm not sure if there's much more to research at the moment -- Stefan says
> it is possible, and that's what I wanted to know at this stage. If I want
> it, I obviously need to implement it myself. (And if such a patch
> implements PEP 3121 and there's a demonstrated need for it with some users,
> I really can't see it getting rejected just out of it being in "poor taste").
>
> I.e., I'm going to make Fwrap spit out multiple pyx files and worry about
> this later. If multiple .pyx in one .so was fundamentally impossible, I
> might have gone another route with Fwrap. That was all.

The feature I could imagine becoming part of Cython is "compiling 
(Continue reading)

Dag Sverre Seljebotn | 2 Mar 18:31 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

On 03/02/2011 05:01 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 02.03.2011 16:37:
>> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote:
>>> On 2 March 2011 08:35, Stefan Behnel<stefan_ml@...> wrote:
>>>> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>>>> Problem is that Fortran code often has...interesting...programming
>>>>> practices. Global variables abound, and are often initialised between
>>>>> modules. Imagine:
>>>>>
>>>>> settings_mod.set_alpha(0.34)
>>>>> print compute_mod.get_alpha_squared()
>>>>>
>>>>> This behaves quite differently with two static versions rather 
>>>>> than one...
>>>> Then I'd suggest always linking dynamically.
>>>>
>>> And where are you going to put your fortran shared libraries? Dynamic
>>> linking details varies wildly across platforms... I'm very much
>>> understand Dag's use case and concerns, and I do think that some
>>> research in all this is worth it.
>>
>> I'm not sure if there's much more to research at the moment -- Stefan 
>> says
>> it is possible, and that's what I wanted to know at this stage. If I 
>> want
>> it, I obviously need to implement it myself. (And if such a patch
>> implements PEP 3121 and there's a demonstrated need for it with some 
>> users,
>> I really can't see it getting rejected just out of it being in "poor 
>> taste").
(Continue reading)

Lisandro Dalcin | 2 Mar 19:58 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 2 March 2011 13:01, Stefan Behnel <stefan_ml@...> wrote:
> Dag Sverre Seljebotn, 02.03.2011 16:37:
>>
>> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote:
>>>
>>> On 2 March 2011 08:35, Stefan Behnel<stefan_ml@...> wrote:
>>>>
>>>> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>>>>
>>>>> Problem is that Fortran code often has...interesting...programming
>>>>> practices. Global variables abound, and are often initialised between
>>>>> modules. Imagine:
>>>>>
>>>>> settings_mod.set_alpha(0.34)
>>>>> print compute_mod.get_alpha_squared()
>>>>>
>>>>> This behaves quite differently with two static versions rather than
>>>>> one...
>>>>
>>>> Then I'd suggest always linking dynamically.
>>>>
>>> And where are you going to put your fortran shared libraries? Dynamic
>>> linking details varies wildly across platforms... I'm very much
>>> understand Dag's use case and concerns, and I do think that some
>>> research in all this is worth it.
>>
>> I'm not sure if there's much more to research at the moment -- Stefan says
>> it is possible, and that's what I wanted to know at this stage. If I want
>> it, I obviously need to implement it myself. (And if such a patch
>> implements PEP 3121 and there's a demonstrated need for it with some
(Continue reading)

Greg Ewing | 3 Mar 01:01 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

Stefan Behnel wrote:
> you'd call "cython" on a package and it would output a 
> directory with a single __init__.so that contains the modules compiled 
> from all .pyx/.py files in that package. Importing the package would 
> then trigger an import of that __init__.so, which in turn will execute 
> code in its init__init__() function to register the other modules.

I don't think it even has to be a directory with an __init__,
it could just be an ordinary .so file with the name of the
package.

I just tried an experiment in Python:

# onefilepackage.py
import new, sys
blarg = new.module("blarg")
blarg.thing = "This is the thing"
sys.modules["onefilepackage.blarg"] = blarg

and two different ways of importing it:

 >>> from onefilepackage import blarg
 >>> blarg
<module 'blarg' (built-in)>
 >>> blarg.thing
'This is the thing'

 >>> import onefilepackage.blarg
 >>> onefilepackage.blarg.thing
'This is the thing'
(Continue reading)

Lisandro Dalcin | 3 Mar 05:38 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 2 March 2011 21:01, Greg Ewing <greg.ewing@...> wrote:
> Stefan Behnel wrote:
>>
>> you'd call "cython" on a package and it would output a directory with a
>> single __init__.so that contains the modules compiled from all .pyx/.py
>> files in that package. Importing the package would then trigger an import of
>> that __init__.so, which in turn will execute code in its init__init__()
>> function to register the other modules.
>
> I don't think it even has to be a directory with an __init__,
> it could just be an ordinary .so file with the name of the
> package.
>
> I just tried an experiment in Python:
>
> # onefilepackage.py
> import new, sys
> blarg = new.module("blarg")
> blarg.thing = "This is the thing"
> sys.modules["onefilepackage.blarg"] = blarg
>
> and two different ways of importing it:
>
>>>> from onefilepackage import blarg
>>>> blarg
> <module 'blarg' (built-in)>
>>>> blarg.thing
> 'This is the thing'
>
>>>> import onefilepackage.blarg
(Continue reading)

Stefan Behnel | 3 Mar 07:43 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Lisandro Dalcin, 03.03.2011 05:38:
> On 2 March 2011 21:01, Greg Ewing<greg.ewing@...>  wrote:
>> Stefan Behnel wrote:
>>>
>>> you'd call "cython" on a package and it would output a directory with a
>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>> files in that package. Importing the package would then trigger an import of
>>> that __init__.so, which in turn will execute code in its init__init__()
>>> function to register the other modules.
>>
>> I don't think it even has to be a directory with an __init__,
>> it could just be an ordinary .so file with the name of the
>> package.
>>
>> I just tried an experiment in Python:
>>
>> # onefilepackage.py
>> import new, sys
>> blarg = new.module("blarg")
>> blarg.thing = "This is the thing"
>> sys.modules["onefilepackage.blarg"] = blarg
>>
>> and two different ways of importing it:
>>
>> >>> from onefilepackage import blarg
>> >>> blarg
>> <module 'blarg' (built-in)>
>> >>> blarg.thing
>> 'This is the thing'
>>
(Continue reading)

mark florisson | 3 Mar 10:32 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 3 March 2011 07:43, Stefan Behnel <stefan_ml <at> behnel.de> wrote:
> Lisandro Dalcin, 03.03.2011 05:38:
>>
>> On 2 March 2011 21:01, Greg Ewing<greg.ewing <at> canterbury.ac.nz>  wrote:
>>>
>>> Stefan Behnel wrote:
>>>>
>>>> you'd call "cython" on a package and it would output a directory with a
>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>> files in that package. Importing the package would then trigger an
>>>> import of
>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>> function to register the other modules.
>>>
>>> I don't think it even has to be a directory with an __init__,
>>> it could just be an ordinary .so file with the name of the
>>> package.
>>>
>>> I just tried an experiment in Python:
>>>
>>> # onefilepackage.py
>>> import new, sys
>>> blarg = new.module("blarg")
>>> blarg.thing = "This is the thing"
>>> sys.modules["onefilepackage.blarg"] = blarg
>>>
>>> and two different ways of importing it:
>>>
>>> >>> from onefilepackage import blarg
>>> >>> blarg
(Continue reading)

Vitja Makarov | 3 Mar 10:48 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

2011/3/3 mark florisson <markflorisson88@...>:
> On 3 March 2011 07:43, Stefan Behnel <stefan_ml@...> wrote:
>> Lisandro Dalcin, 03.03.2011 05:38:
>>>
>>> On 2 March 2011 21:01, Greg Ewing<greg.ewing@...>  wrote:
>>>>
>>>> Stefan Behnel wrote:
>>>>>
>>>>> you'd call "cython" on a package and it would output a directory with a
>>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>>> files in that package. Importing the package would then trigger an
>>>>> import of
>>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>>> function to register the other modules.
>>>>
>>>> I don't think it even has to be a directory with an __init__,
>>>> it could just be an ordinary .so file with the name of the
>>>> package.
>>>>
>>>> I just tried an experiment in Python:
>>>>
>>>> # onefilepackage.py
>>>> import new, sys
>>>> blarg = new.module("blarg")
>>>> blarg.thing = "This is the thing"
>>>> sys.modules["onefilepackage.blarg"] = blarg
>>>>
>>>> and two different ways of importing it:
>>>>
>>>> >>> from onefilepackage import blarg
(Continue reading)

mark florisson | 3 Mar 10:57 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 3 March 2011 10:48, Vitja Makarov <vitja.makarov <at> gmail.com> wrote:
> 2011/3/3 mark florisson <markflorisson88 <at> gmail.com>:
>> On 3 March 2011 07:43, Stefan Behnel <stefan_ml <at> behnel.de> wrote:
>>> Lisandro Dalcin, 03.03.2011 05:38:
>>>>
>>>> On 2 March 2011 21:01, Greg Ewing<greg.ewing <at> canterbury.ac.nz>  wrote:
>>>>>
>>>>> Stefan Behnel wrote:
>>>>>>
>>>>>> you'd call "cython" on a package and it would output a directory with a
>>>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>>>> files in that package. Importing the package would then trigger an
>>>>>> import of
>>>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>>>> function to register the other modules.
>>>>>
>>>>> I don't think it even has to be a directory with an __init__,
>>>>> it could just be an ordinary .so file with the name of the
>>>>> package.
>>>>>
>>>>> I just tried an experiment in Python:
>>>>>
>>>>> # onefilepackage.py
>>>>> import new, sys
>>>>> blarg = new.module("blarg")
>>>>> blarg.thing = "This is the thing"
>>>>> sys.modules["onefilepackage.blarg"] = blarg
>>>>>
>>>>> and two different ways of importing it:
>>>>>
(Continue reading)

Stefan Behnel | 3 Mar 10:59 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Vitja Makarov, 03.03.2011 10:48:
> To share common sources is a good idea, we can also share "code" in
> libcython-<version>.so
> But then we should handle ABI compatibility problems.

There is little constant code overlap between modules, and putting that 
into a separate library would lead to performance regressions (no more 
inlining etc.).

Stefan
Robert Bradshaw | 3 Mar 20:14 2011

Re: [Cython] Multiple modules in one compilation unit

On Thu, Mar 3, 2011 at 1:59 AM, Stefan Behnel <stefan_ml@...> wrote:
> Vitja Makarov, 03.03.2011 10:48:
>>
>> To share common sources is a good idea, we can also share "code" in
>> libcython-<version>.so
>> But then we should handle ABI compatibility problems.
>
> There is little constant code overlap between modules, and putting that into
> a separate library would lead to performance regressions (no more inlining
> etc.).

There are types that would be nice to share, such as the binding C
function and generator types.

Both common headers and shared object libraries could be produced,
shared, managed by the cythonize(...) build command.

- Robert
Stefan Behnel | 3 Mar 10:58 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

mark florisson, 03.03.2011 10:32:
> On 3 March 2011 07:43, Stefan Behnel wrote:
>> Lisandro Dalcin, 03.03.2011 05:38:
>>>
>>> On 2 March 2011 21:01, Greg Ewing wrote:
>>>>
>>>> Stefan Behnel wrote:
>>>>>
>>>>> you'd call "cython" on a package and it would output a directory with a
>>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>>> files in that package. Importing the package would then trigger an
>>>>> import of
>>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>>> function to register the other modules.
>>>>
>>>> I don't think it even has to be a directory with an __init__,
>>>> it could just be an ordinary .so file with the name of the
>>>> package.
>>>>
>>>> I just tried an experiment in Python:
>>>>
>>>> # onefilepackage.py
>>>> import new, sys
>>>> blarg = new.module("blarg")
>>>> blarg.thing = "This is the thing"
>>>> sys.modules["onefilepackage.blarg"] = blarg
>>>>
>>>> and two different ways of importing it:
>>>>
>>>> >>> from onefilepackage import blarg
(Continue reading)

Greg Ewing | 3 Mar 21:04 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

Lisandro Dalcin wrote:
> However, in Python
> 3 that is not te case (and only for the .so, for .py is the same as in
> Py2), the import machinery adds the entry later, after the
> finalization of the module init function.

Might be an idea to raise this issue on python-dev, to see
if there is a reason for this and whether it could be changed.

> I'm tempted to workaround
> this by setting the entry in sys.modules right after the call to
> PyModule_Create() ... What do you think about this?

It's worth a try. I can't think of anything it might break
offhand -- but then I'm not really up with the play on what's
been happening to the importing machinery over the years.

--

-- 
Greg


Gmane