Stefan Behnel | 2 Mar 12:35 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Dag Sverre Seljebotn, 02.03.2011 11:54:
> On 03/02/2011 11:48 AM, Stefan Behnel wrote:
>> Dag Sverre Seljebotn, 02.03.2011 11:20:
>>> c) Somehow provide more than one module in the same compilation unit.
>>> Again, this requires the build to work correctly, but seems less dangerous,
>>> and also has the advantage of *allowing* static linking of the Fortran
>>> library, if one wants to.
>>>
>>> But is something like this possible? Could one implement
>>>
>>> cython -o assembly.c file1.pyx file2.pyx file3.pyx
>>>
>>> ...where assembly.so would contain three Python modules? (initfile1,
>>> initfile2, and so on...)
>>
>> Can't currently work because the three modules would define the same
>> static C names. This could be fixed as part of the PEP 3121 implementation:
>>
>> http://trac.cython.org/cython_trac/ticket/173
>>
>> Or it could be worked around by overriding the prefixes in Naming.py
>> (which sounds ugly).
>>
>> Generally speaking, I'm -1 on this idea. I don't see a real use case, and
>> you're saying yourself that this isn't required to make your Fortran use
>> case work either.
>
> But assuming work is spent on Cython, it *could* work? I.e. there's not a
> problem with import mechanisms

(Continue reading)

Lisandro Dalcin | 2 Mar 16:11 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 2 March 2011 08:35, Stefan Behnel <stefan_ml@...> wrote:
> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>
>> Problem is that Fortran code often has...interesting...programming
>> practices. Global variables abound, and are often initialised between
>> modules. Imagine:
>>
>> settings_mod.set_alpha(0.34)
>> print compute_mod.get_alpha_squared()
>>
>> This behaves quite differently with two static versions rather than one...
>
> Then I'd suggest always linking dynamically.
>

And where are you going to put your fortran shared libraries? Dynamic
linking details varies wildly across platforms... I'm very much
understand Dag's use case and concerns, and I do think that some
research in all this is worth it.

--

-- 
Lisandro Dalcin
---------------
CIMEC (INTEC/CONICET-UNL)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1011)
Tel/Fax: +54-342-4511169
(Continue reading)

Dag Sverre Seljebotn | 2 Mar 16:37 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

On 03/02/2011 04:11 PM, Lisandro Dalcin wrote:
> On 2 March 2011 08:35, Stefan Behnel<stefan_ml@...>  wrote:
>> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>> Problem is that Fortran code often has...interesting...programming
>>> practices. Global variables abound, and are often initialised between
>>> modules. Imagine:
>>>
>>> settings_mod.set_alpha(0.34)
>>> print compute_mod.get_alpha_squared()
>>>
>>> This behaves quite differently with two static versions rather than one...
>> Then I'd suggest always linking dynamically.
>>
> And where are you going to put your fortran shared libraries? Dynamic
> linking details varies wildly across platforms... I'm very much
> understand Dag's use case and concerns, and I do think that some
> research in all this is worth it.

I'm not sure if there's much more to research at the moment -- Stefan 
says it is possible, and that's what I wanted to know at this stage. If 
I want it, I obviously need to implement it myself. (And if such a patch 
implements PEP 3121 and there's a demonstrated need for it with some 
users, I really can't see it getting rejected just out of it being in 
"poor taste").

I.e., I'm going to make Fwrap spit out multiple pyx files and worry 
about this later. If multiple .pyx in one .so was fundamentally 
impossible, I might have gone another route with Fwrap. That was all.

Thanks Stefan!,
(Continue reading)

Stefan Behnel | 2 Mar 17:01 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Dag Sverre Seljebotn, 02.03.2011 16:37:
> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote:
>> On 2 March 2011 08:35, Stefan Behnel<stefan_ml@...> wrote:
>>> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>>> Problem is that Fortran code often has...interesting...programming
>>>> practices. Global variables abound, and are often initialised between
>>>> modules. Imagine:
>>>>
>>>> settings_mod.set_alpha(0.34)
>>>> print compute_mod.get_alpha_squared()
>>>>
>>>> This behaves quite differently with two static versions rather than one...
>>> Then I'd suggest always linking dynamically.
>>>
>> And where are you going to put your fortran shared libraries? Dynamic
>> linking details varies wildly across platforms... I'm very much
>> understand Dag's use case and concerns, and I do think that some
>> research in all this is worth it.
>
> I'm not sure if there's much more to research at the moment -- Stefan says
> it is possible, and that's what I wanted to know at this stage. If I want
> it, I obviously need to implement it myself. (And if such a patch
> implements PEP 3121 and there's a demonstrated need for it with some users,
> I really can't see it getting rejected just out of it being in "poor taste").
>
> I.e., I'm going to make Fwrap spit out multiple pyx files and worry about
> this later. If multiple .pyx in one .so was fundamentally impossible, I
> might have gone another route with Fwrap. That was all.

The feature I could imagine becoming part of Cython is "compiling 
(Continue reading)

Dag Sverre Seljebotn | 2 Mar 18:31 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

On 03/02/2011 05:01 PM, Stefan Behnel wrote:
> Dag Sverre Seljebotn, 02.03.2011 16:37:
>> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote:
>>> On 2 March 2011 08:35, Stefan Behnel<stefan_ml@...> wrote:
>>>> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>>>> Problem is that Fortran code often has...interesting...programming
>>>>> practices. Global variables abound, and are often initialised between
>>>>> modules. Imagine:
>>>>>
>>>>> settings_mod.set_alpha(0.34)
>>>>> print compute_mod.get_alpha_squared()
>>>>>
>>>>> This behaves quite differently with two static versions rather 
>>>>> than one...
>>>> Then I'd suggest always linking dynamically.
>>>>
>>> And where are you going to put your fortran shared libraries? Dynamic
>>> linking details varies wildly across platforms... I'm very much
>>> understand Dag's use case and concerns, and I do think that some
>>> research in all this is worth it.
>>
>> I'm not sure if there's much more to research at the moment -- Stefan 
>> says
>> it is possible, and that's what I wanted to know at this stage. If I 
>> want
>> it, I obviously need to implement it myself. (And if such a patch
>> implements PEP 3121 and there's a demonstrated need for it with some 
>> users,
>> I really can't see it getting rejected just out of it being in "poor 
>> taste").
(Continue reading)

Lisandro Dalcin | 2 Mar 19:58 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 2 March 2011 13:01, Stefan Behnel <stefan_ml@...> wrote:
> Dag Sverre Seljebotn, 02.03.2011 16:37:
>>
>> On 03/02/2011 04:11 PM, Lisandro Dalcin wrote:
>>>
>>> On 2 March 2011 08:35, Stefan Behnel<stefan_ml@...> wrote:
>>>>
>>>> Dag Sverre Seljebotn, 02.03.2011 11:54:
>>>>>
>>>>> Problem is that Fortran code often has...interesting...programming
>>>>> practices. Global variables abound, and are often initialised between
>>>>> modules. Imagine:
>>>>>
>>>>> settings_mod.set_alpha(0.34)
>>>>> print compute_mod.get_alpha_squared()
>>>>>
>>>>> This behaves quite differently with two static versions rather than
>>>>> one...
>>>>
>>>> Then I'd suggest always linking dynamically.
>>>>
>>> And where are you going to put your fortran shared libraries? Dynamic
>>> linking details varies wildly across platforms... I'm very much
>>> understand Dag's use case and concerns, and I do think that some
>>> research in all this is worth it.
>>
>> I'm not sure if there's much more to research at the moment -- Stefan says
>> it is possible, and that's what I wanted to know at this stage. If I want
>> it, I obviously need to implement it myself. (And if such a patch
>> implements PEP 3121 and there's a demonstrated need for it with some
(Continue reading)

Greg Ewing | 3 Mar 01:01 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

Stefan Behnel wrote:
> you'd call "cython" on a package and it would output a 
> directory with a single __init__.so that contains the modules compiled 
> from all .pyx/.py files in that package. Importing the package would 
> then trigger an import of that __init__.so, which in turn will execute 
> code in its init__init__() function to register the other modules.

I don't think it even has to be a directory with an __init__,
it could just be an ordinary .so file with the name of the
package.

I just tried an experiment in Python:

# onefilepackage.py
import new, sys
blarg = new.module("blarg")
blarg.thing = "This is the thing"
sys.modules["onefilepackage.blarg"] = blarg

and two different ways of importing it:

 >>> from onefilepackage import blarg
 >>> blarg
<module 'blarg' (built-in)>
 >>> blarg.thing
'This is the thing'

 >>> import onefilepackage.blarg
 >>> onefilepackage.blarg.thing
'This is the thing'
(Continue reading)

Lisandro Dalcin | 3 Mar 05:38 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 2 March 2011 21:01, Greg Ewing <greg.ewing@...> wrote:
> Stefan Behnel wrote:
>>
>> you'd call "cython" on a package and it would output a directory with a
>> single __init__.so that contains the modules compiled from all .pyx/.py
>> files in that package. Importing the package would then trigger an import of
>> that __init__.so, which in turn will execute code in its init__init__()
>> function to register the other modules.
>
> I don't think it even has to be a directory with an __init__,
> it could just be an ordinary .so file with the name of the
> package.
>
> I just tried an experiment in Python:
>
> # onefilepackage.py
> import new, sys
> blarg = new.module("blarg")
> blarg.thing = "This is the thing"
> sys.modules["onefilepackage.blarg"] = blarg
>
> and two different ways of importing it:
>
>>>> from onefilepackage import blarg
>>>> blarg
> <module 'blarg' (built-in)>
>>>> blarg.thing
> 'This is the thing'
>
>>>> import onefilepackage.blarg
(Continue reading)

Stefan Behnel | 3 Mar 07:43 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Lisandro Dalcin, 03.03.2011 05:38:
> On 2 March 2011 21:01, Greg Ewing<greg.ewing@...>  wrote:
>> Stefan Behnel wrote:
>>>
>>> you'd call "cython" on a package and it would output a directory with a
>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>> files in that package. Importing the package would then trigger an import of
>>> that __init__.so, which in turn will execute code in its init__init__()
>>> function to register the other modules.
>>
>> I don't think it even has to be a directory with an __init__,
>> it could just be an ordinary .so file with the name of the
>> package.
>>
>> I just tried an experiment in Python:
>>
>> # onefilepackage.py
>> import new, sys
>> blarg = new.module("blarg")
>> blarg.thing = "This is the thing"
>> sys.modules["onefilepackage.blarg"] = blarg
>>
>> and two different ways of importing it:
>>
>> >>> from onefilepackage import blarg
>> >>> blarg
>> <module 'blarg' (built-in)>
>> >>> blarg.thing
>> 'This is the thing'
>>
(Continue reading)

mark florisson | 3 Mar 10:32 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 3 March 2011 07:43, Stefan Behnel <stefan_ml <at> behnel.de> wrote:
> Lisandro Dalcin, 03.03.2011 05:38:
>>
>> On 2 March 2011 21:01, Greg Ewing<greg.ewing <at> canterbury.ac.nz>  wrote:
>>>
>>> Stefan Behnel wrote:
>>>>
>>>> you'd call "cython" on a package and it would output a directory with a
>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>> files in that package. Importing the package would then trigger an
>>>> import of
>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>> function to register the other modules.
>>>
>>> I don't think it even has to be a directory with an __init__,
>>> it could just be an ordinary .so file with the name of the
>>> package.
>>>
>>> I just tried an experiment in Python:
>>>
>>> # onefilepackage.py
>>> import new, sys
>>> blarg = new.module("blarg")
>>> blarg.thing = "This is the thing"
>>> sys.modules["onefilepackage.blarg"] = blarg
>>>
>>> and two different ways of importing it:
>>>
>>> >>> from onefilepackage import blarg
>>> >>> blarg
(Continue reading)

Vitja Makarov | 3 Mar 10:48 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

2011/3/3 mark florisson <markflorisson88@...>:
> On 3 March 2011 07:43, Stefan Behnel <stefan_ml@...> wrote:
>> Lisandro Dalcin, 03.03.2011 05:38:
>>>
>>> On 2 March 2011 21:01, Greg Ewing<greg.ewing@...>  wrote:
>>>>
>>>> Stefan Behnel wrote:
>>>>>
>>>>> you'd call "cython" on a package and it would output a directory with a
>>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>>> files in that package. Importing the package would then trigger an
>>>>> import of
>>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>>> function to register the other modules.
>>>>
>>>> I don't think it even has to be a directory with an __init__,
>>>> it could just be an ordinary .so file with the name of the
>>>> package.
>>>>
>>>> I just tried an experiment in Python:
>>>>
>>>> # onefilepackage.py
>>>> import new, sys
>>>> blarg = new.module("blarg")
>>>> blarg.thing = "This is the thing"
>>>> sys.modules["onefilepackage.blarg"] = blarg
>>>>
>>>> and two different ways of importing it:
>>>>
>>>> >>> from onefilepackage import blarg
(Continue reading)

mark florisson | 3 Mar 10:57 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

On 3 March 2011 10:48, Vitja Makarov <vitja.makarov <at> gmail.com> wrote:
> 2011/3/3 mark florisson <markflorisson88 <at> gmail.com>:
>> On 3 March 2011 07:43, Stefan Behnel <stefan_ml <at> behnel.de> wrote:
>>> Lisandro Dalcin, 03.03.2011 05:38:
>>>>
>>>> On 2 March 2011 21:01, Greg Ewing<greg.ewing <at> canterbury.ac.nz>  wrote:
>>>>>
>>>>> Stefan Behnel wrote:
>>>>>>
>>>>>> you'd call "cython" on a package and it would output a directory with a
>>>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>>>> files in that package. Importing the package would then trigger an
>>>>>> import of
>>>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>>>> function to register the other modules.
>>>>>
>>>>> I don't think it even has to be a directory with an __init__,
>>>>> it could just be an ordinary .so file with the name of the
>>>>> package.
>>>>>
>>>>> I just tried an experiment in Python:
>>>>>
>>>>> # onefilepackage.py
>>>>> import new, sys
>>>>> blarg = new.module("blarg")
>>>>> blarg.thing = "This is the thing"
>>>>> sys.modules["onefilepackage.blarg"] = blarg
>>>>>
>>>>> and two different ways of importing it:
>>>>>
(Continue reading)

Stefan Behnel | 3 Mar 10:59 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

Vitja Makarov, 03.03.2011 10:48:
> To share common sources is a good idea, we can also share "code" in
> libcython-<version>.so
> But then we should handle ABI compatibility problems.

There is little constant code overlap between modules, and putting that 
into a separate library would lead to performance regressions (no more 
inlining etc.).

Stefan
Robert Bradshaw | 3 Mar 20:14 2011

Re: [Cython] Multiple modules in one compilation unit

On Thu, Mar 3, 2011 at 1:59 AM, Stefan Behnel <stefan_ml@...> wrote:
> Vitja Makarov, 03.03.2011 10:48:
>>
>> To share common sources is a good idea, we can also share "code" in
>> libcython-<version>.so
>> But then we should handle ABI compatibility problems.
>
> There is little constant code overlap between modules, and putting that into
> a separate library would lead to performance regressions (no more inlining
> etc.).

There are types that would be nice to share, such as the binding C
function and generator types.

Both common headers and shared object libraries could be produced,
shared, managed by the cythonize(...) build command.

- Robert
Stefan Behnel | 3 Mar 10:58 2011
Picon

Re: [Cython] Multiple modules in one compilation unit

mark florisson, 03.03.2011 10:32:
> On 3 March 2011 07:43, Stefan Behnel wrote:
>> Lisandro Dalcin, 03.03.2011 05:38:
>>>
>>> On 2 March 2011 21:01, Greg Ewing wrote:
>>>>
>>>> Stefan Behnel wrote:
>>>>>
>>>>> you'd call "cython" on a package and it would output a directory with a
>>>>> single __init__.so that contains the modules compiled from all .pyx/.py
>>>>> files in that package. Importing the package would then trigger an
>>>>> import of
>>>>> that __init__.so, which in turn will execute code in its init__init__()
>>>>> function to register the other modules.
>>>>
>>>> I don't think it even has to be a directory with an __init__,
>>>> it could just be an ordinary .so file with the name of the
>>>> package.
>>>>
>>>> I just tried an experiment in Python:
>>>>
>>>> # onefilepackage.py
>>>> import new, sys
>>>> blarg = new.module("blarg")
>>>> blarg.thing = "This is the thing"
>>>> sys.modules["onefilepackage.blarg"] = blarg
>>>>
>>>> and two different ways of importing it:
>>>>
>>>> >>> from onefilepackage import blarg
(Continue reading)

Greg Ewing | 3 Mar 21:04 2011
Picon
Picon

Re: [Cython] Multiple modules in one compilation unit

Lisandro Dalcin wrote:
> However, in Python
> 3 that is not te case (and only for the .so, for .py is the same as in
> Py2), the import machinery adds the entry later, after the
> finalization of the module init function.

Might be an idea to raise this issue on python-dev, to see
if there is a reason for this and whether it could be changed.

> I'm tempted to workaround
> this by setting the entry in sys.modules right after the call to
> PyModule_Create() ... What do you think about this?

It's worth a try. I can't think of anything it might break
offhand -- but then I'm not really up with the play on what's
been happening to the importing machinery over the years.

--

-- 
Greg


Gmane