Yang Zhang | 3 Oct 21:42 2011
Picon

Long-standing issue with using numpy in embedded CPython

It turns out that there's a long-standing problem in numpy that
prevents it from being used in embedded CPython environments:

http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-why-does-this-hang/7630992#7630992
http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html
Is there any fix or workaround for this?  Thanks.
--

-- 
Yang Zhang
http://yz.mit.edu/
Olivier Delalleau | 3 Oct 21:51 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

As far as a workaround in concerned, that scipy archive post says you can disable threads in numpy.
Sorry can't help more, I don't know much about how to bypass such GIL issues.

-=- Olivier

2011/10/3 Yang Zhang <yanghatespam <at> gmail.com>
It turns out that there's a long-standing problem in numpy that
prevents it from being used in embedded CPython environments:

http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-why-does-this-hang/7630992#7630992
http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html
Is there any fix or workaround for this?  Thanks.
--
Yang Zhang
http://yz.mit.edu/
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Robin | 4 Oct 10:28 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam <at> gmail.com> wrote:
> It turns out that there's a long-standing problem in numpy that
> prevents it from being used in embedded CPython environments:

Just wanted to make the point for reference that in general Numpy does
work fine in (non-threaded) embedded CPython situations, see for
example pymex [1] which embeds Python + Numpy in a Matlab mex file and
works really well.

This seems to a be a problem specific to Jepp.

Just wanted to mention it in case it puts someone off trying something
unnecessarily in the future.

Cheers

Robin

[1] https://github.com/kw/pymex

>
> http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-why-does-this-hang/7630992#7630992
> http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html
> Is there any fix or workaround for this?  Thanks.
> --
> Yang Zhang
> http://yz.mit.edu/
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
Yang Zhang | 4 Oct 21:05 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince <at> gmail.com> wrote:
> On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>> It turns out that there's a long-standing problem in numpy that
>> prevents it from being used in embedded CPython environments:
>
> Just wanted to make the point for reference that in general Numpy does
> work fine in (non-threaded) embedded CPython situations, see for
> example pymex [1] which embeds Python + Numpy in a Matlab mex file and
> works really well.
>
> This seems to a be a problem specific to Jepp.
>
> Just wanted to mention it in case it puts someone off trying something
> unnecessarily in the future.

My (second-hand) understanding is that this is a problem with having
multiple CPython interpreters, which both Jepp and numpy utilize,
incompatibly - is that right?  I.e., if either one were restricted to
using a single CPython interpreter, we wouldn't see this problem?

I'm curious how to disable threads in numpy (not an ideal solution).
Googling seems to point me to setting NPY_ALLOW_THREADS to
0....somewhere.

>
> Cheers
>
> Robin
>
> [1] https://github.com/kw/pymex
>
>>
>> http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-why-does-this-hang/7630992#7630992
>> http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html
>> Is there any fix or workaround for this?  Thanks.
>> --
>> Yang Zhang
>> http://yz.mit.edu/
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion <at> scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>

--

-- 
Yang Zhang
http://yz.mit.edu/
Yang Zhang | 9 Dec 00:15 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam <at> gmail.com> wrote:
> On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince <at> gmail.com> wrote:
>> On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>>> It turns out that there's a long-standing problem in numpy that
>>> prevents it from being used in embedded CPython environments:
>>
>> Just wanted to make the point for reference that in general Numpy does
>> work fine in (non-threaded) embedded CPython situations, see for
>> example pymex [1] which embeds Python + Numpy in a Matlab mex file and
>> works really well.
>>
>> This seems to a be a problem specific to Jepp.
>>
>> Just wanted to mention it in case it puts someone off trying something
>> unnecessarily in the future.
>
> My (second-hand) understanding is that this is a problem with having
> multiple CPython interpreters, which both Jepp and numpy utilize,
> incompatibly - is that right?  I.e., if either one were restricted to
> using a single CPython interpreter, we wouldn't see this problem?
>
> I'm curious how to disable threads in numpy (not an ideal solution).
> Googling seems to point me to setting NPY_ALLOW_THREADS to
> 0....somewhere.

Anyone?

>
>>
>> Cheers
>>
>> Robin
>>
>> [1] https://github.com/kw/pymex
>>
>>>
>>> http://stackoverflow.com/questions/7592565/when-embedding-cpython-in-java-why-does-this-hang/7630992#7630992
>>> http://mail.scipy.org/pipermail/numpy-discussion/2009-July/044046.html
>>> Is there any fix or workaround for this?  Thanks.
>>> --
>>> Yang Zhang
>>> http://yz.mit.edu/
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion <at> scipy.org
>>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion <at> scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
>
> --
> Yang Zhang
> http://yz.mit.edu/

--

-- 
Yang Zhang
http://yz.mit.edu/
Robert Kern | 9 Dec 09:31 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

On Thu, Dec 8, 2011 at 23:15, Yang Zhang <yanghatespam <at> gmail.com> wrote:
> On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>> On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince <at> gmail.com> wrote:
>>> On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>>>> It turns out that there's a long-standing problem in numpy that
>>>> prevents it from being used in embedded CPython environments:
>>>
>>> Just wanted to make the point for reference that in general Numpy does
>>> work fine in (non-threaded) embedded CPython situations, see for
>>> example pymex [1] which embeds Python + Numpy in a Matlab mex file and
>>> works really well.
>>>
>>> This seems to a be a problem specific to Jepp.
>>>
>>> Just wanted to mention it in case it puts someone off trying something
>>> unnecessarily in the future.
>>
>> My (second-hand) understanding is that this is a problem with having
>> multiple CPython interpreters, which both Jepp and numpy utilize,
>> incompatibly - is that right?  I.e., if either one were restricted to
>> using a single CPython interpreter, we wouldn't see this problem?
>>
>> I'm curious how to disable threads in numpy (not an ideal solution).
>> Googling seems to point me to setting NPY_ALLOW_THREADS to
>> 0....somewhere.
>
> Anyone?

numpy does not use multiple interpreters. The threading options have
nothing to do with multiple interpreters, and will not let you use
multiple CPython interpreters in your application. The problem is that
Python does not have good isolation between multiple interpreters for
extension modules. Many extension modules happen to work in this
environment, but numpy is not one of them. We have some global state
that we need to keep, and this gets interfered with in a multiple
interpreter environment.

--

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Yang Zhang | 9 Dec 12:00 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

On Fri, Dec 9, 2011 at 12:31 AM, Robert Kern <robert.kern <at> gmail.com> wrote:
> On Thu, Dec 8, 2011 at 23:15, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>> On Tue, Oct 4, 2011 at 12:05 PM, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>>> On Tue, Oct 4, 2011 at 1:28 AM, Robin <robince <at> gmail.com> wrote:
>>>> On Mon, Oct 3, 2011 at 9:42 PM, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>>>>> It turns out that there's a long-standing problem in numpy that
>>>>> prevents it from being used in embedded CPython environments:
>>>>
>>>> Just wanted to make the point for reference that in general Numpy does
>>>> work fine in (non-threaded) embedded CPython situations, see for
>>>> example pymex [1] which embeds Python + Numpy in a Matlab mex file and
>>>> works really well.
>>>>
>>>> This seems to a be a problem specific to Jepp.
>>>>
>>>> Just wanted to mention it in case it puts someone off trying something
>>>> unnecessarily in the future.
>>>
>>> My (second-hand) understanding is that this is a problem with having
>>> multiple CPython interpreters, which both Jepp and numpy utilize,
>>> incompatibly - is that right?  I.e., if either one were restricted to
>>> using a single CPython interpreter, we wouldn't see this problem?
>>>
>>> I'm curious how to disable threads in numpy (not an ideal solution).
>>> Googling seems to point me to setting NPY_ALLOW_THREADS to
>>> 0....somewhere.
>>
>> Anyone?
>
> numpy does not use multiple interpreters. The threading options have
> nothing to do with multiple interpreters, and will not let you use
> multiple CPython interpreters in your application. The problem is that
> Python does not have good isolation between multiple interpreters for
> extension modules. Many extension modules happen to work in this
> environment, but numpy is not one of them. We have some global state
> that we need to keep, and this gets interfered with in a multiple
> interpreter environment.

Thanks for the clarification.  Alas.  So is there no simple workaround
to making numpy work in environments such as Jepp?

>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless
> enigma that is made terrible by our own mad attempt to interpret it as
> though it had an underlying truth."
>   -- Umberto Eco
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

--

-- 
Yang Zhang
http://yz.mit.edu/
Robert Kern | 9 Dec 12:05 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam <at> gmail.com> wrote:

> Thanks for the clarification.  Alas.  So is there no simple workaround
> to making numpy work in environments such as Jepp?

I don't think so, no.

--

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Vicente Sole | 9 Dec 14:05 2011
Picon
Picon

Re: Long-standing issue with using numpy in embedded CPython

Quoting Robert Kern <robert.kern <at> gmail.com>:

> On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>
>> Thanks for the clarification.  Alas.  So is there no simple workaround
>> to making numpy work in environments such as Jepp?
>
> I don't think so, no.
>

It is far from being an optimal solution (in fact I dislike it) but  
there is a couple of research facilities that like the python  
interpreter, they like numpy, but prefer to use java for all their  
graphical interfaces. They have rewritten part of numpy in java in  
order to use it from Jython.

http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.16/scisoftpy.html

Armando
Peter CYC | 9 Dec 16:53 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

Hi Armando,

No comment on the Java thing ;-)

However, http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.18/contents.html
is more up-to-date and we are on github too:
https://github.com/DiamondLightSource

Peter

On 9 December 2011 13:05, Vicente Sole <sole <at> esrf.fr> wrote:
> Quoting Robert Kern <robert.kern <at> gmail.com>:
>
>> On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>>
>>> Thanks for the clarification.  Alas.  So is there no simple workaround
>>> to making numpy work in environments such as Jepp?
>>
>> I don't think so, no.
>>
>
> It is far from being an optimal solution (in fact I dislike it) but
> there is a couple of research facilities that like the python
> interpreter, they like numpy, but prefer to use java for all their
> graphical interfaces. They have rewritten part of numpy in java in
> order to use it from Jython.
>
> http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.16/scisoftpy.html
>
>
> Armando
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
Vicente Sole | 10 Dec 14:43 2011
Picon
Picon

Re: Long-standing issue with using numpy in embedded CPython

Hi Peter,

The obsolete link was not deliberate. It was the first reference I  
found via google.

Best regards,

Armando

Quoting Peter CYC <pcyc.uk <at> gmail.com>:

> Hi Armando,
>
> No comment on the Java thing ;-)
>
> However,   
> http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.18/contents.html
> is more up-to-date and we are on github too:
> https://github.com/DiamondLightSource
>
> Peter
>
>
> On 9 December 2011 13:05, Vicente Sole <sole <at> esrf.fr> wrote:
>> Quoting Robert Kern <robert.kern <at> gmail.com>:
>>
>>> On Fri, Dec 9, 2011 at 11:00, Yang Zhang <yanghatespam <at> gmail.com> wrote:
>>>
>>>> Thanks for the clarification.  Alas.  So is there no simple workaround
>>>> to making numpy work in environments such as Jepp?
>>>
>>> I don't think so, no.
>>>
>>
>> It is far from being an optimal solution (in fact I dislike it) but
>> there is a couple of research facilities that like the python
>> interpreter, they like numpy, but prefer to use java for all their
>> graphical interfaces. They have rewritten part of numpy in java in
>> order to use it from Jython.
>>
>> http://www.opengda.org/documentation/manuals/Diamond_SciSoft_Python_Guide/8.16/scisoftpy.html
>>
>>
>> Armando
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion <at> scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
Pierre Haessig | 9 Dec 14:18 2011

Re: Long-standing issue with using numpy in embedded CPython

Le 09/12/2011 09:31, Robert Kern a écrit :
> We have some global state
> that we need to keep, and this gets interfered with in a multiple
> interpreter environment.
I recently got interested in multiprocessing computation with numpy and 
now I get scare by your statement !
Please don't tell me it is unsafe to launch multiple jobs (for instance 
with multiprocressing's Pool.map) just doing some ndarray arithmetic !

That's why I'd like to understand better the issue raised by Yang. For 
instance, what does exactly "multiple CPython interpreters" stands for ?

Best,
Pierre
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Robert Kern | 9 Dec 15:00 2011
Picon

Re: Long-standing issue with using numpy in embedded CPython

On Fri, Dec 9, 2011 at 13:18, Pierre Haessig <pierre.haessig <at> crans.org> wrote:
> Le 09/12/2011 09:31, Robert Kern a écrit :
>> We have some global state
>> that we need to keep, and this gets interfered with in a multiple
>> interpreter environment.
> I recently got interested in multiprocessing computation with numpy and
> now I get scare by your statement !
> Please don't tell me it is unsafe to launch multiple jobs (for instance
> with multiprocressing's Pool.map) just doing some ndarray arithmetic !
>
> That's why I'd like to understand better the issue raised by Yang. For
> instance, what does exactly "multiple CPython interpreters" stands for ?

Using multiprocessing is fine. That starts up multiple interpreters in
*different* processes. Yang is using a non-Python program that embeds
the CPython interpreter and starts up multiple copies of it in the
same process.

--

-- 
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
  -- Umberto Eco
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Pierre Haessig | 9 Dec 15:28 2011

Re: Long-standing issue with using numpy in embedded CPython

Le 09/12/2011 15:00, Robert Kern a écrit :
Using multiprocessing is fine. That starts up multiple interpreters in *different* processes. Yang is using a non-Python program that embeds the CPython interpreter and starts up multiple copies of it in the same process.
Ok, now I think I understand. I was not aware it was possible to embed multiple CPython instances  into one process.
So I guess IPython's multiprocessing infrastructure I once briefly considered is also safe since it runs multiple kernels. I'm relieved to hear that !

Thank you very much for the explanation.

Pierre
_______________________________________________
NumPy-Discussion mailing list
NumPy-Discussion <at> scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Raphael de Feraudy | 24 Sep 17:54 2012

Re: Long-standing issue with using numpy in embedded CPython

Yang Zhang <yanghatespam <at> gmail.com> writes:

> 
> > I'm curious how to disable threads in numpy (not an ideal solution).
> > Googling seems to point me to setting NPY_ALLOW_THREADS to
> > 0....somewhere.
> 
> Anyone?
> 

It's appearing to me I had to face this very issue,
which I reported  <at> Numpy TRAC : http://projects.scipy.org/numpy/ticket/2213.

I just tried your suggestion :
set NPY_ALLOW_THREADS to 0 in numpy/core/include/numpy/ndarraytypes.h.
It allowed my atomic example to run without stalling,
and also fixed the issue in my application.

Though i'm not entirely satisfied by this workaround,
which might slow down heavy computations.
I also find it too intrusive in numpy source code 
and don't wish to maintain a powerless numpy fork.

Has anyone else settled with this fix ? 
Or may anybody have any other suggestion / comments ?

Thanks.
Raphael.
Aron Ahmadia | 25 Sep 13:10 2012
Picon

Re: Long-standing issue with using numpy in embedded CPython

Can you expand a bit?  Are you trying to disable threads at
compile-time or at run-time?  Which threaded functionality are you
trying to disable?  Are you using numpy as a computational library
with multiple threads making calls into its functions?  I think
NPY_ALLOW_THREADS is for interacting with the GIL, but I have not
played with it much.

A

On Mon, Sep 24, 2012 at 6:54 PM, Raphael de Feraudy <feraudy <at> phimeca.com> wrote:
> Yang Zhang <yanghatespam <at> gmail.com> writes:
>
>>
>> > I'm curious how to disable threads in numpy (not an ideal solution).
>> > Googling seems to point me to setting NPY_ALLOW_THREADS to
>> > 0....somewhere.
>>
>> Anyone?
>>
>
> It's appearing to me I had to face this very issue,
> which I reported  <at> Numpy TRAC : http://projects.scipy.org/numpy/ticket/2213.
>
> I just tried your suggestion :
> set NPY_ALLOW_THREADS to 0 in numpy/core/include/numpy/ndarraytypes.h.
> It allowed my atomic example to run without stalling,
> and also fixed the issue in my application.
>
> Though i'm not entirely satisfied by this workaround,
> which might slow down heavy computations.
> I also find it too intrusive in numpy source code
> and don't wish to maintain a powerless numpy fork.
>
> Has anyone else settled with this fix ?
> Or may anybody have any other suggestion / comments ?
>
> Thanks.
> Raphael.
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion <at> scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

Gmane