Steven G. Johnson | 21 Feb 03:08 2013
Picon

UPDATE: PyCall module to call Python functions from Julia

My PyCall module has made a lot of progress, and now supports much 
greater functionality and nicer syntax:

     https://github.com/stevengj/PyCall.jl

(NOTE: You need to patch Julia git master  as described here for most of 
this to work: https://github.com/JuliaLang/julia/pull/2317)

Some examples:

julia> using PyCall
julia> pyinitialize() # will go away once #2378 is fixed

julia>  <at> pyimport math
julia> math.sin(3) - sin(3)
0.0

julia>  <at> pyimport scipy.special as s
julia> s.airy(4.2)
(0.0006274958683091624,-0.0013210006638876843,124.03800986864213,246.14599171178563)

julia>  <at> pyimport numpy
julia> numpy.transpose(rand(3,5))
5x3 Float64 Array:
  0.139254  0.492393  0.617765
  0.971371  0.945398  0.55352
  0.883429  0.477562  0.604881
  0.622781  0.207569  0.918686
  0.117342  0.531707  0.350668

(Continue reading)

Fernando Perez | 21 Feb 04:17 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia

On Wed, Feb 20, 2013 at 6:08 PM, Steven G. Johnson <stevenj@...> wrote:
> julia> x = linspace(0,2*pi,100); y = sin(3*x + 4*cos(5*y));
> julia>  <at> pyimport pylab
> julia> pylab.plot(x,y)
> julia> pylab.show()
> ...opens plot window...

Awesome! Let's get that julia IPython kernel going, and we'll have
native Julia notebooks with inline plots in no time flat.  I'd love to
see the first julia section in our gallery
(https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks).

Cheers,

f

Rahul Dave | 21 Feb 04:32 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia

Fernando,
Working on this as soon as I finish my ipython/nltk/pandas course this week (for librarians!). 

First try is to basically copy the matlab magic (using the python-matlab bridge) to create a simple %%julia magic talking to the existing webserver implementation...

The second one would be to use Avik's zmq2 module to communicate directly with the tornado front end.

The third one would be (once the notebook js/css is refactored according to the roadmap you guys posted) to communicate directly with julia using the websockets protocol, but this is not as flexible as the second one as you cant mix in pythonic and R'ic and other fun.

Thoughts?

Are there any plans for any direct file editing in the notebook? RStudio has this very nice capability of sending stuff back and forth from the repl to the source and vice versa. And from history to source. Would be lovely to have this in the notebook....in tomorrow's course I'm finally getting to modules and code organization and such, and will be using a text editor. But a browser based one would be so good for teaching....

The notebook rocks!
Thanks!
Rahul

-- 
Rahul Dave
Sent with Sparrow

On Wednesday, February 20, 2013 at 10:17 PM, Fernando Perez wrote:

On Wed, Feb 20, 2013 at 6:08 PM, Steven G. Johnson <stevenj <at> alum.mit.edu> wrote:
julia> x = linspace(0,2*pi,100); y = sin(3*x + 4*cos(5*y));
julia> <at> pyimport pylab
julia> pylab.plot(x,y)
julia> pylab.show()
...opens plot window...


Awesome! Let's get that julia IPython kernel going, and we'll have
native Julia notebooks with inline plots in no time flat. I'd love to
see the first julia section in our gallery

Cheers,

f

Fernando Perez | 21 Feb 06:14 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia

Hi Rahul,

On Wed, Feb 20, 2013 at 7:32 PM, Rahul Dave
<rahuldave.mailinglists@...> wrote:
> Fernando,
> Working on this as soon as I finish my ipython/nltk/pandas course this week
> (for librarians!).

Great! BTW, I'll be giving a talk at MIT next Friday about IPython
(time/place TBD), if you can make it there I'd be happy to talk more
about the details there.

> First try is to basically copy the matlab magic (using the python-matlab
> bridge) to create a simple %%julia magic talking to the existing webserver
> implementation...

Yup.  That would be useful in and of itself, as it would also let
python users use Julia as a way to speed up loopy code.  I'd love to
have this also as an easy way of learning about Julia from the
comforts of code I already know well, the python side.

> The second one would be to use Avik's zmq2 module to communicate directly
> with the tornado front end.
>
> The third one would be (once the notebook js/css is refactored according to
> the roadmap you guys posted) to communicate directly with julia using the
> websockets protocol, but this is not as flexible as the second one as you
> cant mix in pythonic and R'ic and other fun.

Actually I don't think you should go to #3.  The way we think about
the architecture, kernels must speak the zmq protocol
(http://ipython.org/ipython-doc/rel-0.13.1/development/messaging.html).
 That's because the http server is a particular detail of running via
a  web browser, but for example the Qt console could also be used for
Julia, and other, different web services could be built using a
different server architecture.

The protocol *is* the zmq/json layer, that's where your kernel should
sit.  If you speak the protocol correctly, you'll be able to use every
compliant client, including the terminal, qt console and notebook.

I'm sure we'll find places where we've made assumptions too close to
python for this to work perfectly, but that's the point of this
exercise: to identify them and fix them so the protocol is really
language-agnostic.

> Thoughts?
>
> Are there any plans for any direct file editing in the notebook? RStudio has
> this very nice capability of sending stuff back and forth from the repl to
> the source and vice versa. And from history to source. Would be lovely to
> have this in the notebook....in tomorrow's course I'm finally getting to
> modules and code organization and such, and will be using a text editor. But
> a browser based one would be so good for teaching....

Yup, we know we need it.  Patience, patience :)

> The notebook rocks!

Glad you like it!

Cheers,

f

Stefan Karpinski | 21 Feb 16:58 2013

Re: UPDATE: PyCall module to call Python functions from Julia

This is incredibly exciting stuff all around.


On Thu, Feb 21, 2013 at 12:14 AM, Fernando Perez <fperez.net-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
Hi Rahul,

On Wed, Feb 20, 2013 at 7:32 PM, Rahul Dave
<rahuldave.mailinglists-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Fernando,
> Working on this as soon as I finish my ipython/nltk/pandas course this week
> (for librarians!).

Great! BTW, I'll be giving a talk at MIT next Friday about IPython
(time/place TBD), if you can make it there I'd be happy to talk more
about the details there.

> First try is to basically copy the matlab magic (using the python-matlab
> bridge) to create a simple %%julia magic talking to the existing webserver
> implementation...

Yup.  That would be useful in and of itself, as it would also let
python users use Julia as a way to speed up loopy code.  I'd love to
have this also as an easy way of learning about Julia from the
comforts of code I already know well, the python side.

> The second one would be to use Avik's zmq2 module to communicate directly
> with the tornado front end.
>
> The third one would be (once the notebook js/css is refactored according to
> the roadmap you guys posted) to communicate directly with julia using the
> websockets protocol, but this is not as flexible as the second one as you
> cant mix in pythonic and R'ic and other fun.

Actually I don't think you should go to #3.  The way we think about
the architecture, kernels must speak the zmq protocol
(http://ipython.org/ipython-doc/rel-0.13.1/development/messaging.html).
 That's because the http server is a particular detail of running via
a  web browser, but for example the Qt console could also be used for
Julia, and other, different web services could be built using a
different server architecture.

The protocol *is* the zmq/json layer, that's where your kernel should
sit.  If you speak the protocol correctly, you'll be able to use every
compliant client, including the terminal, qt console and notebook.

I'm sure we'll find places where we've made assumptions too close to
python for this to work perfectly, but that's the point of this
exercise: to identify them and fix them so the protocol is really
language-agnostic.

> Thoughts?
>
> Are there any plans for any direct file editing in the notebook? RStudio has
> this very nice capability of sending stuff back and forth from the repl to
> the source and vice versa. And from history to source. Would be lovely to
> have this in the notebook....in tomorrow's course I'm finally getting to
> modules and code organization and such, and will be using a text editor. But
> a browser based one would be so good for teaching....

Yup, we know we need it.  Patience, patience :)

> The notebook rocks!

Glad you like it!

Cheers,

f

Stefan Karpinski | 21 Feb 17:01 2013

Re: UPDATE: PyCall module to call Python functions from Julia

On Wed, Feb 20, 2013 at 9:08 PM, Steven G. Johnson <stevenj-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:

julia> using PyCall
julia> pyinitialize() # will go away once #2378 is fixed

Can't the pyinitialize() call go at the end of the PyCall module load? I know that might not be the best form, but it seems like it would be reasonable to do the pyinitilized there and it can also setup an atexit handler to call pydeinitialize (or whatever it's called, I forget).

Steven G. Johnson | 21 Feb 20:15 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia



On Thursday, February 21, 2013 11:01:10 AM UTC-5, Stefan Karpinski wrote:
On Wed, Feb 20, 2013 at 9:08 PM, Steven G. Johnson <ste...-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:

julia> using PyCall
julia> pyinitialize() # will go away once #2378 is fixed

Can't the pyinitialize() call go at the end of the PyCall module load? I know that might not be the best form, but it seems like it would be reasonable to do the pyinitilized there and it can also setup an atexit handler to call pydeinitialize (or whatever it's called, I forget).

No, pyinitialize() can't go in the module load.  The user has to have the option to call pyinitialize() manually (after loading the module but before calling any Python functions), for example to specify a different version of Python to load.

--SGJ
Stefan Karpinski | 21 Feb 20:54 2013

Re: UPDATE: PyCall module to call Python functions from Julia

Hmm. That tempts me to want to put perl-style hooks into using so that you can write things like "using PyCall python=2.7.3" or something like that. But that's probably an unhealthy inclination.


On Thu, Feb 21, 2013 at 2:15 PM, Steven G. Johnson <stevenj.mit-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:


On Thursday, February 21, 2013 11:01:10 AM UTC-5, Stefan Karpinski wrote:
On Wed, Feb 20, 2013 at 9:08 PM, Steven G. Johnson <ste...-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:

julia> using PyCall
julia> pyinitialize() # will go away once #2378 is fixed

Can't the pyinitialize() call go at the end of the PyCall module load? I know that might not be the best form, but it seems like it would be reasonable to do the pyinitilized there and it can also setup an atexit handler to call pydeinitialize (or whatever it's called, I forget).

No, pyinitialize() can't go in the module load.  The user has to have the option to call pyinitialize() manually (after loading the module but before calling any Python functions), for example to specify a different version of Python to load.

--SGJ

Gustavo Goretkin | 22 Feb 03:12 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia

Does this mean we can use matplotlib from julia?


On Thu, Feb 21, 2013 at 2:54 PM, Stefan Karpinski <stefan-X/JvLeVQC/3srOwW+9ziJQ@public.gmane.org> wrote:
Hmm. That tempts me to want to put perl-style hooks into using so that you can write things like "using PyCall python=2.7.3" or something like that. But that's probably an unhealthy inclination.


On Thu, Feb 21, 2013 at 2:15 PM, Steven G. Johnson <stevenj.mit-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:


On Thursday, February 21, 2013 11:01:10 AM UTC-5, Stefan Karpinski wrote:
On Wed, Feb 20, 2013 at 9:08 PM, Steven G. Johnson <ste...-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:

julia> using PyCall
julia> pyinitialize() # will go away once #2378 is fixed

Can't the pyinitialize() call go at the end of the PyCall module load? I know that might not be the best form, but it seems like it would be reasonable to do the pyinitilized there and it can also setup an atexit handler to call pydeinitialize (or whatever it's called, I forget).

No, pyinitialize() can't go in the module load.  The user has to have the option to call pyinitialize() manually (after loading the module but before calling any Python functions), for example to specify a different version of Python to load.

--SGJ


Steven G. Johnson | 22 Feb 05:46 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia

Gustavo Goretkin wrote:
> Does this mean we can use matplotlib from julia?

Yes.

Steven G. Johnson | 22 Feb 18:00 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia



On Thursday, February 21, 2013 11:01:10 AM UTC-5, Stefan Karpinski wrote:
On Wed, Feb 20, 2013 at 9:08 PM, Steven G. Johnson <ste...-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:

julia> using PyCall
julia> pyinitialize() # will go away once #2378 is fixed

Can't the pyinitialize() call go at the end of the PyCall module load? I know that might not be the best form, but it seems like it would be reasonable to do the pyinitilized there and it can also setup an atexit handler to call pydeinitialize (or whatever it's called, I forget).

Update: I found a workaround for #2378, so pyinitialize() is no longer needed (unless you want to override the default python version).
Diego Javier Zea | 24 Feb 03:10 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia

Looks like a fantastic module. I'm without time right now, but I'm going to use this with BioPython next week (or the next next) :D



El viernes, 22 de febrero de 2013 14:00:35 UTC-3, Steven G. Johnson escribió:



On Thursday, February 21, 2013 11:01:10 AM UTC-5, Stefan Karpinski wrote:
On Wed, Feb 20, 2013 at 9:08 PM, Steven G. Johnson <ste...-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:

julia> using PyCall
julia> pyinitialize() # will go away once #2378 is fixed

Can't the pyinitialize() call go at the end of the PyCall module load? I know that might not be the best form, but it seems like it would be reasonable to do the pyinitilized there and it can also setup an atexit handler to call pydeinitialize (or whatever it's called, I forget).

Update: I found a workaround for #2378, so pyinitialize() is no longer needed (unless you want to override the default python version).
lgautier | 25 Feb 10:21 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia



On Friday, February 22, 2013 6:00:35 PM UTC+1, Steven G. Johnson wrote:


On Thursday, February 21, 2013 11:01:10 AM UTC-5, Stefan Karpinski wrote:
On Wed, Feb 20, 2013 at 9:08 PM, Steven G. Johnson <ste...-FrUbXkNCsVf2fBVCVOL8/A@public.gmane.org> wrote:

julia> using PyCall
julia> pyinitialize() # will go away once #2378 is fixed

Can't the pyinitialize() call go at the end of the PyCall module load? I know that might not be the best form, but it seems like it would be reasonable to do the pyinitilized there and it can also setup an atexit handler to call pydeinitialize (or whatever it's called, I forget).

Update: I found a workaround for #2378, so pyinitialize() is no longer needed (unless you want to override the default python version).

Does this work by initializing Python with whatever initialization parameters are set if Python is not already whenever the first call to Python functionalities is made ?

That's a good idea. I will use it for the R interface.
Steven G. Johnson | 25 Feb 15:16 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia



On Monday, February 25, 2013 4:21:20 AM UTC-5, lgautier wrote:
Does this work by initializing Python with whatever initialization parameters are set if Python is not already whenever the first call to Python functionalities is made ?
 
Not sure I understand your question, but let me just explain in more detail.

Every (high-level) PyCall function first checks an "initialized" global to see whether Python is initialized (shared library loaded, Python interpreter started, etcetera).  If not, it calls pyinitialize() to initialize Python with the default parameters (via running the "python" executable to query the Python library name).  Optionally, the user can call pyinitialize manually in order to initialize using a different Python version or at a different time.

The problem with #2378 (which is now fixed anyway) had to do with calling the "python" executable from the <at> pyimport macro, which deadlocked with the parser. I worked around this by changing the <at> pyimport macro so that most of the heavy lifting is done from a pywrap function called by the macro.  That way, the initialization is moved from parse-time to run-time, which seemed like a good idea anyway.

--SGJ
R. Michael Weylandt | 25 Feb 16:08 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia

On Mon, Feb 25, 2013 at 2:16 PM, Steven G. Johnson
<stevenj.mit@...> wrote:
>
> The problem with #2378 (which is now fixed anyway) had to do with calling
> the "python" executable from the  <at> pyimport macro, which deadlocked with the
> parser.

I'm not sure I follow this bit (which is probably my fault for not
looking at the code): couldn't you have shelled out to something like
'env python' or 'which python' instead?

Michael

Steven G. Johnson | 25 Feb 17:26 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia



On Monday, February 25, 2013 10:08:25 AM UTC-5, Michael Weylandt wrote:
I'm not sure I follow this bit (which is probably my fault for not
looking at the code): couldn't you have shelled out to something like
'env python' or 'which python' instead?

Bug #2378 meant that shelling out (the run command) deadlocked with the parser.  But anyway, it turned out nicer to do this at runtime rather than parse time.

Also, `which python` is not what I want.  I don't want the path of the python executable, I want the name (and possibly the path) of the libpython library.  I do this by running
    python -c "import distutils.sysconfig; print distutils.sysconfig.get_config_var('LDLIBRARY')"

--SGJ
lgautier | 26 Feb 02:01 2013
Picon

Re: UPDATE: PyCall module to call Python functions from Julia



On Monday, February 25, 2013 3:16:08 PM UTC+1, Steven G. Johnson wrote:


On Monday, February 25, 2013 4:21:20 AM UTC-5, lgautier wrote:
Does this work by initializing Python with whatever initialization parameters are set if Python is not already whenever the first call to Python functionalities is made ?
 
Not sure I understand your question, but let me just explain in more detail.
 

Every (high-level) PyCall function first checks an "initialized" global to see whether Python is initialized (shared library loaded, Python interpreter started, etcetera).  If not, it calls pyinitialize() to initialize Python with the default parameters (via running the "python" executable to query the Python library name).  Optionally, the user can call pyinitialize manually in order to initialize using a different Python version or at a different time.

Ok. The answer to the (long) question is then: "yes".
;-)
 

The problem with #2378 (which is now fixed anyway) had to do with calling the "python" executable from the <at> pyimport macro, which deadlocked with the parser. I worked around this by changing the <at> pyimport macro so that most of the heavy lifting is done from a pywrap function called by the macro.  That way, the initialization is moved from parse-time to run-time, which seemed like a good idea anyway.

Yes. The (conditional) Initialization is a definitely a run-time event.


Thanks for the answer.

L.  

--SGJ

Gmane