On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
> You need at least a slightly different Python API to get anywhere, so
> numexpr/Theano is the right place to work on an implementation of this
> idea. Of course it would be nice if numexpr/Theano offered something as
> convenient as
>
> with lazy:
>     arr = A + B + C # with all of these NumPy arrays

> # compute upon exiting…

Hmm, that would be cute indeed.  Do you have an idea on how the code in the with context could be passed to the Python AST compiler (à la numexpr.evaluate("A + B + C"))?

On Mon, Feb 20, 2012 at 12:28 PM, Francesc Alted <francesc <at> continuum.io> wrote:

On Feb 20, 2012, at 6:18 PM, Dag Sverre Seljebotn wrote:
> You need at least a slightly different Python API to get anywhere, so
> numexpr/Theano is the right place to work on an implementation of this
> idea. Of course it would be nice if numexpr/Theano offered something as
> convenient as
>
> with lazy:
>     arr = A + B + C # with all of these NumPy arrays

> # compute upon exiting…

Hmm, that would be cute indeed.  Do you have an idea on how the code in the with context could be passed to the Python AST compiler (à la numexpr.evaluate("A + B + C"))?

The biggest problem with the numexpr approach (e.g. evaluate("A + B + C")) whether the programmer has to type the quotes or not, is that the sub-program has to be completely expressed in the sub-language.

If I write 

>>> def f(x): return x[:3]

>>> numexpr.evaluate("A + B + f(C)")

I would like that to be fast, but it's not obvious at all how that would work. We would be asking numexpr to introspect arbitrary callable python objects, and recompile arbitrary Python code, effectively setting up the expectation in the user's mind that numexpr is re-implementing an entire compiler. That can be fast obviously, but it seems to me to represent significant departure from numpy's focus, which I always thought was the data-container rather than the expression evaluation (though maybe this firestorm of discussion is aimed at changing this?)

Theano went with another option which was to replace the A, B, and C variables with objects that have a modified __add__. Theano's back-end can be slow at times and the codebase can feel like a heavy dependency, but my feeling is still that this is a great approach to getting really fast implementations of compound expressions.

The context syntax you suggest using is a little ambiguous in that the indented block of a with statement block includes *statements* whereas what you mean to build in the indented block is a *single expression* graph.  You could maybe get the right effect with something like

A, B, C = np.random.rand(3, 5)

expr = np.compound_expression()

with np.expression_builder(expr) as foo:

   arr = A + B + C

   brr = A + B * C

   foo.return((arr, brr))

# compute arr and brr as quickly as possible

a, b = expr.run()

# modify one of the arrays that the expression was compiled to use

A[:] += 1

# re-run the compiled expression on the new value

a, b = expr.run()

- JB