Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef foo(size):
  cdef double a[size]    # This is like the dynamic arrays in C99
  # do stuff with a, but don't worry about deallocating the memory!!!

Underneath the hood cython would do the following:

1) It would use malloc to allocate the actual memory and cast it to
the correct type.

(Continue reading)

Michael.Abshoff | 11 Apr 05:25 2008

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Brian Granger wrote:
> Hi all,

Hi,

> After todays discussion about the best way to allocate dynamic memory
> in a cython method or function, Lisandro Dalcin and I discussed other
> ways of making a cleaner interface for this that doesn't have memory
> leaks.  The purpose of this email is to summarize Lisandro's idea for
> how cython could handle this in a very nice manner.
> 
> Background
> -----------------
> 
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
> 
> Underneath the hood cython would do the following:
> 
(Continue reading)

Stefan Behnel | 11 Apr 10:12 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

Hi,

Brian Granger wrote:
> In a function it is often necessary to allocate dynamic memory.  It is
> desirable to have something that it 1) fast and 2) doesn't lead to
> memory leaks if something goes wrong and 3) is easy to debug.
> 
> Proposal
> -------------
> 
> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
> 
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!

I actually like the syntax as it is rather clear that this is memory local to
the function. It's just like having it allocated on the stack, with the
exception that it depends on run-time state. So the automatic deallocation at
function exit is not unexpected.

Regarding PyMalloc, I think it's a good idea to use it, but you have to take
care in "nogil" functions. They require plain malloc()/free() calls.

Stefan
Dag Sverre Seljebotn | 11 Apr 10:16 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory


> The main idea (Lisandro came up with this) is to build these
> capabilities into cython itself by introducing the following syntax:
>
> cdef foo(size):
>   cdef double a[size]    # This is like the dynamic arrays in C99
>   # do stuff with a, but don't worry about deallocating the memory!!!
>   
It sure looks like it would be convenient; but I'll take the role as an 
advocate against it :-)

1) Having "cdef double a[size]" allocate anything is very against the 
Python syntax -- nothing is ever allocated in Python without a function 
call and an assignment (same as in Java). To somebody who comes from a 
strict Python background and not C, it really looks like only a variable 
type declaration, not object allocation. In Cython, this Python way of 
working is mostly kept, and this would be a step *away* from Python in 
terms of the "feel" of the language.

2) This kind of duplicates the behaviour of the "with" keyword (which is 
not present in Cython today (either), but definitely is a goal, and is 
present in current Python).

However, using the with keyword would be a bit less convenient, given 
function template support and with keyword support in Cython it could 
look something like this:

with carray(double, size) as buf:
  cdef double* a = buf.data
   # do stuff with a
(Continue reading)

Brian Granger | 11 Apr 16:56 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

>  1) Having "cdef double a[size]" allocate anything is very against the
> Python syntax -- nothing is ever allocated in Python without a function call
> and an assignment (same as in Java). To somebody who comes from a strict
> Python background and not C, it really looks like only a variable type
> declaration, not object allocation. In Cython, this Python way of working is
> mostly kept, and this would be a step *away* from Python in terms of the
> "feel" of the language.

As other people have mentioned, Cython is (unapologetically) Python
with C data types, so this syntax will be at home.

>  2) This kind of duplicates the behaviour of the "with" keyword (which is
> not present in Cython today (either), but definitely is a goal, and is
> present in current Python).

I very much like his idea of using with for these things.  But, with
is 2.5 only and I think this is an important enough issue that it
should be a lower level syntax that cython supports.  But, with could
definitely be used for more complex or custom allocators.

Question:  does cython have something like a standard library?  It
seems like such context managers should go there.  Also such a
standard library would provide a place for lots of commonly used
things that shouldn't be in the actual core language.  I think that is
better than the language itself growing all of these (with)
capabilities.  I do think that dynamic arrays should be a part of the
language though.
Dag Sverre Seljebotn | 11 Apr 17:09 2008
Picon
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

> I guess my view is that the buffer interface is not orthogonal to this
> type of thing - rather it will make it even easier to build things
> that could interoperate.  Indeed one possible implementation strategy
> for these dynamic arrays would be to use buffers.

It is completely orthogonal to the "with" statement, but only almost to 
"cdef double a[size]".

"cdef double a[size]" should probably be implemented as one specific 
type, and what has been considered natural in the discussion until now 
(I think) is pure double* with getmem and auto-free. That's what the 
syntax suggests anyway. This means that the "default" Cython array 
object then probably won't support that standard. Which I now think is 
fine anyway. I'm just saying that with a slightly different syntax, it 
could be more natural to support the buffer API with "native" Cython 
arrays. But one can do both.

Or are you suggesting that the array allocation syntax you propose can 
have overridable allocators and create many different types of arrays?

Dag Sverre
Lisandro Dalcin | 11 Apr 18:59 2008
Picon

Re: [Cython] Proposal: idea for automatic management of dynamic memory

After this long discussion, I would like to add some points:

* variable-lenght arrays are already valid constructs in C99.
Furthermore,  'sizeof' returns at runtime the size of the array!!!.

* They are almost identical in syntax and semantics to the dynamic
array extension of GCC, wich is available since long time ago. An
example from the manpages of GCC:

     FILE *
     concat_fopen (char *s1, char *s2, char *mode)
     {
       char str[strlen (s1) + strlen (s2) + 1];
       strcpy (str, s1);
       strcat (str, s2);
       return fopen (str, mode);
     }

* Other no very well know way of  semi-automatic dynamic memory
allocation si the function 'alloca' (please consult your linux info
pages). I'm not sure if this is GCC-only or supported on every
platform. An example from the libc info pages:

          int
          open2 (char *str1, char *str2, int flags, int mode)
          {
            char *name = (char *) alloca (strlen (str1) + strlen (str2) + 1);
            stpcpy (stpcpy (name, str1), str2);
            return open_or_report_error (name, flags, mode);
          }
(Continue reading)

Brian Granger | 11 Apr 05:24 2008
Picon

[Cython] Proposal: idea for automatic management of dynamic memory

Hi all,

After todays discussion about the best way to allocate dynamic memory
in a cython method or function, Lisandro Dalcin and I discussed other
ways of making a cleaner interface for this that doesn't have memory
leaks.  The purpose of this email is to summarize Lisandro's idea for
how cython could handle this in a very nice manner.

Background
-----------------

In a function it is often necessary to allocate dynamic memory.  It is
desirable to have something that it 1) fast and 2) doesn't lead to
memory leaks if something goes wrong and 3) is easy to debug.

Proposal
-------------

The main idea (Lisandro came up with this) is to build these
capabilities into cython itself by introducing the following syntax:

cdef fo