santa | 3 Aug 12:24 2011
Picon

Problem with ValueError: <classname> has the wrong size, try recompiling

Hi,
I've been running against a wall trying to solve the following
problem: in a flat package without subpackages I have a class defined
in cython with a .pyx and .pxd file, another .pyx file using that
class and a regular python file that imports the other pyx file. This
setup constantly throws a ValueError at runtime. I've created a
minimized testcase to show the issue:
http://www.senarclens.eu/~gerald/tc/c_test.tar.gz shows the issue
(needs to be extracted to a path that is included in PYTHONPATH)
http://www.senarclens.eu/~gerald/tc/c_test_w.tar.gz works properly
The only difference between the two is that the second does not use
package imports, thus a simple diff yields
$ diff c_test/ c_test_w/
Only in c_test/: __init__.py
diff c_test//problemreader.pyx c_test_w//problemreader.pyx
1,2c1,2
< from c_test.node cimport Node
< from c_test.node import Node
---
> from node cimport Node
> from node import Node
diff c_test//test.py c_test_w//test.py
2c2
< from c_test import problemreader
---
> import problemreader

Which options do I have to solve/ circumvent the problem? Why does it
even occur and is it just me who believes that the ValueError is
highly misleading? Any help is greatly appreciated,
(Continue reading)

Robert Bradshaw | 3 Aug 19:28 2011

Re: Problem with ValueError: <classname> has the wrong size, try recompiling

On Wed, Aug 3, 2011 at 3:24 AM, santa <gerald.senarclens <at> gmail.com> wrote:
> Hi,
> I've been running against a wall trying to solve the following
> problem: in a flat package without subpackages I have a class defined
> in cython with a .pyx and .pxd file, another .pyx file using that
> class and a regular python file that imports the other pyx file. This
> setup constantly throws a ValueError at runtime. I've created a
> minimized testcase to show the issue:
> http://www.senarclens.eu/~gerald/tc/c_test.tar.gz shows the issue
> (needs to be extracted to a path that is included in PYTHONPATH)
> http://www.senarclens.eu/~gerald/tc/c_test_w.tar.gz works properly
> The only difference between the two is that the second does not use
> package imports, thus a simple diff yields
> $ diff c_test/ c_test_w/
> Only in c_test/: __init__.py
> diff c_test//problemreader.pyx c_test_w//problemreader.pyx
> 1,2c1,2
> < from c_test.node cimport Node
> < from c_test.node import Node
> ---
>> from node cimport Node
>> from node import Node
> diff c_test//test.py c_test_w//test.py
> 2c2
> < from c_test import problemreader
> ---
>> import problemreader
>
> Which options do I have to solve/ circumvent the problem? Why does it
> even occur and is it just me who believes that the ValueError is
(Continue reading)

santa | 4 Aug 10:31 2011
Picon

Re: Problem with ValueError: <classname> has the wrong size, try recompiling

Hi,

thanks for your answer! I just upgraded to the current version of cython to make sure I have a working cythonize (w/ sudo pip install cython --upgrade). 
cython --version yields Cython version 0.14.1

Recompile all your .pyx files again (you can force this by removing

all your .c files)

Doesn't work when removing all .c, .o and .so files and then directly calling setup.py afterwards; same issue. Manually cythonizing the pyx files first (cython *.pyx) and then calling the old setup works like a charm. :)

or use a setup like
http://wiki.cython.org/enhancements/distutils_preprocessing that
understands interdependencies.

Using the exact setup.py script proposed in your link (with cython 0.14.1), I get the following:
<terminal>
~/bitbucket/c_test >python setup.py build_ext --inplace
Traceback (most recent call last):
  File "setup.py", line 6, in <module>
    ext_modules = cythonize(["*.pyx"]),
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 456, in cythonize
    aliases=aliases)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 426, in create_extension_list
    pkg = deps.package(file)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 21, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 287, in package
    return self.package(dir) + (os.path.basename(dir),)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 21, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 287, in package
    return self.package(dir) + (os.path.basename(dir),)
... # removed a few hundred repeating lines
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 287, in package
    return self.package(dir) + (os.path.basename(dir),)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 21, in wrapper
    res = cache[args] = f(self, *args)
  File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 286, in package
    if os.path.exists(os.path.join(dir, '__init__.py')):
  File "/usr/lib/python2.7/posixpath.py", line 68, in join
    elif path == '' or path.endswith('/'):
RuntimeError: maximum recursion depth exceeded in cmp
</terminal>
 

The long answer:

You have node.pyx, node.pxd, problemreader.pyx. When compiling
problemreader.pyx it looks at node.pxd and uses information about
object struct layouts, etc. to generate the .c file.

ok

If you then
modify node.pxd without re-compiling problemreader.pyx the C-level
layout of your Node class can change rendering it incompatible with
problemreader.c. Cython detects this at runtime (when it can) and
trows this value error.

 I think this would be much easier to understand if I actually modified anything... but the problem occurs on executing the setup script on a version without any c etc. files. Everything is generated, then nothing is changed and the error still occurs. Would be nice to be able to understand this, however I'm happy that I have now one way of creating everything that doesn't fail (manually cythonizing the .pyx files and then calling the old setup). I guess whatever goes wrong with the new setup script deserves a look as well - do you want me to file a bug? [1]

/gerald

[1] I couldn't find any related bug on http://trac.cython.org.
Robert Bradshaw | 4 Aug 10:37 2011

Re: Problem with ValueError: <classname> has the wrong size, try recompiling

On Thu, Aug 4, 2011 at 1:31 AM, santa <gerald.senarclens <at> gmail.com> wrote:
> Hi,
> thanks for your answer! I just upgraded to the current version of cython to
> make sure I have a working cythonize (w/ sudo pip install cython
> --upgrade).
> cython --version yields Cython version 0.14.1
>>
>> Recompile all your .pyx files again (you can force this by removing
>>
>> all your .c files)
>
> Doesn't work when removing all .c, .o and .so files and then directly
> calling setup.py afterwards; same issue. Manually cythonizing the pyx files
> first (cython *.pyx) and then calling the old setup works like a charm. :)

Removing the .c files should cause it to create new ones as if you did
it manually. Are you sure you didn't have some old object files cached
somewhere? Are you invoking the same version of Cython from setup.py
as you are from the command line?

>> or use a setup like
>> http://wiki.cython.org/enhancements/distutils_preprocessing that
>> understands interdependencies.
>
> Using the exact setup.py script proposed in your link (with cython 0.14.1),
> I get the following:
> <terminal>
> ~/bitbucket/c_test >python setup.py build_ext --inplace
> Traceback (most recent call last):
>   File "setup.py", line 6, in <module>
>     ext_modules = cythonize(["*.pyx"]),
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 456, in cythonize
>     aliases=aliases)
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 426, in create_extension_list
>     pkg = deps.package(file)
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 21, in wrapper
>     res = cache[args] = f(self, *args)
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 287, in package
>     return self.package(dir) + (os.path.basename(dir),)
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 21, in wrapper
>     res = cache[args] = f(self, *args)
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 287, in package
>     return self.package(dir) + (os.path.basename(dir),)
> ... # removed a few hundred repeating lines
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 287, in package
>     return self.package(dir) + (os.path.basename(dir),)
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 21, in wrapper
>     res = cache[args] = f(self, *args)
>   File
> "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line
> 286, in package
>     if os.path.exists(os.path.join(dir, '__init__.py')):
>   File "/usr/lib/python2.7/posixpath.py", line 68, in join
>     elif path == '' or path.endswith('/'):
> RuntimeError: maximum recursion depth exceeded in cmp
> </terminal>

Hmm... I'll take a look at that. Could you re-post your code so I can
try to reproduce it?

>>
>> The long answer:
>>
>> You have node.pyx, node.pxd, problemreader.pyx. When compiling
>> problemreader.pyx it looks at node.pxd and uses information about
>> object struct layouts, etc. to generate the .c file.
>
> ok
>>
>> If you then
>> modify node.pxd without re-compiling problemreader.pyx the C-level
>> layout of your Node class can change rendering it incompatible with
>> problemreader.c. Cython detects this at runtime (when it can) and
>> trows this value error.
>
>  I think this would be much easier to understand if I actually modified
> anything... but the problem occurs on executing the setup script on a
> version without any c etc. files. Everything is generated, then nothing is
> changed and the error still occurs. Would be nice to be able to understand
> this, however I'm happy that I have now one way of creating everything that
> doesn't fail (manually cythonizing the .pyx files and then calling the old
> setup). I guess whatever goes wrong with the new setup script deserves a
> look as well - do you want me to file a bug? [1]
> /gerald
> [1] I couldn't find any related bug on http://trac.cython.org.

Yes, please do.

- Robert

santa | 4 Aug 11:34 2011
Picon

Re: Problem with ValueError: <classname> has the wrong size, try recompiling

On Thursday, August 4, 2011 10:37:56 AM UTC+2, robertwb wrote:

 
> Doesn't work when removing all .c, .o and .so files and then directly
> calling setup.py afterwards; same issue. Manually cythonizing the pyx files
> first (cython *.pyx) and then calling the old setup works like a charm. :)

Removing the .c files should cause it to create new ones as if you did
it manually. Are you sure you didn't have some old object files cached
somewhere? Are you invoking the same version of Cython from setup.py
as you are from the command line?

I'm fairly sure that there's nothing but the .p* files left when running the old style setup. The curiosity is that the issue only occurs w/ package imports.
Concerning the version: is there any easy way to find out (there's no [cC]ython.VERSION or the like)? I had removed the outdated cython packages before pip installing the current one, but I'll gladly verify.

Hmm... I'll take a look at that. Could you re-post your code so I can

try to reproduce it?

Of course; the complete testcase is available at:http://www.senarclens.eu/~gerald/tc/c_test.tar.gz and needs to be extracted to a directory on the class path to "work".
 

> ... do you want me to file a bug? [1] 

Yes, please do.

 As soon as I get access to trac.cython.org (pm is on the way). If you prefer, I could post it to cython-devel instead.

/gerald


Gmane