headers
Stefan Behnel | 3 Aug 15:41
Picon
Favicon

Re: [Pyrex] New Distutils implementation for Pyrex.

Hi,

Billy G. Allie wrote:
> I've implesmentd a new version of
> buld_ext.py and created a version of core.py and extension.py.

first thing I noticed when running a patched Cython: setuptools break. They
already patch into distutils, so the patching in your code doesn't work. I
would prefer doing without changes to core.py and extensions.py. AFAICT, this
removes the support for configuring import directories at the Extension level.

Alternatively, we could also import things from setuptools first and only fall
back to distutils. However, this would bug people who have setuptools
installed and want to build with distutils (for whatever reason, don't know
how common that is).

I'll take a deeper look into this.

Stefan
Permalink | Reply |
headers
Billy G. Allie | 4 Aug 05:50
Picon

Re: [Pyrex] New Distutils implementation for Pyrex.

Stefan Behnel wrote:
Hi, Billy G. Allie wrote:
I've implesmentd a new version of buld_ext.py and created a version of core.py and extension.py.
first thing I noticed when running a patched Cython: setuptools break. They already patch into distutils, so the patching in your code doesn't work. I would prefer doing without changes to core.py and extensions.py. AFAICT, this removes the support for configuring import directories at the Extension level. Alternatively, we could also import things from setuptools first and only fall back to distutils. However, this would bug people who have setuptools installed and want to build with distutils (for whatever reason, don't know how common that is). I'll take a deeper look into this. Stefa
By the way, didn't the package name change from Pyrex to Cython break setuptools?  (It would in the version of setuptools I installed). 
_______________________________________________
Cython-dev mailing list
Cython-dev@...
https://lists.berlios.de/mailman/listinfo/cython-dev
Permalink | Reply |
headers
Billy G. Allie | 4 Aug 05:41
Picon

Re: [Pyrex] New Distutils implementation for Pyrex.

Stefan Behnel wrote:
Hi, Billy G. Allie wrote:
I've implesmentd a new version of buld_ext.py and created a version of core.py and extension.py.
first thing I noticed when running a patched Cython: setuptools break. They already patch into distutils, so the patching in your code doesn't work. I would prefer doing without changes to core.py and extensions.py. AFAICT, this removes the support for configuring import directories at the Extension level. Alternatively, we could also import things from setuptools first and only fall back to distutils. However, this would bug people who have setuptools installed and want to build with distutils (for whatever reason, don't know how common that is). I'll take a deeper look into this. Stefan
Stefan,

I've looked at the setuptools source.  Since setuptools is importing Pyrex.Distutils to perform work, setuptools will have to be patched to work with the new code.  From my initial walkthrough of the code (by no meas through), it appeas that the only change would be to change occurances of 'distutils.core' to 'Pyrex.Distutils.core', wrapped in a try ... except statement.  For example:
try:
    from Pyrex.Distutils.core import Extension
except ImportError:
    from distutils.core import Extension
-- or --
try:
    import Pyrex.Distutils.core
except ImportError:
    import distutils.core
The files that would have to change are:
command/upload.py
command/easy_install.py
dist.py
extension.py
__init__.py
package_index.py
tests/__init__.py
Extension is always importted from core, so with the change stated above it will always get the correct Extension class.
build_ext is already imported from Pyrex.Distutils.
_______________________________________________
Cython-dev mailing list
Cython-dev@...
https://lists.berlios.de/mailman/listinfo/cython-dev
Permalink | Reply |
headers
Phillip J. Eby | 6 Aug 00:37
Gravatar

Re: New Distutils implementation for Pyrex.

At 11:41 PM 8/3/2007 -0400, Billy G. Allie wrote:
>The files that would have to change are:
>command/upload.py
>command/easy_install.py
>dist.py
>extension.py
>__init__.py
>package_index.py
>tests/__init__.py

No need to change all this; setuptools already works correctly with 
Pyrex.Distutils.  If you want it to work with a different extension 
type all you need is a different build_ext command, and for that you 
don't need to patch anything.  Just have your setup script specify 
your build_ext class using setup(... 
cmdclass=dict(build_ext=my_buildext_class), ...).

That's really all you need; no patching is required.

Indeed, the only reason that setuptools does anything with Pyrex at 
all is to support *not* compiling .pyx files when Pyrex is *not* 
installed.  If there are .pyx sources for an Extension and 
Pyrex.Distutils can't be imported, setuptools changes the source file 
extensions to '.c' so that a distributed pre-built .c file can be 
compiled without needing Pyrex.
Permalink | Reply |
headers
Billy G. Allie | 6 Aug 03:44
Picon

Re: [Pyrex] New Distutils implementation for Pyrex.

Phillip J. Eby wrote:
> At 11:41 PM 8/3/2007 -0400, Billy G. Allie wrote:
>> The files that would have to change are:
>> command/upload.py
>> command/easy_install.py
>> dist.py
>> extension.py
>> __init__.py
>> package_index.py
>> tests/__init__.py
>
> No need to change all this; setuptools already works correctly with 
> Pyrex.Distutils. If you want it to work with a different extension 
> type all you need is a different build_ext command, and for that you 
> don't need to patch anything. Just have your setup script specify your 
> build_ext class using setup(... 
> cmdclass=dict(build_ext=my_buildext_class), ...).
>
> That's really all you need; no patching is required.
>
> Indeed, the only reason that setuptools does anything with Pyrex at 
> all is to support *not* compiling .pyx files when Pyrex is *not* 
> installed. If there are .pyx sources for an Extension and 
> Pyrex.Distutils can't be imported, setuptools changes the source file 
> extensions to '.c' so that a distributed pre-built .c file can be 
> compiled without needing Pyrex.
>
If you want to give new options to extension you need to subclass 
entension.py. If you want to have those new options available as command 
line options (to setup.py), you need to wrap core.py (it can't be 
sub-classed). My new Distutils for Pyrex does both. In order for 
setuptools to compile pyrex extension (.pyx), it needs to use 
Pyrex.Distutils.core instead of distutils.core. If you don't, just 
loading build_ext will fail because of undefined extension variables.
Permalink | Reply |
headers
Phillip J. Eby | 6 Aug 04:11
Gravatar

Re: New Distutils implementation for Pyrex.

At 09:44 PM 8/5/2007 -0400, Billy G. Allie wrote:
>If you want to give new options to extension you need to subclass
>entension.py.

And then just use those classes in your setup.py.  You don't need to 
patch setuptools for that.

>  If you want to have those new options available as command
>line options (to setup.py), you need to wrap core.py (it can't be
>sub-classed).

Why do you need options to setup.py instead of just options to the 
build_ext command?

>  My new Distutils for Pyrex does both. In order for
>setuptools to compile pyrex extension (.pyx), it needs to use
>Pyrex.Distutils.core instead of distutils.core. If you don't, just
>loading build_ext will fail because of undefined extension variables.

Per the above, none of this should be necessary.
Permalink | Reply |
headers
Stefan Behnel | 3 Aug 08:51
Picon
Favicon

Re: [Pyrex] New Distutils implementation for Pyrex.

Billy G. Allie wrote:
> Stefan Behnel wrote:
>> Billy G. Allie wrote:
>>
>>> I am re-implementing my PostgreSQL interface (pyPgSQL) using Pyrex to
>>> implement an extension for the libpq C-API.  When using the supplied
>>> build_ext, I notices that I had to have the .pxd files in the same
>>> directory as the source .pyx file.  Also, the generated c source file
>>> was placed in the same directory as the Pyrex source.  I also could not
>>> specify Pyrex options to create a listing file, etc.  I wanted to have
>>> the .pxd files in an separate include directory and I also did not want
>>> to clean up the generated files from the Pyrex source directory.  To
>>> correct these (minor) annoyances, I've implemented a new version of
>>> buld_ext.py and created a version of core.py and extension.py.
>>
>> is this an independent implementation or is it based on the old one? If the
>> latter, could you provide a patch against Cython and submit it at the
>> Cython
>> bug tracker? If it's new, could you still provide a single-file patch that
>> includes the new files?
>>
>> https://bugs.launchpad.net/cython
>>
>> http://www.cython.org
>>
>>
> I am attaching a diff generated by the 'diff -cbN <originale Distutils
> directory> <new Distutils directory>' command.
> I also changed all references to Pyrex to Cython and pyrex to cython in the
> files.
>
> I used the patched Distutils modules to successfully build my libpq
> extension (after I fixed some syntax errors that Pyrex missed :-(  )

Thanks a lot for the patches! I'm resending them to the Cython list for now.

You mentioned that you used distutils in Python 2.4 as a base. Python 2.3
compatibility is currently a requirement for Cython. Would you say there are
any 2.4 specifics in there?

Thanks again,
Stefan

diff -cbN /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/build_ext.py Distutils/build_ext.py
*** /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/build_ext.py	2007-07-28 16:41:03.000000000 -0400
--- Distutils/build_ext.py	2007-08-03 00:36:10.000000000 -0400
***************
*** 1,63 ****
! # Subclasses disutils.command.build_ext,
! # replacing it with a Cython version that compiles pyx->c
! # before calling the original build_ext command.
! # July 2002, Graham Fawcett
! # Modified by Darrell Gallion <dgallion1@...>
! # to allow inclusion of .c files along with .pyx files.
! # Pyrex is (c) Greg Ewing.
! 
! import distutils.command.build_ext
! #import Cython.Compiler.Main
! from Cython.Compiler.Main import CompilationOptions, default_options, compile
! from Cython.Compiler.Errors import PyrexError
! from distutils.dep_util import newer
! import os
! import sys

! def replace_suffix(path, new_suffix):
!     return os.path.splitext(path)[0] + new_suffix

! class build_ext (distutils.command.build_ext.build_ext):

!     description = "compile Cython scripts, then build C/C++ extensions (compile/link to build directory)"

      def finalize_options (self):
!         distutils.command.build_ext.build_ext.finalize_options(self)

!         # The following hack should no longer be needed.
!         if 0:
!             # compiling with mingw32 gets an "initializer not a constant" error
!             # doesn't appear to happen with MSVC!
!             # so if we are compiling with mingw32,
!             # switch to C++ mode, to avoid the problem
!             if self.compiler == 'mingw32':
!                 self.swig_cpp = 1
! 
!     def swig_sources (self, sources, extension = None):
!         if not self.extensions:
!             return
! 
!         # collect the names of the source (.pyx) files
!         pyx_sources = []
!         pyx_sources = [source for source in sources if source.endswith('.pyx')]
!         other_sources = [source for source in sources if not source.endswith('.pyx')]
! 
!         #suffix = self.swig_cpp and '.cpp' or '.c'
!         suffix = '.c'
!         for pyx in pyx_sources:
!             # should I raise an exception if it doesn't exist?
!             if os.path.exists(pyx):
!                 source = pyx
!                 target = replace_suffix(source, suffix)
!                 if newer(source, target) or self.force:
!                     self.cython_compile(source)
! 
!         return [replace_suffix(src, suffix) for src in pyx_sources] + other_sources
! 
!     def cython_compile(self, source):
!         options = CompilationOptions(default_options,
!             include_path = self.include_dirs)
!         result = compile(source, options)
!         if result.num_errors <> 0:
!             sys.exit(1)

--- 1,159 ----
! """Cython.Distutils.build_ext

! Implements a version of the Distutils 'build_ext' command, for
! building Cython extension modules."""

! # This module should be kept compatible with Python 2.1.

! __revision__ = "$Id:$"
! 
! import sys, os, string, re
! from types import *
! from Cython.Distutils.core import Command
! from distutils.errors import *
! from distutils.sysconfig import customize_compiler, get_python_version
! from distutils.dep_util import newer_group
! from Cython.Distutils.extension import Extension
! from distutils import log
! from distutils.dir_util import mkpath
! try:
!     from Cython.Compiler.Main \
!         import CompilationOptions, \
!                default_options as cython_default_options, \
!                compile as cython_compile
!     from Cython.Compiler.Errors import PyrexError
! except ImportError:
!     PyrexError = None
! 
! from distutils.command import build_ext as _build_ext
! 
! extension_name_re = _build_ext.extension_name_re
! 
! show_compilers = _build_ext.show_compilers
! 
! class build_ext(_build_ext.build_ext):
! 
!     description = "build C/C++ and Cython extensions (compile/link to build directory)"
! 
!     sep_by = _build_ext.build_ext.sep_by
!     user_options = _build_ext.build_ext.user_options
!     boolean_options = _build_ext.build_ext.boolean_options
!     help_options = _build_ext.build_ext.help_options
! 
!     # Add the cython specific data.
!     user_options.extend([
!         ('cython-cplus', None,
!          "generate C++ source files"),
!         ('cython-create-listing', None,
!          "write errors to a listing file"),
!         ('cython-include-dirs=', None,
!          "path to the Cython include files" + sep_by),
!         ])
! 
!     boolean_options.extend(['cython-cplus', 'cython-create-listing'])
! 
!     def initialize_options(self):
!         _build_ext.build_ext.initialize_options(self)
!         self.cython_cplus = 0
!         self.cython_create_listing = 0
!         self.cython_include_dirs = None

      def finalize_options (self):
!         _build_ext.build_ext.finalize_options(self)
! 
!         if self.cython_include_dirs is None:
!             self.cython_include_dirs = []
!         elif type(self.cython_include_dirs) is StringType:
!             self.cython_include_dirs = \
!                 string.split(self.cython_include_dirs, os.pathsep)
!     # finalize_options ()
! 
! 
!     def build_extensions(self):
!         # First, sanity-check the 'extensions' list
!         self.check_extensions_list(self.extensions)
! 
!         for ext in self.extensions:
!             ext.sources = self.cython_sources(ext.sources, ext)
!             self.build_extension(ext)
! 
!     def cython_sources (self, sources, extension):
! 
!         """
!         Walk the list of source files in 'sources', looking for Cython
!         source (.pyx) files.  Run Cython on all that are found, and return
!         a modified 'sources' list with Cython source files replaced by the
!         generated C (or C++) files.
!         """
! 
!         if PyrexError == None:
!             raise DistutilsPlatformError, \
!                   ("Cython does not appear to be installed "
!                    "on platform '%s'") % os.name
! 
!         new_sources = []
!         cython_sources = []
!         cython_targets = {}
! 
!         create_listing = self.cython_create_listing or \
!                             extension.cython_create_listing
!         cplus = self.cython_cplus or extension.cython_cplus
! 
!         #Set up the include_path for the Pyres compiler:
!         #   1.  Start with the command line option.
!         #   2.  Add in any (unique) paths from the extension
!         #       cython_include_dirs.
!         #   3.  Add in any (unique) paths from the extension include_dirs
!         #       if and only if there there were no cython_include_dirs defined..
!         includes = self.cython_include_dirs
!         for i in extension.cython_include_dirs:
!             if not i in includes:
!                 includes.append(i)
!         if len(includes) < 1:
!             for i in extension.include_dirs:
!                 if not i in includes:
!                     includes.append(i)
! 
!         # Set the target_ext to '.c'.  Cython will change this to '.cpp' if
!         # needed.
!         if cplus:
!             target_ext = '.cpp'
!         else:
!             target_ext = '.c'
! 
!         # Drop the generated C files into the temp dir, unless the inplace
!         # option is present.  In that case, drop them into the source tree.
! 
!         if self.inplace:
!             target_dir = ""
!         else:
!             target_dir = os.path.join(self.build_temp, "cython")
! 
!         for source in sources:
!             (base, ext) = os.path.splitext(source)
!             if ext == ".pyx":             # Cython source file
!                 new_sources.append(os.path.join(target_dir, base + target_ext))
!                 cython_sources.append(source)
!                 cython_targets[source] = new_sources[-1]
!             else:
!                 new_sources.append(source)
! 
!         if not cython_sources:
!             return new_sources
! 
!         for source in cython_sources:
!             target = cython_targets[source]
!             log.info("cythoning %s to %s", source, target)
!             self.mkpath(os.path.dirname(target))
!             options = CompilationOptions(cython_default_options, 
!                                          use_listing_file = create_listing,
!                                          include_path = includes,
!                                          output_file = target,
!                                          cplus = cplus)
!             result = cython_compile(source, options=options)
! 
!         return new_sources

!     # cython_sources ()

+ # class build_ext
diff -cbN /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/core.py Distutils/core.py
*** /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/core.py	1969-12-31 19:00:00.000000000 -0500
--- Distutils/core.py	2007-08-03 00:36:10.000000000 -0400
***************
*** 0 ****
--- 1,34 ----
+ """Cython.Distutils.core
+ 
+ This module wraps the 'distutils.core' module and extends it to support
+ Cython extension modules.
+ 
+ This is the only module that needs to be imported to use the Distutils;
+ provides the 'setup' function (which is to be called from the setup script).
+ Also indirectly provides the Distribution and Command classes, although
+ they are really defined in distutils.dist and distutils.cmd.
+ """
+ 
+ # This module should be kept compatible with Python 2.1.
+ 
+ __revision__ = "$Id:$"
+ 
+ import sys, os
+ from types import *
+ 
+ from distutils.debug import DEBUG
+ from distutils.errors import *
+ from distutils.util import grok_environment_error
+ 
+ # Mainly import these so setup scripts can "from distutils.core import" them.
+ from distutils.dist import Distribution
+ from distutils.cmd import Command
+ 
+ # Make sure we get Cython's Extension class.
+ from Cython.Distutils.extension import Extension
+ 
+ from distutils.core import *
+ 
+ extension_keywords = extension_keywords + \
+                      ('cython_include_dirs', 'cython_create_listing',
+                       'cython_cplus')
diff -cbN /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/extension.py Distutils/extension.py
*** /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/extension.py	1969-12-31 19:00:00.000000000 -0500
--- Distutils/extension.py	2007-08-03 00:36:10.000000000 -0400
***************
*** 0 ****
--- 1,73 ----
+ """Cython.Distutils.extension
+ 
+ Provides a modified Extension class, that understands hou to describe
+ Cython extension modules in setup scripts."""
+ 
+ __revision__ = "$Id:$"
+ 
+ import os, string, sys
+ from types import *
+ import distutils.extension as _Extension
+ 
+ try:
+     import warnings
+ except ImportError:
+     warnings = None
+ 
+ class Extension(_Extension.Extension):
+     _Extension.Extension.__doc__ + \
+     """cython_include_dirs : [string]
+         list of directories to search for Cython header files (.pxd) (in
+         Unix form for portability)
+     cython_create_listing_file : boolean
+         write cython error messages to a listing (.lis) file.
+     cython_cplus : boolean
+         use the C++ compiler for compiling and linking.
+     """
+ 
+     # When adding arguments to this constructor, be sure to update
+     # setup_keywords in core.py.
+     def __init__ (self, name, sources,
+                   include_dirs=None,
+                   define_macros=None,
+                   undef_macros=None,
+                   library_dirs=None,
+                   libraries=None,
+                   runtime_library_dirs=None,
+                   extra_objects=None,
+                   extra_compile_args=None,
+                   extra_link_args=None,
+                   export_symbols=None,
+                   swig_opts = None,
+                   depends=None,
+                   language=None,
+                   cython_include_dirs=None,
+                   cython_create_listing=0,
+                   cython_cplus=0,
+                   **kw                      # To catch unknown keywords
+                  ):
+ 
+         _Extension.Extension.__init__(self, name, sources,
+                        include_dirs,
+                        define_macros,
+                        undef_macros,
+                        library_dirs,
+                        libraries,
+                        runtime_library_dirs,
+                        extra_objects,
+                        extra_compile_args,
+                        extra_link_args,
+                        export_symbols,
+                        swig_opts ,
+                        depends,
+                        language,
+                        **kw
+                 )
+ 
+         self.cython_include_dirs = cython_include_dirs or []
+         self.cython_create_listing = cython_create_listing
+         self.cython_cplus = cython_cplus
+ 
+ # class Extension
+ 
+ read_setup_file = _Extension.read_setup_file
diff -cbN /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/__init__.py Distutils/__init__.py
*** /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/__init__.py	2007-07-28 15:29:52.000000000 -0400
--- Distutils/__init__.py	2007-08-03 00:36:10.000000000 -0400
***************
*** 1,22 ****
  # July 2002, Graham Fawcett
- 
  #
- 
  # this hack was inspired by the way Thomas Heller got py2exe
- 
  # to appear as a distutil command
- 
  #
- 
  # we replace distutils.command.build_ext with our own version
- 
  # and keep the old one under the module name _build_ext,
- 
  # so that *our* build_ext can make use of it.

  

! from build_ext import build_ext
! 

  
--- 1,16 ----
  # July 2002, Graham Fawcett
  #
  # this hack was inspired by the way Thomas Heller got py2exe
  # to appear as a distutil command
  #
  # we replace distutils.command.build_ext with our own version
  # and keep the old one under the module name _build_ext,
  # so that *our* build_ext can make use of it.

+ __revision__ = "$Id:$"

+ __all__ = [ 'core.py', 'build_ext.py', 'extension.py' ]

! import core, build_ext, extension


_______________________________________________
Cython-dev mailing list
Cython-dev@...
https://lists.berlios.de/mailman/listinfo/cython-dev
Permalink | Reply |
headers
Billy G. Allie | 3 Aug 22:02
Picon

Re: New Distutils implementation for Pyrex.

Stefan Behnel wrote:

Thanks a lot for the patches! I'm resending them to the Cython list for now. You mentioned that you used distutils in Python 2.4 as a base. Python 2.3 compatibility is currently a requirement for Cython. Would you say there are any 2.4 specifics in there? Thanks again, Stefan
Not that I know of.  I just looked at the 2.3 distutils code and didn't see anything that would cause problems with my distutils extension.
_______________________________________________
Pyrex mailing list
Pyrex <at> lists.copyleft.no
http://lists.copyleft.no/mailman/listinfo/pyrex
Permalink | Reply |
headers
Billy G. Allie | 3 Aug 22:31
Picon

Re: [Pyrex] New Distutils implementation for Pyrex.

Stefan Behnel wrote:
Hi, Billy G. Allie wrote:
I've implesmentd a new version of buld_ext.py and created a version of core.py and extension.py.
first thing I noticed when running a patched Cython: setuptools break. They already patch into distutils, so the patching in your code doesn't work. I would prefer doing without changes to core.py and extensions.py. AFAICT, this removes the support for configuring import directories at the Extension level. Alternatively, we could also import things from setuptools first and only fall back to distutils. However, this would bug people who have setuptools installed and want to build with distutils (for whatever reason, don't know how common that is). I'll take a deeper look into this. Stefan
Hmmm.  I've never used setuptools, so it wasn't a problem with me :-)
A possible solution would be to try to import setuptools version of core and or extension if they exist, otherwise import the distutils version.   I'll get the source for setuptools and see what they are doing.  (This may take me a couple of days to investigate).
_______________________________________________
Cython-dev mailing list
Cython-dev@...
https://lists.berlios.de/mailman/listinfo/cython-dev
Permalink | Reply |

Gmane