Re: New Distutils implementation for Pyrex.

At 09:44 PM 8/5/2007 -0400, Billy G. Allie wrote:
>If you want to give new options to extension you need to subclass
>entension.py.

And then just use those classes in your setup.py.  You don't need to 
patch setuptools for that.

>  If you want to have those new options available as command
>line options (to setup.py), you need to wrap core.py (it can't be
>sub-classed).

Why do you need options to setup.py instead of just options to the 
build_ext command?

>  My new Distutils for Pyrex does both. In order for
>setuptools to compile pyrex extension (.pyx), it needs to use
>Pyrex.Distutils.core instead of distutils.core. If you don't, just
>loading build_ext will fail because of undefined extension variables.

Per the above, none of this should be necessary.

Re: [Pyrex] New Distutils implementation for Pyrex.

Billy G. Allie wrote:
> Stefan Behnel wrote:
>> Billy G. Allie wrote:
>>
>>> I am re-implementing my PostgreSQL interface (pyPgSQL) using Pyrex to
>>> implement an extension for the libpq C-API.  When using the supplied
>>> build_ext, I notices that I had to have the .pxd files in the same
>>> directory as the source .pyx file.  Also, the generated c source file
>>> was placed in the same directory as the Pyrex source.  I also could not
>>> specify Pyrex options to create a listing file, etc.  I wanted to have
>>> the .pxd files in an separate include directory and I also did not want
>>> to clean up the generated files from the Pyrex source directory.  To
>>> correct these (minor) annoyances, I've implemented a new version of
>>> buld_ext.py and created a version of core.py and extension.py.
>>
>> is this an independent implementation or is it based on the old one? If the
>> latter, could you provide a patch against Cython and submit it at the
>> Cython
>> bug tracker? If it's new, could you still provide a single-file patch that
>> includes the new files?
>>
>> https://bugs.launchpad.net/cython
>>
>> http://www.cython.org
>>
>>
> I am attaching a diff generated by the 'diff -cbN <originale Distutils
> directory> <new Distutils directory>' command.
> I also changed all references to Pyrex to Cython and pyrex to cython in the
> files.
>
> I used the patched Distutils modules to successfully build my libpq
> extension (after I fixed some syntax errors that Pyrex missed   )

Thanks a lot for the patches! I'm resending them to the Cython list for now.

You mentioned that you used distutils in Python 2.4 as a base. Python 2.3
compatibility is currently a requirement for Cython. Would you say there are
any 2.4 specifics in there?

Thanks again,
Stefan

diff -cbN /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/build_ext.py Distutils/build_ext.py
*** /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/build_ext.py	2007-07-28 16:41:03.000000000 -0400
--- Distutils/build_ext.py	2007-08-03 00:36:10.000000000 -0400
***************
*** 1,63 ****
! # Subclasses disutils.command.build_ext,
! # replacing it with a Cython version that compiles pyx->c
! # before calling the original build_ext command.
! # July 2002, Graham Fawcett
! # Modified by Darrell Gallion <dgallion1@...>
! # to allow inclusion of .c files along with .pyx files.
! # Pyrex is (c) Greg Ewing.
! 
! import distutils.command.build_ext
! #import Cython.Compiler.Main
! from Cython.Compiler.Main import CompilationOptions, default_options, compile
! from Cython.Compiler.Errors import PyrexError
! from distutils.dep_util import newer
! import os
! import sys

! def replace_suffix(path, new_suffix):
!     return os.path.splitext(path)[0] + new_suffix

! class build_ext (distutils.command.build_ext.build_ext):

!     description = "compile Cython scripts, then build C/C++ extensions (compile/link to build directory)"

      def finalize_options (self):
!         distutils.command.build_ext.build_ext.finalize_options(self)

!         # The following hack should no longer be needed.
!         if 0:
!             # compiling with mingw32 gets an "initializer not a constant" error
!             # doesn't appear to happen with MSVC!
!             # so if we are compiling with mingw32,
!             # switch to C++ mode, to avoid the problem
!             if self.compiler == 'mingw32':
!                 self.swig_cpp = 1
! 
!     def swig_sources (self, sources, extension = None):
!         if not self.extensions:
!             return
! 
!         # collect the names of the source (.pyx) files
!         pyx_sources = []
!         pyx_sources = [source for source in sources if source.endswith('.pyx')]
!         other_sources = [source for source in sources if not source.endswith('.pyx')]
! 
!         #suffix = self.swig_cpp and '.cpp' or '.c'
!         suffix = '.c'
!         for pyx in pyx_sources:
!             # should I raise an exception if it doesn't exist?
!             if os.path.exists(pyx):
!                 source = pyx
!                 target = replace_suffix(source, suffix)
!                 if newer(source, target) or self.force:
!                     self.cython_compile(source)
! 
!         return [replace_suffix(src, suffix) for src in pyx_sources] + other_sources
! 
!     def cython_compile(self, source):
!         options = CompilationOptions(default_options,
!             include_path = self.include_dirs)
!         result = compile(source, options)
!         if result.num_errors <> 0:
!             sys.exit(1)

--- 1,159 ----
! """Cython.Distutils.build_ext

! Implements a version of the Distutils 'build_ext' command, for
! building Cython extension modules."""

! # This module should be kept compatible with Python 2.1.

! __revision__ = "$Id:$"
! 
! import sys, os, string, re
! from types import *
! from Cython.Distutils.core import Command
! from distutils.errors import *
! from distutils.sysconfig import customize_compiler, get_python_version
! from distutils.dep_util import newer_group
! from Cython.Distutils.extension import Extension
! from distutils import log
! from distutils.dir_util import mkpath
! try:
!     from Cython.Compiler.Main \
!         import CompilationOptions, \
!                default_options as cython_default_options, \
!                compile as cython_compile
!     from Cython.Compiler.Errors import PyrexError
! except ImportError:
!     PyrexError = None
! 
! from distutils.command import build_ext as _build_ext
! 
! extension_name_re = _build_ext.extension_name_re
! 
! show_compilers = _build_ext.show_compilers
! 
! class build_ext(_build_ext.build_ext):
! 
!     description = "build C/C++ and Cython extensions (compile/link to build directory)"
! 
!     sep_by = _build_ext.build_ext.sep_by
!     user_options = _build_ext.build_ext.user_options
!     boolean_options = _build_ext.build_ext.boolean_options
!     help_options = _build_ext.build_ext.help_options
! 
!     # Add the cython specific data.
!     user_options.extend([
!         ('cython-cplus', None,
!          "generate C++ source files"),
!         ('cython-create-listing', None,
!          "write errors to a listing file"),
!         ('cython-include-dirs=', None,
!          "path to the Cython include files" + sep_by),
!         ])
! 
!     boolean_options.extend(['cython-cplus', 'cython-create-listing'])
! 
!     def initialize_options(self):
!         _build_ext.build_ext.initialize_options(self)
!         self.cython_cplus = 0
!         self.cython_create_listing = 0
!         self.cython_include_dirs = None

      def finalize_options (self):
!         _build_ext.build_ext.finalize_options(self)
! 
!         if self.cython_include_dirs is None:
!             self.cython_include_dirs = []
!         elif type(self.cython_include_dirs) is StringType:
!             self.cython_include_dirs = \
!                 string.split(self.cython_include_dirs, os.pathsep)
!     # finalize_options ()
! 
! 
!     def build_extensions(self):
!         # First, sanity-check the 'extensions' list
!         self.check_extensions_list(self.extensions)
! 
!         for ext in self.extensions:
!             ext.sources = self.cython_sources(ext.sources, ext)
!             self.build_extension(ext)
! 
!     def cython_sources (self, sources, extension):
! 
!         """
!         Walk the list of source files in 'sources', looking for Cython
!         source (.pyx) files.  Run Cython on all that are found, and return
!         a modified 'sources' list with Cython source files replaced by the
!         generated C (or C++) files.
!         """
! 
!         if PyrexError == None:
!             raise DistutilsPlatformError, \
!                   ("Cython does not appear to be installed "
!                    "on platform '%s'") % os.name
! 
!         new_sources = []
!         cython_sources = []
!         cython_targets = {}
! 
!         create_listing = self.cython_create_listing or \
!                             extension.cython_create_listing
!         cplus = self.cython_cplus or extension.cython_cplus
! 
!         #Set up the include_path for the Pyres compiler:
!         #   1.  Start with the command line option.
!         #   2.  Add in any (unique) paths from the extension
!         #       cython_include_dirs.
!         #   3.  Add in any (unique) paths from the extension include_dirs
!         #       if and only if there there were no cython_include_dirs defined..
!         includes = self.cython_include_dirs
!         for i in extension.cython_include_dirs:
!             if not i in includes:
!                 includes.append(i)
!         if len(includes) < 1:
!             for i in extension.include_dirs:
!                 if not i in includes:
!                     includes.append(i)
! 
!         # Set the target_ext to '.c'.  Cython will change this to '.cpp' if
!         # needed.
!         if cplus:
!             target_ext = '.cpp'
!         else:
!             target_ext = '.c'
! 
!         # Drop the generated C files into the temp dir, unless the inplace
!         # option is present.  In that case, drop them into the source tree.
! 
!         if self.inplace:
!             target_dir = ""
!         else:
!             target_dir = os.path.join(self.build_temp, "cython")
! 
!         for source in sources:
!             (base, ext) = os.path.splitext(source)
!             if ext == ".pyx":             # Cython source file
!                 new_sources.append(os.path.join(target_dir, base + target_ext))
!                 cython_sources.append(source)
!                 cython_targets[source] = new_sources[-1]
!             else:
!                 new_sources.append(source)
! 
!         if not cython_sources:
!             return new_sources
! 
!         for source in cython_sources:
!             target = cython_targets[source]
!             log.info("cythoning %s to %s", source, target)
!             self.mkpath(os.path.dirname(target))
!             options = CompilationOptions(cython_default_options, 
!                                          use_listing_file = create_listing,
!                                          include_path = includes,
!                                          output_file = target,
!                                          cplus = cplus)
!             result = cython_compile(source, options=options)
! 
!         return new_sources

!     # cython_sources ()

+ # class build_ext
diff -cbN /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/core.py Distutils/core.py
*** /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/core.py	1969-12-31 19:00:00.000000000 -0500
--- Distutils/core.py	2007-08-03 00:36:10.000000000 -0400
***************
*** 0 ****
--- 1,34 ----
+ """Cython.Distutils.core
+ 
+ This module wraps the 'distutils.core' module and extends it to support
+ Cython extension modules.
+ 
+ This is the only module that needs to be imported to use the Distutils;
+ provides the 'setup' function (which is to be called from the setup script).
+ Also indirectly provides the Distribution and Command classes, although
+ they are really defined in distutils.dist and distutils.cmd.
+ """
+ 
+ # This module should be kept compatible with Python 2.1.
+ 
+ __revision__ = "$Id:$"
+ 
+ import sys, os
+ from types import *
+ 
+ from distutils.debug import DEBUG
+ from distutils.errors import *
+ from distutils.util import grok_environment_error
+ 
+ # Mainly import these so setup scripts can "from distutils.core import" them.
+ from distutils.dist import Distribution
+ from distutils.cmd import Command
+ 
+ # Make sure we get Cython's Extension class.
+ from Cython.Distutils.extension import Extension
+ 
+ from distutils.core import *
+ 
+ extension_keywords = extension_keywords + \
+                      ('cython_include_dirs', 'cython_create_listing',
+                       'cython_cplus')
diff -cbN /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/extension.py Distutils/extension.py
*** /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/extension.py	1969-12-31 19:00:00.000000000 -0500
--- Distutils/extension.py	2007-08-03 00:36:10.000000000 -0400
***************
*** 0 ****
--- 1,73 ----
+ """Cython.Distutils.extension
+ 
+ Provides a modified Extension class, that understands hou to describe
+ Cython extension modules in setup scripts."""
+ 
+ __revision__ = "$Id:$"
+ 
+ import os, string, sys
+ from types import *
+ import distutils.extension as _Extension
+ 
+ try:
+     import warnings
+ except ImportError:
+     warnings = None
+ 
+ class Extension(_Extension.Extension):
+     _Extension.Extension.__doc__ + \
+     """cython_include_dirs : [string]
+         list of directories to search for Cython header files (.pxd) (in
+         Unix form for portability)
+     cython_create_listing_file : boolean
+         write cython error messages to a listing (.lis) file.
+     cython_cplus : boolean
+         use the C++ compiler for compiling and linking.
+     """
+ 
+     # When adding arguments to this constructor, be sure to update
+     # setup_keywords in core.py.
+     def __init__ (self, name, sources,
+                   include_dirs=None,
+                   define_macros=None,
+                   undef_macros=None,
+                   library_dirs=None,
+                   libraries=None,
+                   runtime_library_dirs=None,
+                   extra_objects=None,
+                   extra_compile_args=None,
+                   extra_link_args=None,
+                   export_symbols=None,
+                   swig_opts = None,
+                   depends=None,
+                   language=None,
+                   cython_include_dirs=None,
+                   cython_create_listing=0,
+                   cython_cplus=0,
+                   **kw                      # To catch unknown keywords
+                  ):
+ 
+         _Extension.Extension.__init__(self, name, sources,
+                        include_dirs,
+                        define_macros,
+                        undef_macros,
+                        library_dirs,
+                        libraries,
+                        runtime_library_dirs,
+                        extra_objects,
+                        extra_compile_args,
+                        extra_link_args,
+                        export_symbols,
+                        swig_opts ,
+                        depends,
+                        language,
+                        **kw
+                 )
+ 
+         self.cython_include_dirs = cython_include_dirs or []
+         self.cython_create_listing = cython_create_listing
+         self.cython_cplus = cython_cplus
+ 
+ # class Extension
+ 
+ read_setup_file = _Extension.read_setup_file
diff -cbN /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/__init__.py Distutils/__init__.py
*** /home/bga/tmp/cython-0.9.6.3/Cython/Distutils/__init__.py	2007-07-28 15:29:52.000000000 -0400
--- Distutils/__init__.py	2007-08-03 00:36:10.000000000 -0400
***************
*** 1,22 ****
  # July 2002, Graham Fawcett
- 
  #
- 
  # this hack was inspired by the way Thomas Heller got py2exe
- 
  # to appear as a distutil command
- 
  #
- 
  # we replace distutils.command.build_ext with our own version
- 
  # and keep the old one under the module name _build_ext,
- 
  # so that *our* build_ext can make use of it.

  

! from build_ext import build_ext
! 

  
--- 1,16 ----
  # July 2002, Graham Fawcett
  #
  # this hack was inspired by the way Thomas Heller got py2exe
  # to appear as a distutil command
  #
  # we replace distutils.command.build_ext with our own version
  # and keep the old one under the module name _build_ext,
  # so that *our* build_ext can make use of it.

+ __revision__ = "$Id:$"

+ __all__ = [ 'core.py', 'build_ext.py', 'extension.py' ]

! import core, build_ext, extension

_______________________________________________
Cython-dev mailing list
Cython-dev@...
https://lists.berlios.de/mailman/listinfo/cython-dev