Larry Hastings | 15 Jun 2012 23:24

PEP 362 implementation issue: C callables



One wrinkle of PEP 362 in CPython is callables implemented directly in C.  Right now there's no good way to generate such signatures automatically.  That's why we kept __signature__ in the spec, to allow implementors to generate signatures by hand for these callables.

But the wrinkle gets wrinklier.  The problem is, you often can't add arbitrary attributes to these objects:
>>> import os
>>> os.stat.random_new_attribute = 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'builtin_function_or_method' object has no attribute 'random_new_attribute'
If users are to attach precomputed signatures to callables implemented in C, those callables may need to *already* have a __signature__ attribute.


I've already added __signature__ to PyCFunctionObject in the PEP 362 patch.  This takes care of the vast bulk of such callables.  But there are a bunch of obscure callable objects in CPython, and I suspect some of them may need a __signature__ attribute.

Here's an almost- complete list, listed as their Python type:
functools.KeyWrapper (result of functools.cmp_to_key)
weakref.weakref
method (bound method objects, aka "x = Class(); x.method")
instancemethod (is this still even used?  it's never used in trunk)
operator.itemgetter
operator.attrgetter
operator.methodcaller
_json.Scanner (result of _json.make_scanner)
_json.Encoder (result of _json.make_encoder)
_ctypes.DictRemover (I'm not even sure how you can get your hands on one of these)
sqlite3.Connection
I produced this by grepping for "/* tp_call */" and ignoring all of them that were set to "0", then creating one of those objects and trying (and failing) to set a __signature__ attribute.

There are four more candidates I found with the grep but couldn't figure out how to instantiate and test.  They have to do with the descriptor protocol, aka properties, but the types aren't directly exposed by Python.  They're all defined in Object/descrobject.c.  The internal class names are:
method_descriptor
classmethod_descriptor
wrapper_descriptor
method-wrapper (you get one of these out of a "wrapper_descriptor")

I'm happy to do the work, but I don't have a good sense of which callables need a __signature__ attribute.  Can someone here tell me which ones might need a __signature__ attribute?


/arry
<div>
    <br><br>
    One wrinkle of PEP 362 in CPython is callables implemented directly
    in C.&nbsp; Right now there's no good way to generate such signatures
    automatically.&nbsp; That's why we kept __signature__ in the spec, to
    allow implementors to generate signatures by hand for these
    callables.<br><br>
    But the wrinkle gets wrinklier.&nbsp; The problem is, you often can't add
    arbitrary attributes to these objects:<br><blockquote>&gt;&gt;&gt; import os<br>
      &gt;&gt;&gt; os.stat.random_new_attribute = 3<br>
      Traceback (most recent call last):<br>
      &nbsp; File "&lt;stdin&gt;", line 1, in &lt;module&gt;<br>
      AttributeError: 'builtin_function_or_method' object has no
      attribute 'random_new_attribute'<br>
</blockquote>
    If users are to attach precomputed signatures to callables
    implemented in C, those callables may need to *already* have a
    __signature__ attribute.<br><br><br>
    I've already added __signature__ to PyCFunctionObject in the PEP 362
    patch.&nbsp; This takes care of the vast bulk of such callables.&nbsp; But
    there are a bunch of obscure callable objects in CPython, and I
    suspect some of them may need a __signature__ attribute.<br><br>
    Here's an almost- complete list, listed as their Python type:<br><blockquote>functools.KeyWrapper (result of functools.cmp_to_key)<br>
      weakref.weakref<br>
      method (bound method objects, aka "x = Class(); x.method")<br>
      instancemethod (is this still even used?&nbsp; it's never used in
      trunk)<br>
      operator.itemgetter<br>
      operator.attrgetter<br>
      operator.methodcaller<br>
      _json.Scanner (result of _json.make_scanner)<br>
      _json.Encoder (result of _json.make_encoder)<br>
      _ctypes.DictRemover (I'm not even sure how you can get your hands
      on one of these)<br>
      sqlite3.Connection<br>
</blockquote>
    I produced this by grepping for "/* tp_call */" and ignoring all of
    them that were set to "0", then creating one of those objects and
    trying (and failing) to set a __signature__ attribute.<br><br>
    There are four more candidates I found with the grep but couldn't
    figure out how to instantiate and test.&nbsp; They have to do with the
    descriptor protocol, aka properties, but the types aren't directly
    exposed by Python.&nbsp; They're all defined in Object/descrobject.c.&nbsp;
    The internal class names are:<br><blockquote>
      method_descriptor<br>
      classmethod_descriptor<br>
      wrapper_descriptor<br>
      method-wrapper (you get one of these out of a
      "wrapper_descriptor")<br>
</blockquote>
    <br>
    I'm happy to do the work, but I don't have a good sense of which
    callables need a __signature__ attribute.&nbsp; Can someone here tell me
    which ones might need a __signature__ attribute?<br><br><br>/arry<br>
</div>
Yury Selivanov | 15 Jun 2012 23:33
Picon

Re: PEP 362 implementation issue: C callables

On 2012-06-15, at 5:24 PM, Larry Hastings wrote:
[snip]
> I'm happy to do the work, but I don't have a good sense of which callables need a __signature__ attribute. 
Can someone here tell me which ones might need a __signature__ attribute?

I think may we need to be able to write __signature__ attribute
to 'classmethod', 'staticmethod', 'functools.partial' and 
'method' ('types.MethodType').

-
Yury
Daniel Urban | 16 Jun 2012 12:21
Picon

Re: PEP 362 implementation issue: C callables

On Fri, Jun 15, 2012 at 11:24 PM, Larry Hastings <larry <at> hastings.org> wrote:
> There are four more candidates I found with the grep but couldn't figure out
> how to instantiate and test.  They have to do with the descriptor protocol,
> aka properties, but the types aren't directly exposed by Python.  They're
> all defined in Object/descrobject.c.  The internal class names are:
>
> method_descriptor
> classmethod_descriptor
> wrapper_descriptor
> method-wrapper (you get one of these out of a "wrapper_descriptor")

'method_descriptor' is apparently used for the methods of built-in
(implemented in C) objects:
>>> set.__dict__['union'].__class__
<class 'method_descriptor'>

'classmethod_descriptor' is similarly for the classmethods of built-in classes:
>>> import itertools
>>> itertools.chain.__dict__['from_iterable'].__class__
<class 'classmethod_descriptor'>

'wrapper_descriptor' is used for example the operators of built-in types:
>>> int.__dict__['__add__'].__class__
<class 'wrapper_descriptor'>

And 'method-wrapper':
>>> (5).__add__.__class__
<class 'method-wrapper'>

Daniel
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Gmane