gmane.comp.python.devel

Kristján Valur Jónsson | 8 Oct 21:24

getattr and new style classes

Hello there.

I‘ve just noticed what I consider a performance problem:

Using new style classes to provide attribute-like access using __getattr__ is considerably slower than old style classes: Observe:

s = """

class dude:

def bar(self):pass

def __getattr__(self, a): return a

class dude2(object):

def bar(self):pass

def __getattr__(self, a): return a

d = dude()

d2 = dude2()

d.a = d2.a = 1

"""

timeit.Timer(´d.foo´, s).timeit()

>0.32979211801421116

timeit.Timer(´d2.foo´, s).timeit()

> 1.1119853719342245

The overhead is almost 3 times as high. I imagine that this is because new style classes must search further and harder before giving up and going to __getattr__.

For the bound method the difference is less:

timeit.Timer(´d.bar´, s).timeit()

> 0.11835480370018558

timeit.Timer(´d2.bar´, s).timeit()

> 0.17820851929263881

For fun, I also tested regular attributes, and see:

timeit.Timer(´d.a´, s).timeit()

> 0.069161394202183146

timeit.Timer(´d2.a´, s).timeit()

> 0.17966275972594303

I’m surprised that accessing instance attributes like this is twice as slow using new style classes.

Any thoughts on this?

We are using a lot of low-level attribute access magic in EVE and so it would appear that we are best served by sticking with old-style classes. But these are going away eventually, so what to do?

Where is this extra overhead coming from?

And oh, it is no use using __getattribute__ instead, since it will always involve calls to object.__getattribute__ and become very slow.

Cheers,

Kristján

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Nick Coghlan | 8 Oct 23:37

Re: getattr and new style classes

Kristján Valur Jónsson wrote:
> Hello there.
> 
> I‘ve just noticed what I consider a performance problem:
> 
> Using new style classes to provide attribute-like access using
> __getattr__ is considerably slower than old style classes:  Observe:

I can't reproduce those relative numbers using SVN trunk. Using your
setup code (copied and pasted directly from your message to my
interpreter session) I got the following numbers:

>>> timeit.Timer('d.foo', s).timeit()
1.2362558841705322
>>> timeit.Timer('d2.foo', s).timeit()
1.1634600162506104
>>> timeit.Timer('d.foo', s).timeit()
1.1840031147003174
>>> timeit.Timer('d2.foo', s).timeit()
1.1554200649261475

(a slight speed advantage to the new-style class for __getattr__)

>>> timeit.Timer('d.bar', s).timeit()
0.17601609230041504
>>> timeit.Timer('d2.bar', s).timeit()
0.18697309494018555
>>> timeit.Timer('d.bar', s).timeit()
0.1711127758026123
>>> timeit.Timer('d2.bar', s).timeit()
0.1827549934387207

(very slight speed advantage to the old-style class for the unbound method)

Cheers,
Nick.

--

-- 
Nick Coghlan   |   ncoghlan <at> gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Kristján Valur Jónsson | 9 Oct 13:21

Re: getattr and new style classes

Thanks for trying this out.
When I saw your comments, I realized that the difference is due to a speed patch that we have her at CCP.
The test was run using current 2.5 python, with a patch applied, as attached this message.
The patch was written by me a few years back when I noticed that python spends enormous time during regular
run creating exceptions that then get discarded internally.  Most of the cost comes from FormatException
when it is generating AttributeErrors.
So, when I thought I was complaining about slow new objects, I was really showing off my cool optimization.

The problem is, it is difficult to generalize this to python in general.  I spent some time last year to
try to improve new-style classes, by adding ts fields and telling the exception system to "expect a certain
type of exception, and if it is raised, use PyErr_SetNone() instead of the PyErr_Format because I will
clear it anyway".  I had macros such as
PyErr_Expect(AttributeError);
attr = PyObject_GetAttr(o, a);
PyErr_Expect(0);
if (!attr)
        PyErr_Clear()

The problem was, I wasn't able to come up with a simple patch that showed consistent speed improvements
in the performance testsuite without showing some slowdowns in other places.

Running regular python code through a profiler, and especially code that relies much on the use of
__getattr__() to emulate attribute access, will show hideous amounts of time spent formatting
attribute exceptions that get thrown away.

Any thoughts on how to do this better?

And, for completeness, a new test run of this, using python25, first using the neutered patch version
(where I have set softfail=0 in the relevant places) and then with the patch active:

import timeit
s = """
class dude:
        def bar(self):pass
        def __getattr__(self, a): return a
class dude2(object):
        def bar(self):pass
        def __getattr__(self, a): return a
d = dude()
d2 = dude2()
d.a = d2.a = 1
"""
print timeit.Timer("d.foo", s).timeit()
print timeit.Timer("d.bar", s).timeit()
print timeit.Timer("d.a", s).timeit()

print timeit.Timer("d2.foo", s).timeit()
print timeit.Timer("d2.bar", s).timeit()
print timeit.Timer("d2.a", s).timeit()

patch neutered
1.35734336404
0.157773452422
0.0937950608722
1.48494915604
0.240154539405
0.186524222345

patch active:
0.352850669641
0.147599760073
0.0910020300097
1.4453737036
0.212842069748
0.203442097864

Cheers,

Kristján
> -----Original Message-----
> From: Nick Coghlan [mailto:ncoghlan <at> gmail.com]
> Sent: Wednesday, October 08, 2008 21:37
> To: Kristján Valur Jónsson
> Cc: Python-Dev
> Subject: Re: [Python-Dev] __getattr__ and new style classes
>
> Kristján Valur Jónsson wrote:
> > Hello there.
> >
> > I‘ve just noticed what I consider a performance problem:
> >
> > Using new style classes to provide attribute-like access using
> > __getattr__ is considerably slower than old style classes:  Observe:
>
> I can't reproduce those relative numbers using SVN trunk. Using your
> setup code (copied and pasted directly from your message to my
> interpreter session) I got the following numbers:
>
>

Attachment (speedpatch.rar): application/octet-stream, 1639 bytes

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Nick Coghlan | 9 Oct 13:53

Re: getattr and new style classes

Kristján Valur Jónsson wrote:
> Running regular python code through a profiler, and especially code that relies much on the use of
> __getattr__() to emulate attribute access, will show hideous amounts of time spent formatting
> attribute exceptions that get thrown away.
> 
> Any thoughts on how to do this better?

If the time is being spent in PyErr_Format, how far could you get adding
a dedicated function for creating AttributeErrors? Something along the
lines of:

PyErr_AttributeError(PyObject *object, PyObject *attr_name)

It would then be possible to modify AttributeError to have obj_type and
attr_name attributes (holding the type name and the name of the missing
attribute), so that any string formatting could be deferred until repr()
was called on the AttributeError instance.

Cheers,
Nick.

--

-- 
Nick Coghlan   |   ncoghlan <at> gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Kristján Valur Jónsson | 9 Oct 16:09

Re: getattr and new style classes

It's an interesting idea...  But it seems hard to forge the magic that
does this (creating it in args[0] on demand) without some heavy work.

In my opinion, the problem is that we are raising all these errors
at all, with object creation and all that it entails.  We are using
exceptions very much as program logic to test for the existence of
attributes.

Python itself has the getattr(obj, attr, default) thing, encouraging
you to not use exceptions where the absence of an attribute is
fully excpected.  But the C layer doesn't have this.

In my opinion, the C api should have this capability, which is why
I bothered with implementing PyObject_GetAttrSoft() there.
Ideally, it should have a signature such as:
int PyObject_GetAttrSoft(PyObject **result, PyObject *obj, PyObject *attr, int raise);
where the return value indicates success (0) or an exception (-1) and the
"result" gets either NULL or something else in case of success.
"raise" would indicate whether we want the absence of an attibute to raise
an exception or not.

Now this is all well and good, but the big problem is that we invariably
end up calling the tp->tp_getattr slots that have no such functionality.

I'd actually be more inclined to try to go for a full solution using a
tp_gettattrc (c for conditional) with a default implementation....
Maybe I'll play with that over the weekend, see where it takes us.

K

> -----Original Message-----
> From: Nick Coghlan [mailto:ncoghlan <at> gmail.com]
> Sent: Thursday, October 09, 2008 11:54
> To: Kristján Valur Jónsson
> Cc: Python-Dev
> Subject: Re: [Python-Dev] __getattr__ and new style classes
>
> Kristján Valur Jónsson wrote:
> > Running regular python code through a profiler, and especially code
> that relies much on the use of
> > __getattr__() to emulate attribute access, will show hideous amounts
> of time spent formatting
> > attribute exceptions that get thrown away.
> >
> > Any thoughts on how to do this better?
>
> If the time is being spent in PyErr_Format, how far could you get
> adding
> a dedicated function for creating AttributeErrors? Something along the
> lines of:
>
> PyErr_AttributeError(PyObject *object, PyObject *attr_name)
>
> It would then be possible to modify AttributeError to have obj_type and
> attr_name attributes (holding the type name and the name of the missing
> attribute), so that any string formatting could be deferred until
> repr()
> was called on the AttributeError instance.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan <at> gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>             http://www.boredomandlaziness.org

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Greg Ewing | 10 Oct 02:15

Re: getattr and new style classes

Nick Coghlan wrote:

> If the time is being spent in PyErr_Format, how far could you get adding
> a dedicated function for creating AttributeErrors? Something along the
> lines of:
> 
> PyErr_AttributeError(PyObject *object, PyObject *attr_name)

More generally, it might be useful to have some mechanism for
deferred instantiation of exceptions, so you can do something
like indicate what type of exception you want to raise, and
specify a function and some arguments to call to instantiate
the exception, but the instantiation itself doesn't happen
unless the exception object is actually needed by Python
code.

--

-- 
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Amaury Forgeot d'Arc | 10 Oct 02:28

Re: getattr and new style classes

Hello,

On Fri, Oct 10, 2008 at 2:15 AM, Greg Ewing <greg.ewing <at> canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>
>> If the time is being spent in PyErr_Format, how far could you get adding
>> a dedicated function for creating AttributeErrors? Something along the
>> lines of:
>>
>> PyErr_AttributeError(PyObject *object, PyObject *attr_name)
>
> More generally, it might be useful to have some mechanism for
> deferred instantiation of exceptions, so you can do something
> like indicate what type of exception you want to raise, and
> specify a function and some arguments to call to instantiate
> the exception, but the instantiation itself doesn't happen
> unless the exception object is actually needed by Python
> code.

But this is already the case, and the reason why there are three
variable to describe an exception: type, value and traceback.
At the C level, the value is often a string (when using PyErr_Format,
for example), or a tuple.
Normalization (=creation of the exception object) only occurs when
needed, e.g when entering an "except:" handler in python code, or when
the exception is printed.
However, the "value" string object must be created anyway, and this
could be avoided in most Getattr cases.

--

-- 
Amaury Forgeot d'Arc
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Greg Ewing | 10 Oct 02:37

Re: getattr and new style classes

Amaury Forgeot d'Arc wrote:

> But this is already the case, and the reason why there are three
> variable to describe an exception: type, value and traceback.

Yes, but you only get one object for the value, which means
at least allocating a tuple if you want to be able to report
something like "AttributeError: 'fooblat' object has no
attribute 'asdf'".

I'm thinking about a C api that would let you supply multiple
pre-existing values without having to allocate anything.
Not sure exactly how it would work, though.

Also, what about Py3? Has the type/value separation gone away
completely, or is it still there at the C level?

--

-- 
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Brett Cannon | 10 Oct 03:43

Re: getattr and new style classes

On Thu, Oct 9, 2008 at 5:37 PM, Greg Ewing <greg.ewing <at> canterbury.ac.nz> wrote:
> Amaury Forgeot d'Arc wrote:
>
>> But this is already the case, and the reason why there are three
>> variable to describe an exception: type, value and traceback.
>
> Yes, but you only get one object for the value, which means
> at least allocating a tuple if you want to be able to report
> something like "AttributeError: 'fooblat' object has no
> attribute 'asdf'".
>
> I'm thinking about a C api that would let you supply multiple
> pre-existing values without having to allocate anything.
> Not sure exactly how it would work, though.
>
> Also, what about Py3? Has the type/value separation gone away
> completely, or is it still there at the C level?
>

It's still there. I think the major C changes for exceptions was
adding support for the three new attributes (__traceback__,
__context__, and __cause__).

-Brett
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Steven D'Aprano | 9 Oct 00:35

Re: getattr and new style classes

On Thu, 9 Oct 2008 06:27:06 am Kristján Valur Jónsson wrote:
> Hello there.
> I've just noticed what I consider a performance problem:
> Using new style classes to provide attribute-like access using
> __getattr__ is considerably slower than old style classes:  Observe:
>
> s = """
> class dude:
>         def bar(self):pass
>         def __getattr__(self, a): return a
> class dude2(object):
>         def bar(self):pass
>         def __getattr__(self, a): return a
> d = dude()
> d2 = dude2()
> d.a = d2.a = 1
> """
> timeit.Timer(´d.foo´, s).timeit()
> >0.32979211801421116
>
> timeit.Timer(´d2.foo´, s).timeit()
> > 1.1119853719342245

Not only don't I observe the same results as you, I'm afraid I can't 
even get your code to run. I get a SyntaxError from the funny quotes 
you're using: ´d.foo´ instead of 'd.foo' or "d.foo".

Also, I trust you know enough not to pay too much attention to a single 
test result? Timing results are notoriously subject to interference 
from external processes.

Finally, when reporting performance problems, it will be VERY helpful to 
report the version and platform you are using. There's no point in 
having people looking for a slow-down in Python 2.6 if you're actually 
using Python 2.3 (say).

Anyway, for the record here are my results for Python 2.5. I don't 
believe the differences are significant.

>>> min(Timer('d.foo', s).repeat())
1.9228429794311523
>>> min(Timer('d2.foo', s).repeat())
2.1040639877319336

>>> min(Timer('d.bar', s).repeat())
0.68272304534912109
>>> min(Timer('d2.bar', s).repeat())
0.41949796676635742

>>> min(Timer('d.a', s).repeat())
0.45727396011352539
>>> min(Timer('d2.a', s).repeat())
0.59516501426696777

--

-- 
Steven
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Christian Heimes | 9 Oct 00:53

Re: getattr and new style classes

Steven D'Aprano wrote:
> Not only don't I observe the same results as you, I'm afraid I can't 
> even get your code to run. I get a SyntaxError from the funny quotes 
> you're using: ´d.foo´ instead of 'd.foo' or "d.foo".

Kristján is using the old style and alternative syntax for repr(). 
Somehow the `` got screwed up by either his mailer or the mailing list. 
Don't be ashamed that you aren't aware of the alternative syntax. We 
keep it locked up in the cellar and it has been removed from the new, 
shiny Python 3 world.

 >>> `object`
"<type 'object'>"
 >>> `object` == repr(object)
True

Christian

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Nick Coghlan | 9 Oct 13:16

Re: getattr and new style classes

Christian Heimes wrote:
> Steven D'Aprano wrote:
>> Not only don't I observe the same results as you, I'm afraid I can't
>> even get your code to run. I get a SyntaxError from the funny quotes
>> you're using: ´d.foo´ instead of 'd.foo' or "d.foo".
> 
> Kristján is using the old style and alternative syntax for repr().
> Somehow the `` got screwed up by either his mailer or the mailing list.
> Don't be ashamed that you aren't aware of the alternative syntax. We
> keep it locked up in the cellar and it has been removed from the new,
> shiny Python 3 world.

I think it's actually some single quotes that got mangled by the mailer.
Either way, something else is going on for Kristján to see such wildly
different results between old-style and new-style attribute access, when
the differences are in the noise for the other folks checking it.

I was able to get the old-style class to be consistently faster by going
back and trying the example on 2.4, but the change still wasn't even
close to the dramatic difference Kristján is seeing.

Kristján, is there a chance CCP optimised something in the old-style
class attribute lookup code and didn't get around to submitting the
patch back to python.org?

Cheers,
Nick.

--

-- 
Nick Coghlan   |   ncoghlan <at> gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Christian Heimes | 9 Oct 13:23

Re: getattr and new style classes

Nick Coghlan wrote:
> I think it's actually some single quotes that got mangled by the mailer.
> Either way, something else is going on for Kristján to see such wildly
> different results between old-style and new-style attribute access, when
> the differences are in the noise for the other folks checking it.

I still think they are back ticks. Maybe the repr() function of new
style classes is slower than the old style repr(). It might explain the
difference.

Christian
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Nick Coghlan | 9 Oct 13:34

Re: getattr and new style classes

Christian Heimes wrote:
> Nick Coghlan wrote:
>> I think it's actually some single quotes that got mangled by the mailer.
>> Either way, something else is going on for Kristján to see such wildly
>> different results between old-style and new-style attribute access, when
>> the differences are in the noise for the other folks checking it.
> 
> I still think they are back ticks. Maybe the repr() function of new
> style classes is slower than the old style repr(). It might explain the
> difference.

Nope - if they were backticks, you'd either get a NameError (if d and d2
aren't defined in the scope where the timeit calls are being made), or
you'd end up timing the evaluation of some string literals.

Mailer issues aside though, we still don't know where that initial set
of strange numbers is coming from.

Cheers,
Nick.

--

-- 
Nick Coghlan   |   ncoghlan <at> gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Kristján Valur Jónsson | 9 Oct 13:23

Re: getattr and new style classes

No, it was really me being sloppy using outlook and fighting the editor trying to insert smart quotes :)
Sorry for the confusion.
K

> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com <at> python.org
> [mailto:python-dev-bounces+kristjan=ccpgames.com <at> python.org] On Behalf
> Of Christian Heimes
> Kristján is using the old style and alternative syntax for repr().
> Somehow the `` got screwed up by either his mailer or the mailing list.
> Don't be ashamed that you aren't aware of the alternative syntax. We
> keep it locked up in the cellar and it has been removed from the new,
> shiny Python 3 world.
>

_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Greg Ewing | 9 Oct 01:55

Re: getattr and new style classes

Kristján Valur Jónsson wrote:

> Using new style classes to provide attribute-like access using 
> __getattr__ is considerably slower than old style classes

Do you really need __getattr__, or could you use
properties instead?

--

-- 
Greg
_______________________________________________
Python-Dev mailing list
Python-Dev <at> python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

Return

Return to gmane.comp.python.devel.

Advertisement

Search Archive

Language

Change language

Options

Current view: Threads only / Showing whole messages / Not hiding cited text.
Change to All messages, shortened messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane