headers
gniquil | 4 Sep 06:37

using the native c implementation of ordereddict


Hi All,

I am doing some work with xmlrpc. One thing I realize is that whenever
I pass dict(row) through xmlrpc, I get an key-ordered struct. But this
isn't what i really want. What I want is ordered by insertion or the
original list order. This led me to look at the util.ordereddict
implementation, which is pure python, which is slow. I looked around
and found this:

http://www.xs4all.nl/~anthon/Python/ordereddict/

which is a c-implementation. At the bottom of the page, there are
performance tests. It's much faster. I've got some pretty gigantic
tables to pass around, which i think this would really help. Hopefully
this could somehow find itself into next official python. But before
that, we can use this or we can just incorporate it somehow in
sqlalchemy...as a suggestion.

Frank
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To post to this group, send email to sqlalchemy <at> googlegroups.com
To unsubscribe from this group, send email to sqlalchemy+unsubscribe <at> googlegroups.com
For more options, visit this group at http://groups.google.com/group/sqlalchemy?hl=en
-~----------~----~----~----~------~----~------~--~---
Permalink | Reply |
headers
Michael Bayer | 4 Sep 14:42

Re: using the native c implementation of ordereddict


On Sep 4, 2008, at 12:40 AM, gniquil wrote:

>
> Hi All,
>
> I am doing some work with xmlrpc. One thing I realize is that whenever
> I pass dict(row) through xmlrpc, I get an key-ordered struct. But this
> isn't what i really want. What I want is ordered by insertion or the
> original list order. This led me to look at the util.ordereddict
> implementation, which is pure python, which is slow. I looked around
> and found this:
>
> http://www.xs4all.nl/~anthon/Python/ordereddict/
>
> which is a c-implementation. At the bottom of the page, there are
> performance tests. It's much faster. I've got some pretty gigantic
> tables to pass around, which i think this would really help. Hopefully
> this could somehow find itself into next official python. But before
> that, we can use this or we can just incorporate it somehow in
> sqlalchemy...as a suggestion.
>

the problem with saying "utility class X is slow, therefore use Y" is  
that you haven't evaluated if the slowness of X is really impacting  
the performance of SQLAlchemy overall in a negative way.   I think if  
you ran some profiling results you'd see that OrderedDict calls make  
up a miniscule portion of time spent for doing all operations, so an  
external dependency is not necessarily worth it in this case (though  
it may be).  I have some vague recollection that our own ODict does
(Continue reading)

Permalink | Reply |
headers
az | 4 Sep 15:03

Re: using the native c implementation of ordereddict


On Thursday 04 September 2008 15:42:56 Michael Bayer wrote:
> On Sep 4, 2008, at 12:40 AM, gniquil wrote:
> > Hi All,
> >
> > I am doing some work with xmlrpc. One thing I realize is that
> > whenever I pass dict(row) through xmlrpc, I get an key-ordered
> > struct. But this isn't what i really want. What I want is ordered
> > by insertion or the original list order. This led me to look at
> > the util.ordereddict implementation, which is pure python, which
> > is slow. I looked around and found this:
> >
> > http://www.xs4all.nl/~anthon/Python/ordereddict/
> >
> > which is a c-implementation. At the bottom of the page, there are
> > performance tests. It's much faster. I've got some pretty
> > gigantic tables to pass around, which i think this would really
> > help. Hopefully this could somehow find itself into next official
> > python. But before that, we can use this or we can just
> > incorporate it somehow in sqlalchemy...as a suggestion.
>
> the problem with saying "utility class X is slow, therefore use Y"
> is that you haven't evaluated if the slowness of X is really
> impacting the performance of SQLAlchemy overall in a negative way. 
>  I think if you ran some profiling results you'd see that
> OrderedDict calls make up a miniscule portion of time spent for
> doing all operations, so an external dependency is not necessarily
> worth it in this case (though it may be).  I have some vague
> recollection that our own ODict does some things the native one
> does not but I'd have to dig back into the code to remember what
(Continue reading)

Permalink | Reply |
headers
Michael Bayer | 4 Sep 16:51

Re: using the native c implementation of ordereddict


On Sep 4, 2008, at 9:03 AM, az <at> svilendobrev.com wrote:

>
> i used to set sqlalchemy.util.Set to be sqlalchemy.util.OrderedSet and
> that worked well...
> if all those basic things (dict, set, odict, oset, etc) are always
> routed via sqlachemy.util, then one can replace them with whatever
> fancy. One main usage is that testing cases would be more repeatable,
> because flush plans and other hash-relating things will be same, if
> all sets are ordered and all dicts are ordered.
> this is not going to impact anything speedwise, it only means changing
> several hundred places where {} or set() is used, and keeping some
> discipline of not introducing those in future code.
>
> someone may tell me about a way to directly hack pythons notion of {}
> with something mine... would be good but is going to impact speed of
> *any* code, not just SA.
>
> mike, sorry for repeating myself again on the theme :-)
> i can prepare The patch as long as u decide to keep such "protocol"...

i believe we already have a layer in the test/ suite which can  
globally replace imports with something specific, and it is being  
used.  It's Jason's thing but you can dig into the source to see how  
it works.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "sqlalchemy" group.
To post to this group, send email to sqlalchemy <at> googlegroups.com
(Continue reading)

Permalink | Reply |
headers
az | 4 Sep 19:33

Re: using the native c implementation of ordereddict


On Thursday 04 September 2008 17:51:56 Michael Bayer wrote:
> On Sep 4, 2008, at 9:03 AM, az <at> svilendobrev.com wrote:
> > i used to set sqlalchemy.util.Set to be
> > sqlalchemy.util.OrderedSet and that worked well...
> > if all those basic things (dict, set, odict, oset, etc) are
> > always routed via sqlachemy.util, then one can replace them with
> > whatever fancy. One main usage is that testing cases would be
> > more repeatable, because flush plans and other hash-relating
> > things will be same, if all sets are ordered and all dicts are
> > ordered.
> > this is not going to impact anything speedwise, it only means
> > changing several hundred places where {} or set() is used, and
> > keeping some discipline of not introducing those in future code.
> >
> > someone may tell me about a way to directly hack pythons notion
> > of {} with something mine... would be good but is going to impact
> > speed of *any* code, not just SA.
> >
> > mike, sorry for repeating myself again on the theme :-)
> > i can prepare The patch as long as u decide to keep such
> > "protocol"...
>
> i believe we already have a layer in the test/ suite which can
> globally replace imports with something specific, and it is being
> used.  It's Jason's thing but you can dig into the source to see
> how it works.

nooo, u got me wrong. i'd like all the {} dict() set() usage 
all-over-sa to be routed via util.dict and util.set (which default to
(Continue reading)

Permalink | Reply |
headers
jason kirtland | 4 Sep 16:56
Favicon
Gravatar

Re: using the native c implementation of ordereddict


Michael Bayer wrote:
> 
> On Sep 4, 2008, at 12:40 AM, gniquil wrote:
> 
>> Hi All,
>>
>> I am doing some work with xmlrpc. One thing I realize is that whenever
>> I pass dict(row) through xmlrpc, I get an key-ordered struct. But this
>> isn't what i really want. What I want is ordered by insertion or the
>> original list order. This led me to look at the util.ordereddict
>> implementation, which is pure python, which is slow. I looked around
>> and found this:
>>
>> http://www.xs4all.nl/~anthon/Python/ordereddict/
>>
>> which is a c-implementation. At the bottom of the page, there are
>> performance tests. It's much faster. I've got some pretty gigantic
>> tables to pass around, which i think this would really help. Hopefully
>> this could somehow find itself into next official python. But before
>> that, we can use this or we can just incorporate it somehow in
>> sqlalchemy...as a suggestion.
>>
> 
> the problem with saying "utility class X is slow, therefore use Y" is  
> that you haven't evaluated if the slowness of X is really impacting  
> the performance of SQLAlchemy overall in a negative way.   I think if  
> you ran some profiling results you'd see that OrderedDict calls make  
> up a miniscule portion of time spent for doing all operations, so an  
> external dependency is not necessarily worth it in this case (though
(Continue reading)

Permalink | Reply |

Gmane