using the native c implementation of ordereddict
gniquil
I am doing some work with xmlrpc. One thing I realize is that whenever
I pass dict(row) through xmlrpc, I get an key-ordered struct. But this
isn't what i really want. What I want is ordered by insertion or the
original list order. This led me to look at the util.ordereddict
implementation, which is pure python, which is slow. I looked around
and found this:
http://www.xs4all.nl/~anthon/Python/ordereddict/
which is a c-implementation. At the bottom of the page, there are
performance tests. It's much faster. I've got some pretty gigantic
tables to pass around, which i think this would really help. Hopefully
this could somehow find itself into next official python. But before
that, we can use this or we can just incorporate it somehow in
sqlalchemy...as a suggestion.
Frank
Michael Bayer
the problem with saying "utility class X is slow, therefore use Y" is
that you haven't evaluated if the slowness of X is really impacting
the performance of SQLAlchemy overall in a negative way. I think if
you ran some profiling results you'd see that OrderedDict calls make
up a miniscule portion of time spent for doing all operations, so an
external dependency is not necessarily worth it in this case (though
it may be). I have some vague recollection that our own ODict does
some things the native one does not but I'd have to dig back into the
code to remember what they were. If our own ODict could be swappable
with ordereddict, we could at least try to import it then fall back to
our own (this is what it would look like if ordereddict were
introduced into python core anyway).
a...@svilendobrev.com
i used to set sqlalchemy.util.Set to be sqlalchemy.util.OrderedSet and
that worked well...
if all those basic things (dict, set, odict, oset, etc) are always
routed via sqlachemy.util, then one can replace them with whatever
fancy. One main usage is that testing cases would be more repeatable,
because flush plans and other hash-relating things will be same, if
all sets are ordered and all dicts are ordered.
this is not going to impact anything speedwise, it only means changing
several hundred places where {} or set() is used, and keeping some
discipline of not introducing those in future code.
someone may tell me about a way to directly hack pythons notion of {}
with something mine... would be good but is going to impact speed of
*any* code, not just SA.
mike, sorry for repeating myself again on the theme :-)
i can prepare The patch as long as u decide to keep such "protocol"...
svil
Michael Bayer
On Sep 4, 2008, at 9:03 AM, a...@svilendobrev.com wrote:
>
> i used to set sqlalchemy.util.Set to be sqlalchemy.util.OrderedSet and
> that worked well...
> if all those basic things (dict, set, odict, oset, etc) are always
> routed via sqlachemy.util, then one can replace them with whatever
> fancy. One main usage is that testing cases would be more repeatable,
> because flush plans and other hash-relating things will be same, if
> all sets are ordered and all dicts are ordered.
> this is not going to impact anything speedwise, it only means changing
> several hundred places where {} or set() is used, and keeping some
> discipline of not introducing those in future code.
>
> someone may tell me about a way to directly hack pythons notion of {}
> with something mine... would be good but is going to impact speed of
> *any* code, not just SA.
>
> mike, sorry for repeating myself again on the theme :-)
> i can prepare The patch as long as u decide to keep such "protocol"...
i believe we already have a layer in the test/ suite which can
globally replace imports with something specific, and it is being
used. It's Jason's thing but you can dig into the source to see how
it works.
jason kirtland
fwiw i spiked this out a while back (just before 0.4.0, maybe), and
swapping in a native ordered dict was a very marginal speed improvement,
and most of it was in metadata setup rather than runtime speed.
as svil said, it's easy to try this out by monkeypatching in alternate
implementations and then hitting the various profiling and speed tests
in the test suite.
a...@svilendobrev.com
nooo, u got me wrong. i'd like all the {} dict() set() usage
all-over-sa to be routed via util.dict and util.set (which default to
the builtins), so then one could easily replace them with whatever
s/he wants. i guess one could hack the python builtins/module or
globals() to replace the dict() and set() globally, but i dont think
{} is affected; still, such hack will affect any other code, and not
only SA.
Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52)
>>> dict
<type 'dict'>
>>> globals()
{'__builtins__': <module '__builtin__'
(built-in)>, '__name__': '__main__', '__doc__': None}
>>> locals()
{'__builtins__': <module '__builtin__'
(built-in)>, '__name__': '__main__', '__doc__': None}
>>> __builtins__
<module '__builtin__' (built-in)>
>>> __builtins__.dict
<type 'dict'>
>>> class mydict( dict): pass
...
>>> __builtins__.dict = mydict
>>> dict
<class '__main__.mydict'>
>>> type({})
<type 'dict'>
>>>