Object persistence for Python

Neil Schemenauer | 28 Apr 2010 03:15

Concurrent Durus server, any interest?

First, a bit of history.  Our original work on Durus began because
we loved the OODB model that ZODB gave us but we did not want to
deal with the extremely complex concurrency model of ZODB.  The
combination of threads and asyncore was enough to make almost
anyone's head explode.  Keeping with the YAGNI principle, the first
version of the Durus server was single threaded and completely
sequential.  It's essentially still that way today (the pack
operation now happens in the background).

Amazingly, this simple design seems to have worked very well.
However, I do notice in certain situations that clients are probably
blocking more than necessary.  For example, if there are multiple
clients and one of them is performing lots of requests that cause
the server to block on disk IO.

Is anyone interested in a concurrent Durus server?  It would almost
certainly have more bugs than the current server.  I'm considering
implementing one either in Clojure or in Python but to this point
can't justify the effort.

  Neil

headers

Jesus Cea | 23 Jun 2010 15:34

Re: Concurrent Durus server, any interest?


On 28/04/10 03:15, Neil Schemenauer wrote:
> Is anyone interested in a concurrent Durus server?

I am.

I have private code (I can't share) where the durus server is
multithread and able to paralelize some (not all) actions, like reads.
This was a paid development for a private organization, managing
currently about 300TB of data. The storage support distributed
transactions and multiple databases.

Another change is that a durus cache miss can (optionally) generate an
exception, to do progress in other threads (sharing the same cache)
while the miss is fetched in background.

I can not share code, but I can say that those improvements make a huge
difference, they were "easy", and I would love to have those
improvements in stock Durus. When I talk about Durus with colleagues
those are the backpoints, and I can't share my code .

--

-- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea <at> jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea <at> jabber.org         _/_/    _/_/          _/_/_/_/_/
..                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

(Continue reading)