Object persistence for Python

Jesus Cea | 24 Sep 2010 19:57

Moving cache from object count to size


Durus objects, when in RAM, could keep the object size in a (volatile,
not stored in disk) attribute. This attribute can be generated when
loading the pickle from disk (you directly have the size), or when
storing the object in the disk (you have to create the pickle, so you
have the size too). In fact, this size bookkeeping could be managed in a
separate internal dictionary, don't have to be inside the object.

I have objects of very dissimilar sizes in my storage, so current cache
control (that is, object count) is not representative of actual memory
usage. I have objects 50 bytes long, and 60Kbytes long . I would
suggest to change the cache code to control cache size, instead of
object count.

Comments?.

PS: I could consider patching this myself, if durus developers are
interested but not spare time.

--

-- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea <at> jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea <at> jabber.org         _/_/    _/_/          _/_/_/_/_/
..                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

headers

Neil Schemenauer | 16 Jan 2011 22:44

Re: Moving cache from object count to size

Jesus Cea <jcea <at> jcea.es> wrote:
> I have objects 50 bytes long, and 60Kbytes long . I would
> suggest to change the cache code to control cache size, instead of
> object count.

I played with this idea a little.  It seems like the bookkeeping
will get pretty complicated.  Iterating over all objects in the
cache to find the current size does not seem feasible, for
performance reasons.  Therefore, the cache would have to keep track
of the total size as objects are added and removed.  The fact that
objects are weakly referenced by the cache futher complicates
things.

Maybe I'm missing a simple implementation but it doesn't appear to
be worth the effort, IMHO.

Regards,

  Neil

headers

Jesus Cea | 15 Mar 2011 20:00

Re: Moving cache from object count to size


On 16/01/11 22:44, Neil Schemenauer wrote:
> Jesus Cea <jcea <at> jcea.es> wrote:
>> I have objects 50 bytes long, and 60Kbytes long . I would
>> suggest to change the cache code to control cache size, instead of
>> object count.
> 
> I played with this idea a little.  It seems like the bookkeeping
> will get pretty complicated.  Iterating over all objects in the
> cache to find the current size does not seem feasible, for
> performance reasons.  Therefore, the cache would have to keep track
> of the total size as objects are added and removed.  The fact that
> objects are weakly referenced by the cache futher complicates
> things.
> 
> Maybe I'm missing a simple implementation but it doesn't appear to
> be worth the effort, IMHO.

You have the weak reference callback, for instance. It is not different
to current cache object count.

--

-- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea <at> jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea <at> jabber.org         _/_/    _/_/          _/_/_/_/_/
..                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz

(Continue reading)