gmane.comp.db.metakit

abused by speech recognition | 8 Aug 21:31

fleeing sql


If I remember correctly, metakit is multi-reader single writer right?
since I need multi-writer, guess I'm best solving that with embedding
metakit in a server using internal access locks?  does metakit work in
a python multi threaded environment either as multi thread access or a
single thread fed by multiple request threads/

thanks for any feedback

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the "metakit" group.
To post to this group, send email to metakit <at> googlegroups.com
To unsubscribe from this group, send email to metakit-unsubscribe <at> googlegroups.com
For more options, visit this group at http://groups.google.com/group/metakit?hl=en
-~----------~----~----~----~------~----~------~--~---

headers

David McNab | 8 Aug 23:17

Re: fleeing sql


On Wed, 2007-08-08 at 12:34 -0700, abused by speech recognition wrote:
> If I remember correctly, metakit is multi-reader single writer right?
> since I need multi-writer, guess I'm best solving that with embedding
> metakit in a server using internal access locks?  does metakit work in
> a python multi threaded environment either as multi thread access or a
> single thread fed by multiple request threads/
> 
> thanks for any feedback

In a python single-process multi-threading environment, I'm sure that
metakit could be made to work well, even if you have to surround the
access statements in each thread with .acquire() and .release() of a
threading.Lock object.

I'm also interested in a multi-process scenario, such as CGI-based web
server. This could be made to work well by tuning the http
request/response cycle:

 - create a metakit.storage() object at start of the http response
   cycle, with the database open in read mode

 - for a very small part of the http response cycle:
     - acquire a file lock on the metakit database file
     - create another metakit.storage() object on the file, this one
       with the database open in read/write mode
     - perform all needed write operations
     - commit() the the read-write storage object
     - del the read-write storage object
     - release the file lock

(Continue reading)

headers

Riccardo Cohen | 9 Aug 09:53

Re: fleeing sql

About multi threading in a client application, my opinion is similar : 
you have to lock something for writing. I don't know however the effect 
of a write operation while another thread is reading... maybe memory 
mapped file would save you, but I wouldn't bet on it.

On the other hand, in a web environment, where multiple processes would 
access the same metakit file, my opinion is that the "file lock" method 
as you explained cannot be reliable.

If I had to use metakit in an HTTP server-side application, and if I 
needed a write access, I would build a single process on the server that 
does all metakit access for all processes, and I would talk to that 
single process with sockets. Of course you loose the easy of metakit 
API, but your server is safe.

As I mentioned, I'm not sure of the effect of a write operation while 
another thread/process is reading. So I would also include read 
operations on this single process.

Maybe I'm wrong, please let me know.

David McNab wrote:
> On Wed, 2007-08-08 at 12:34 -0700, abused by speech recognition wrote:
>> If I remember correctly, metakit is multi-reader single writer right?
>> since I need multi-writer, guess I'm best solving that with embedding
>> metakit in a server using internal access locks?  does metakit work in
>> a python multi threaded environment either as multi thread access or a
>> single thread fed by multiple request threads/
>>

(Continue reading)

headers

Eric S. Johansson | 9 Aug 14:05

Re: fleeing sql

Riccardo Cohen wrote:
> About multi threading in a client application, my opinion is similar : 
> you have to lock something for writing. I don't know however the effect 
> of a write operation while another thread is reading... maybe memory 
> mapped file would save you, but I wouldn't bet on it.

sometimes if you ask Google the same question in a different way, you find your 
answer.

http://www.equi4.com/wikis/metakit/13

In other words, multithreading is not really possible or practical just yet. 
the right solution in a multithreaded environment is, assign metakit to a single 
thread and have all of the other threads speak to the single metakit thread.

To tell you the truth, I'm not sure how well that is going to work for me.  I've 
got three or four applications when I need something that behaves like metakit 
and supports the core functionality I can build "test and set" and "update 
before insert" capabilities out of.  I might just need to use a global lock and 
invoke and close metakit on every query.  Ugly as hell but at least it's not 
SQL.  almost as ugly is the single process single thread version you described. 
  I'm sure using psyco can help take some of the ugliness out of it by exporting 
all of the Python interfaces.  But I wonder how bad performance is going to be.

Are there any other non-SQL non-DBM databases out there?  Or is it just 
primarily SQL and metakit? (and no, I'm not interested in becoming a database 
expert.   I have way too much on my plate to even learn about databases let 
alone design them.

(Continue reading)