headers
Fred C | 18 Apr 01:02
Picon

sqlobjects and threads


I was wandering if there is somewhere an sqlobject 101 for threading  
environment?
I have been googleing for infomation but everything seems confusing.

Thanks
-fred- 

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |
headers
Oleg Broytmann | 18 Apr 10:13
X-Face
Picon
Favicon

Re: sqlobjects and threads

On Thu, Apr 17, 2008 at 04:03:08PM -0700, Fred C wrote:
> I was wandering if there is somewhere an sqlobject 101 for threading  
> environment?

   My personal position is: I avoid threads like a plague.

   Other than that extreme position - SQLObject is mostly thread-safe,
there are people who successfully use it in multithreading TurboGears
applications.
   There is a controversial patch, its author claims without the patch
SQLObject hangs, and there are reports that SQLObject hangs with the patch.
The patch is now in SQLObject but will be removed in the next release
because the last bug report was the patch causes more harm then good. (See
why I avoid threads? Threads are evil by nature: everything in programming
is about separation of control and responsibility - methods, classes,
functions, modules, processes - and threads violate the law by allowing
a few processes to share the same process space and memory; that violation
leads to deadlocks, race conditions and subtle, hard to debug bugs.)

Oleg.
--

-- 
     Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
(Continue reading)

Permalink | Reply |
headers
Carlos Ribeiro | 20 Apr 04:19
Picon

Re: sqlobjects and threads

Oleg,

That's a rant, it's off topic, and I'm probably posting on the wrong forum, but anyway.

I feel your pain, I've spent a good few hours this week debugging threaded programs. At one point I found that I was using the same telnet instance on two threads at the same time. No wonder things were messed up. But it took me two hours to find the root of the problem.

But the fact is that today you have to deal with parallelism in one form or another. It can be threads or multiple processes, it's unavoidable. And it will get worse with newer CPUs.

Your argument reminds me of a recent post by Guido van Rossum. If I remember it right it was one of the posts regarding Python 3k, someone asked if there were any plans to remove the GIL. Guido said that the someone wrote a patch to remove the GIL. It was very complex and the performance was poor, because of the number of extra checks. In the end it became impossible to keep the patch synchronized with the development on the main Python trunk. Given that and given Guido's opinion regarding threads, the GIL continues to be what it is - a single generic lock for Python instead of a more granul ar locking mechanism, which would be better for multithreaded programs (at the cost of complexity).

Also, note that Google AppEngine uses short lived processes instead of threads. It's a different philosophy, and it permeates everything in Google's design.

The hidden assumption is that it's better to scale an application with multiple processes than using threads. But performance may suffer, due to the cost of the synchronization that becomes necessary (note that locking is needed in both cases). Libraries such as SQLObject that keep data cached in memory are heavily affected. There's a potential performance advantage if you use threads with SQLObject because you keep only a single copy of the object in memory for all threads. The difference is noticeable specially for long running processes.

Back to the original question, wich asked about a how to for SQLObjects and threads. I believe the same question can be applied if you resort to multiple processes. What's the best way to do it, specially for long running applications?

On Fri, Apr 18, 2008 at 5:13 AM, Oleg Broytmann <phd <at> phd.pp.ru> wrote:
On Thu, Apr 17, 2008 at 04:03:08PM -0700, Fred C wrote:
> I was wandering if there is somewhere an sqlobject 101 for threading
> environment?

  My personal position is: I avoid threads like a plague.

  Other than that extreme position - SQLObject is mostly thread-safe,
there are people who successfully use it in multithreading TurboGears
applications.
  There is a controversial patch, its author claims without the patch
SQLObject hangs, and there are reports that SQLObject hangs with the patch.
The patch is now in SQLObject but will be removed in the next release
because the last bug report was the patch causes more harm then good. (See
why I avoid threads? Threads are evil by nature: everything in programming
is about separation of control and responsibility - methods, classes,
functions, modules, processes - and threads violate the law by allowing
a few processes to share the same process space and memory; that violation
leads to deadlocks, race conditions and subtle, hard to debug bugs.)

Oleg.
--
    Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
          Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss



--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro <at> gmail.com
mail: carribeiro <at> yahoo.com

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Permalink | Reply |
headers
Oleg Broytmann | 20 Apr 15:19
X-Face
Picon
Favicon

Re: sqlobjects and threads

On Sat, Apr 19, 2008 at 11:19:14PM -0300, Carlos Ribeiro wrote:
> But the fact is that today you have to deal with parallelism in one form or
> another. It can be threads or multiple processes, it's unavoidable. And it
> will get worse with newer CPUs.

   Threads are evil but processes are not - they are completely separated
and cannot destroy each other's memory. Processes have less problem (though
they still need proper synchronisation).

> Your argument reminds me of a recent post by Guido van Rossum. If I remember
> it right it was one of the posts regarding Python 3k, someone asked if there
> were any plans to remove the GIL. Guido said that the someone wrote a patch
> to remove the GIL. It was very complex and the performance was poor, because
> of the number of extra checks. In the end it became impossible to keep the
> patch synchronized with the development on the main Python trunk.

   I remember that thread, and Guido resolved that processes are better
than threads even on multicore CPUs. Multiprocess model with IPC is the
future of Python (and I wholeheartedly agree with this).

> Back to the original question, wich asked about a how to for SQLObjects and
> threads. I believe the same question can be applied if you resort to
> multiple processes. What's the best way to do it, specially for long running
> applications?

   What problems - except of caching - do you foresee?

Oleg.
--

-- 
     Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |
headers
Oleg Broytmann | 20 Apr 17:07
X-Face
Picon
Favicon

Re: sqlobjects and multiprocessing

On Sun, Apr 20, 2008 at 05:19:40PM +0400, Oleg Broytmann wrote:
> On Sat, Apr 19, 2008 at 11:19:14PM -0300, Carlos Ribeiro wrote:
> > But the fact is that today you have to deal with parallelism in one form or
> > another. It can be threads or multiple processes, it's unavoidable. And it
> > will get worse with newer CPUs.
> 
>    Threads are evil but processes are not - they are completely separated
> and cannot destroy each other's memory. Processes have less problem (though
> they still need proper synchronisation).

   Well, processes may have a shared resource - filesystem. Like a shared
memory for threads a shared filesystem requires proper synchronization and
locks, and bugs lead to the same old problems - distorted files, deadlocks
and race conditions.
   The real answer for the problem is to avoid filesystem and use databases.
No, databases don't have magic powers but they are written by more
competent programmers than a general programs and properly implement
synchronization and locks.
   That said, multiprocessing instead of multithreading and databases
instead of files is the future of parallel computing.

Oleg.
--

-- 
     Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |
headers
Carlos Ribeiro | 21 Apr 02:45
Picon

Re: sqlobjects and threads

The "problem" IMHO is with long running processes. Short-lived processes, in the CGI (and now Google AppEngine) style, should have no problems. Long running processes keep lots of data in memory, and keeping it in sync is very hard. It can be nearly as hard as doing threads right. That was the case I was worried about.

But thinking again, you're probably right and this is going to be a non-issue. Few applications today rely on long running processes. Most servers can be implemented efficiently in terms of a "service request" model, using short lived processes. This leaves desktop applications, but that's also changing.

It's curious and a bit frustrating. Threads are potentially much more efficient and faster than processes, but are so hard to get right. Perhaps it's just that we got threads because CPUs were slower. Now CPUs are getting fast enough to run entire processes as fast as we expected a threaded program to run. Time to move on?

On Sun, Apr 20, 2008 at 10:19 AM, Oleg Broytmann <phd <at> phd.pp.ru> wrote:
On Sat, Apr 19, 2008 at 11:19:14PM -0300, Carlos Ribeiro wrote:
> But the fact is that today you have to deal with parallelism in one form or
> another. It can be threads or multiple processes, it's unavoidable. And it
> will get worse with newer CPUs.

  Threads are evil but processes are not - they are completely separated
and cannot destroy each other's memory. Processes have less problem (though
they still need proper synchronisation).

> Your argument reminds me of a recent post by Guido van Rossum. If I remember
> it right it was one of the posts regarding Python 3k, someone asked if there
> were any plans to remove the GIL. Guido said that the someone wrote a patch
> to remove the GIL. It was very complex and the performance was poor, because
> of the number of extra checks. In the end it became impossible to keep the
> patch synchronized with the development on the main Python trunk.

  I remember that thread, and Guido resolved that processes are better
than threads even on multicore CPUs. Multiprocess model with IPC is the
future of Python (and I wholeheartedly agree with this).

> Back to the original question, wich asked about a how to for SQLObjects and
> threads. I believe the same question can be applied if you resort to
> multiple processes. What's the best way to do it, specially for long running
> applications?

  What problems - except of caching - do you foresee?

Oleg.
--
    Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
          Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss



--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro <at> gmail.com
mail: carribeiro <at> yahoo.com

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Permalink | Reply |
headers
Oleg Broytmann | 21 Apr 03:03
X-Face
Picon
Favicon

Re: sqlobjects and threads

On Sun, Apr 20, 2008 at 09:45:28PM -0300, Carlos Ribeiro wrote:
> Threads are potentially much more
> efficient and faster than processes, but are so hard to get right.

   They are faster at the cost of fragility. With the speed of hardware
constantly going up the cost becomes too big.

> Time to move on?

   Absolutely!

Oleg.
--

-- 
     Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |
headers
Frank Barknecht | 21 Apr 19:38
Favicon

Re: sqlobjects and threads

Hallo,
Oleg Broytmann hat gesagt: // Oleg Broytmann wrote:

> On Sat, Apr 19, 2008 at 11:19:14PM -0300, Carlos Ribeiro wrote:
> > But the fact is that today you have to deal with parallelism in one form or
> > another. It can be threads or multiple processes, it's unavoidable. And it
> > will get worse with newer CPUs.
> 
>    Threads are evil but processes are not - they are completely separated
> and cannot destroy each other's memory. Processes have less problem (though
> they still need proper synchronisation).

As I'm recently doing more Lua than Python programming: the Lua
community is traditionally very sceptical of threads. The alternative
approach generally taken in Lua are Coroutines, which found their way
into Python some years ago as well ("yield", generators, iterators
etc.), but only in a limited way compared to Lua's more powerful and
flexible asymmetrical coroutines. An interesting paper in this regard
is this: http://www.inf.puc-rio.br/~roberto/docs/corosblp.pdf

Coroutines are more lightweight than processes and don't need special
synchronisation efforts as only one Coroutine is running at any given
time.

Ciao
--

-- 
 Frank Barknecht                                     _ ______footils.org__

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |
headers
Oleg Broytmann | 21 Apr 19:49
X-Face
Picon
Favicon

Re: sqlobjects and threads

On Mon, Apr 21, 2008 at 07:38:55PM +0200, Frank Barknecht wrote:
> As I'm recently doing more Lua than Python programming: the Lua
> community is traditionally very sceptical of threads. The alternative
> approach generally taken in Lua are Coroutines, which found their way
> into Python some years ago as well ("yield", generators, iterators
> etc.), but only in a limited way compared to Lua's more powerful and
> flexible asymmetrical coroutines. An interesting paper in this regard
> is this: http://www.inf.puc-rio.br/~roberto/docs/corosblp.pdf
> 
> Coroutines are more lightweight than processes and don't need special
> synchronisation efforts as only one Coroutine is running at any given
> time.

http://pypi.python.org/pypi/cogen/

Oleg.
--

-- 
     Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |
headers
Frank Barknecht | 22 Apr 00:06
Favicon

Re: sqlobjects and threads

Hallo,
Oleg Broytmann hat gesagt: // Oleg Broytmann wrote:

> On Mon, Apr 21, 2008 at 07:38:55PM +0200, Frank Barknecht wrote:
> > Coroutines are more lightweight than processes and don't need special
> > synchronisation efforts as only one Coroutine is running at any given
> > time.
> 
> http://pypi.python.org/pypi/cogen/

Yep, that's what I mean. It's often easier and saver to program with
coroutines that with threads.

In Lua COPAS would be an equivalent to Cogen:
http://www.keplerproject.org/copas/

Ciao
--

-- 
Frank Barknecht

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |
headers
Carlos Ribeiro | 26 Apr 12:14
Picon

Re: sqlobjects and threads

Coroutines are safer for two reasons. The first is that synchronization is in the programmer's head, while with threads and processes synchronization is done by the operating system. With coroutines you have no deadlocks or race conditions to worry about because you get to explicitly choose your synchronization points.

But the main point is that coroutines are written for a single processor world. They emulate parallelism inside a single process. That's what really make them safer. But this is not helpful in a world of multiple cores. One still have to choose between threads and processes for large scale multiprocessing.

On Mon, Apr 21, 2008 at 7:06 PM, Frank Barknecht <fbar <at> footils.org> wrote:
Hallo,
Oleg Broytmann hat gesagt: // Oleg Broytmann wrote:

> On Mon, Apr 21, 2008 at 07:38:55PM +0200, Frank Barknecht wrote:
> > Coroutines are more lightweight than processes and don't need special
> > synchronisation efforts as only one Coroutine is running at any given
> > time.
>
> http://pypi.python.org/pypi/cogen/

Yep, that's what I mean. It's often easier and saver to program with
coroutines that with threads.

In Lua COPAS would be an equivalent to Cogen:
http://www.keplerproject.org/copas/

Ciao
--
Frank Barknecht

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss



--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro <at> gmail.com
mail: carribeiro <at> yahoo.com

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Permalink | Reply |
headers
Oleg Broytmann | 26 Apr 19:02
X-Face
Picon
Favicon

Re: sqlobjects and threads

Interview with Donald Knuth:
http://www.informit.com/articles/printerfriendly.aspx?p=1193856&rll=1

   Definitely very interesting by itself, but in the context of the
discussion the most interesting part is about parallel computing,
multithreading and multicore hardware.

Oleg.
--

-- 
     Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |
headers
Colin J. Williams | 27 Apr 04:33
Picon
Favicon

Re: sqlobjects and threads

Oleg Broytmann wrote:
Interview with Donald Knuth: http://www.informit.com/articles/printerfriendly.aspx?p=1193856&rll=1 Definitely very interesting by itself, but in the context of the discussion the most interesting part is about parallel computing, multithreading and multicore hardware. Oleg.
Could someone please remove my name from the mailing list?

Thanks,

Colin W.
-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
_______________________________________________
sqlobject-discuss mailing list
sqlobject-discuss <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss
Permalink | Reply |
headers
Oleg Broytmann | 27 Apr 07:22
X-Face
Picon
Favicon

Re: sqlobjects and threads

On Sat, Apr 26, 2008 at 10:33:45PM -0400, Colin J. Williams wrote:
> <font face="Times New Roman, Times, serif">Could someone please remove
> my name from the mailing list?<br>

   You can always unsubscribe here:
http://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss

Oleg.
--

-- 
     Oleg Broytmann            http://phd.pp.ru/            phd <at> phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference 
Don't miss this year's exciting event. There's still time to save $100. 
Use priority code J8TL2D2. 
http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone
Permalink | Reply |

Gmane