headers
m h | 5 Apr 2006 20:27
Picon

Twisted and WSGI...

Hi Folks-

I'm new to this list.  (Am beginning to think more about learning
twisted... for the third time ;) ).  I've got a question for you all
and hope that you don't take it the wrong way.  It's not meant to be
flame bait.

As WSGI is going into python 2.5, I was wondering what the general
consensus is among twisted web devs about it?  Does the WSGI model map
well into twisted's asynchronous model?  Does twisted already have
everything it needs, so WSGI doesn't bring much to the table?

It appears that as AJAX becomes more popular then the asynchronous
model might scale better with many small requests.  Say I have a WSGI
app and I want to port it to twisted.  How easy is that?  (I know
there is a twisted wsgi module)  Do I just need to wrap some logic in
deferreds?

Are any twisted people using WSGI?

Again, I'm just interested in hearing your constructive thoughts. 
Google didn't turn up much on the subject.

thanks,

-matt
Permalink | Reply |
headers
Jim Fulton | 5 Apr 2006 21:04
Gravatar

Re: Twisted and WSGI...

m h wrote:
> Hi Folks-
> 
> I'm new to this list.  (Am beginning to think more about learning
> twisted... for the third time ;) ).  I've got a question for you all
> and hope that you don't take it the wrong way.  It's not meant to be
> flame bait.

I'm not a Twisted developer, so no worries. :)

> As WSGI is going into python 2.5, I was wondering what the general
> consensus is among twisted web devs about it?  Does the WSGI model map
> well into twisted's asynchronous model? 

Not as well as it should IMO.  There seems to be some disdain
for asynchronous servers on the WSGI list.  I tried to
advocate on the WSGI list for better support for asynchronous
servers, especially wrt threading issues.

> Does twisted already have
> everything it needs, so WSGI doesn't bring much to the table?

WSGI makes it much easier for applications to leverage Twisted,
so it increases Twisted's potential user base.

> It appears that as AJAX becomes more popular then the asynchronous
> model might scale better with many small requests. 

An asynchronous model works well for many requests period,
whether they are large or small.

> Say I have a WSGI
> app and I want to port it to twisted.  How easy is that?  (I know
> there is a twisted wsgi module)

Trivial.

>  Do I just need to wrap some logic in deferreds?

No. You don't have to do any twisted programming at all to use the
web server via WSGI.

IMO, to get the full scalability benefits you want, you may
need to address some threading issues.  You might look at the
Zope 3 integration, which isn't as clean as it ought to me.

> Are any twisted people using WSGI?

I don't know. Zope 3 is using it. Zope 2 will be using it in
the future.  I don't think we count as Twisted people.
(Although we might count as twisted people. ;)

Jim

--

-- 
Jim Fulton           mailto:jim <at> zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org
Permalink | Reply |
headers
Ian Bicking | 5 Apr 2006 21:39
Gravatar

Re: Twisted and WSGI...

Jim Fulton wrote:
>> As WSGI is going into python 2.5, I was wondering what the general
>> consensus is among twisted web devs about it?  Does the WSGI model map
>> well into twisted's asynchronous model? 
> 
> 
> Not as well as it should IMO.  There seems to be some disdain
> for asynchronous servers on the WSGI list.  I tried to
> advocate on the WSGI list for better support for asynchronous
> servers, especially wrt threading issues.

I think "disdain" is too strong.  There's a lack of any complete 
proposal which would work, especially one that doesn't have Deferred or 
other particular bits of code as a prerequesite.  There was discussion a 
long time ago, but it drifted off.  More generally, there's no larger 
consensus on how to do asynchronous programming in Python, though 
there's obvious consensus on how to do synchronous programming (that is: 
with functions).

I remember some discussion about allowing an asynchronous application to 
return '' in the WSGI app response iterator, which would signify a 
yield, but I'm not sure where any ready signal would go.

>> Say I have a WSGI
>> app and I want to port it to twisted.  How easy is that?  (I know
>> there is a twisted wsgi module)
> 
> 
> Trivial.

Which reminds me -- I started doing this in Paste, but got bogged down 
in all the setup and imports I didn't understand, and then subscribed 
here and never followed up.  Can someone provide an example of a simple 
function that would look like:

def serve_with_twisted(hosts, wsgi_app, **kw):
     """Serve wsgi_app indefinitely

     hosts is a list of 'address:port', and wsgi_app is a
     WSGI application.  **kw is... whatever other interesting things
     you might want to use to configure a Twisted server
     """

Then I can lightly wrap that and people could use Twisted with their 
Paste configuration files.

>>  Do I just need to wrap some logic in deferreds?
> 
> 
> No. You don't have to do any twisted programming at all to use the
> web server via WSGI.
> 
> IMO, to get the full scalability benefits you want, you may
> need to address some threading issues.  You might look at the
> Zope 3 integration, which isn't as clean as it ought to me.

Is this related to the threadpool, or are you using backdoors to the 
underlying Twisted server to do some things in an asynchronous manner?

--

-- 
Ian Bicking  /  ianb <at> colorstudy.com  /  http://blog.ianbicking.org
Permalink | Reply |
headers
glyph | 6 Apr 2006 02:07
Gravatar

Re: Twisted and WSGI...

On Wed, 05 Apr 2006 14:39:36 -0500, Ian Bicking <ianb <at> colorstudy.com> wrote:

>I think "disdain" is too strong.  There's a lack of any complete proposal 
>which would work, especially one that doesn't have Deferred or other 
>particular bits of code as a prerequesite.  There was discussion a long time 
>ago, but it drifted off.  More generally, there's no larger consensus on how 
>to do asynchronous programming in Python, though there's obvious consensus 
>on how to do synchronous programming (that is: with functions).

There is plenty of consensus on how to do asynchronous programming with Python: use Twisted ;-).

>Which reminds me -- I started doing this in Paste, but got bogged down in 
>all the setup and imports I didn't understand, and then subscribed here and 
>never followed up.  Can someone provide an example of a simple function that 
>would look like:

Taking this approach restricts your deployment to be web-only, which eliminates a substantial advantage
of Twisted.  It isn't hard to implement, though:

from twisted.web2 import server, wsgi
from twisted.web2.channel import http
from twisted.internet import reactor

>def serve_with_twisted(hosts, wsgi_app, **kw):
>     """Serve wsgi_app indefinitely
>
>     hosts is a list of 'address:port', and wsgi_app is a
>     WSGI application.  **kw is... whatever other interesting things
>     you might want to use to configure a Twisted server
>     """
      factory = http.HTTPFactory(server.Site(wsgi.WSGIResource(app)))
      for hostportpair in hosts:
           host, portstr = hostportpair.split(':')
           port = int(portstr)
           reactor.listenTCP(port, factory, interface=host)
      reactor.run()

This is completely untested, but it should at least get you going in the right direction.
Permalink | Reply |
headers
Ed Suominen | 5 Apr 2006 21:01

Re: Twisted and WSGI...

m h wrote:
> I'm new to this list.  (Am beginning to think more about learning
> twisted... for the third time ;) ).  I've got a question for you all
> and hope that you don't take it the wrong way.  It's not meant to be
> flame bait.

Don't worry, I don't see how it could remotely be considered anything
but an honest and constructive inquiry!

> As WSGI is going into python 2.5, I was wondering what the general
> consensus is among twisted web devs about it?  Does the WSGI model map
> well into twisted's asynchronous model?  Does twisted already have
> everything it needs, so WSGI doesn't bring much to the table?

For a Twisted HTTP server (web2) to serve as a WSGI gateway and run a
WSGI application, the application has to either operate under the
Twisted non-blocking model or has to be run in a thread. The latter is
what's being done with everything run under twisted.web2.wsgi right now;
the WSGI Resource object returns the result of a deferToThread(f) where
f is the potentially blocking WSGI app.

However, you could write your WSGI app in a way that quickly returns a
deferred to its output and then crank away at its leisure with some
Twisted-friendly technique like iterations of reactor.callLater(). (The
Twisted devs may correct me and suggest more Twisted-friendly techniques
for "cranking away" at something...) The advantage of the async approach
would be (1) it's more in tune with overall Twisted approach, and (2)
you could have partial results start to show up on the page before the
WSGI app is completely done producing them.

> It appears that as AJAX becomes more popular then the asynchronous
> model might scale better with many small requests.  Say I have a WSGI
> app and I want to port it to twisted.  How easy is that?  (I know
> there is a twisted wsgi module)  Do I just need to wrap some logic in
> deferreds?

Since twisted.web2.wsgi currently runs the WSGI app in a thread (and
will no doubt keep the option of doing so even if it broadens to allow
for async WSGI apps), it's very simple. You just let your app block all
it wants, and the deferred that results from deferToThread will fire
when it's done.

> Are any twisted people using WSGI?

Yes, I am. I use it to run Trac over Twisted.web2 at my
http://foss.eepatents.com site. You can see both an example of it
working and some pertinent code at
http://foss.eepatents.com/DynamicSite/. That is, by the way, one of
several virtual-host sites all running from a single instance of
DynamicSite, my same twisted.web2 HTTP server.

Good luck!

-Ed Suominen
Permalink | Reply |
headers
L. Daniel Burr | 5 Apr 2006 19:48
Picon

Re: Twisted and WSGI...

On Wed, 05 Apr 2006 14:27:30 -0400, m h <sesquile <at> gmail.com> wrote:

> Hi Folks-
>
> I'm new to this list.  (Am beginning to think more about learning
> twisted... for the third time ;) ).  I've got a question for you all
> and hope that you don't take it the wrong way.  It's not meant to be
> flame bait.
>
> As WSGI is going into python 2.5, I was wondering what the general
> consensus is among twisted web devs about it?  Does the WSGI model map
> well into twisted's asynchronous model?  Does twisted already have
> everything it needs, so WSGI doesn't bring much to the table?

In my opinion, no.  WSGI doesn't deal with the asynchronous model at
all.  The original author, Phillip Eby, noted that every time he tried
to figure out what async WSGI would look like, he ended up with something
like twisted or PEAK anyway.

In general, WSGI seems to have caught on with people who like threaded
servers.  I honestly see zero value in it if you are going to use an
integration toolkit like twisted.

> It appears that as AJAX becomes more popular then the asynchronous
> model might scale better with many small requests.  Say I have a WSGI
> app and I want to port it to twisted.  How easy is that?  (I know
> there is a twisted wsgi module)  Do I just need to wrap some logic in
> deferreds?
>

If you want to do AJAX-y things, then you want Athena, which runs on
twisted, and does even more than run-of-the-mill AJAX.  Google for
"comet" and you'll get an idea of the 2-way web functionality that
Athena can provide.

> Are any twisted people using WSGI?

Lord, I hope not.  The Zope 3 team runs their app-server on top of
twisted via WSGI, but they are the only ones I know of.

> Again, I'm just interested in hearing your constructive thoughts.
> Google didn't turn up much on the subject.
>
> thanks,
>
> -matt

Hope this helps,

Daniel
Permalink | Reply |

Gmane