Gregory Collins | 5 Apr 11:05 2012
Picon

Re: thread killed

+haskell-cafe, oops

On Thu, Apr 5, 2012 at 11:04 AM, Gregory Collins <greg <at> gregorycollins.net> wrote:
On Wed, Apr 4, 2012 at 10:09 PM, tsuraan <tsuraan <at> gmail.com> wrote:
> It's hard to rule Snap timeouts out; try building snap-core with the
> "-fdebug" flag and running your app with "DEBUG=1", you'll get a spew of
> debugging output from Snap on stderr.

Heh, that was quite a spew.  I normally get the exceptions tens of MB
into files that are hundreds of MB, and I sometimes don't get them at
all, so printing out the entire request body was a bit slow :)  After
commenting out some of the more talkative debug statements, I got the
exception to happen, and it looks generally like this:

I think I might know what your problem is. You're accepting file uploads using handleMultipart, yes? Snap kills uploads that are going too slow, otherwise you would be vulnerable to slowloris (http://ha.ckers.org/slowloris/) DoS attacks. What's probably happening here is that you're doing slow work inside the "Iteratee IO a" handler you pass to that function, which makes Snap think the client is trickling bytes to you. If that's the case, either finish the iteratee more quickly and do the slow work back in the Snap handler (preferable), or disable the minimum upload rate guard (although that's not recommended on a server talking to the public internet.)

G
--
Gregory Collins <greg <at> gregorycollins.net>



--
Gregory Collins <greg <at> gregorycollins.net>
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Michael Snoyman | 5 Apr 11:41 2012

Re: thread killed

On Thu, Apr 5, 2012 at 12:05 PM, Gregory Collins
<greg <at> gregorycollins.net> wrote:
> +haskell-cafe, oops
>
> On Thu, Apr 5, 2012 at 11:04 AM, Gregory Collins <greg <at> gregorycollins.net>
> wrote:
>>
>> On Wed, Apr 4, 2012 at 10:09 PM, tsuraan <tsuraan <at> gmail.com> wrote:
>>>
>>> > It's hard to rule Snap timeouts out; try building snap-core with the
>>> > "-fdebug" flag and running your app with "DEBUG=1", you'll get a spew
>>> > of
>>> > debugging output from Snap on stderr.
>>>
>>> Heh, that was quite a spew.  I normally get the exceptions tens of MB
>>> into files that are hundreds of MB, and I sometimes don't get them at
>>> all, so printing out the entire request body was a bit slow :)  After
>>> commenting out some of the more talkative debug statements, I got the
>>> exception to happen, and it looks generally like this:
>>
>>
>> I think I might know what your problem is. You're accepting file uploads
>> using handleMultipart, yes? Snap kills uploads that are going too slow,
>> otherwise you would be vulnerable to slowloris
>> (http://ha.ckers.org/slowloris/) DoS attacks. What's probably happening here
>> is that you're doing slow work inside the "Iteratee IO a" handler you pass
>> to that function, which makes Snap think the client is trickling bytes to
>> you. If that's the case, either finish the iteratee more quickly and do the
>> slow work back in the Snap handler (preferable), or disable the minimum
>> upload rate guard (although that's not recommended on a server talking to
>> the public internet.)

Wouldn't it make more sense to pause the timeout handler when running
user code? That's what we do in Warp.

Michael
tsuraan | 5 Apr 18:35 2012
Picon

Re: thread killed

>> I think I might know what your problem is. You're accepting file uploads
>> using handleMultipart, yes? Snap kills uploads that are going too slow,
>> otherwise you would be vulnerable to slowloris
>> (http://ha.ckers.org/slowloris/) DoS attacks. What's probably happening here
>> is that you're doing slow work inside the "Iteratee IO a" handler you pass
>> to that function, which makes Snap think the client is trickling bytes to
>> you. If that's the case, either finish the iteratee more quickly and do the
>> slow work back in the Snap handler (preferable), or disable the minimum
>> upload rate guard (although that's not recommended on a server talking to
>> the public internet.)

I tried adding a "setMinimumUploadRate 0" to my handleMultipart and
doing the upload, and it's still getting killed.  The uploads are
pretty fast; a 150MB file takes around 10s.  I really don't think the
problem is with Snap, but pulling my code out of snap would be pretty
painful.  I have some pretty nasty crap going on with what I'm doing
anyhow, with threads communicating asynchronously, sockets being held
open between requests, that sort of horrid ugliness.  I'm going to try
to get rid of that, and then see if any bugs I had in there were
causing the problem.  Hopefully I'll be done with that by the end of
the weekend, and I'll either have the problem fixed, or have a
reasonable way to reproduce the errors.

Thanks for the idea, anyhow.  And, if setMinimumUploadRate 0 doesn't
actually disable the rate limiter, I'd be happy to try something more
correct.
tsuraan | 5 Apr 19:05 2012
Picon

Re: thread killed

>> I think I might know what your problem is. You're accepting file uploads
>> using handleMultipart, yes? Snap kills uploads that are going too slow,
>> otherwise you would be vulnerable to slowloris
>> (http://ha.ckers.org/slowloris/) DoS attacks. What's probably happening here
>> is that you're doing slow work inside the "Iteratee IO a" handler you pass
>> to that function, which makes Snap think the client is trickling bytes to
>> you. If that's the case, either finish the iteratee more quickly and do the
>> slow work back in the Snap handler (preferable), or disable the minimum
>> upload rate guard (although that's not recommended on a server talking to
>> the public internet.)

Ok, so I butchered Snap by replacing all of snap-server's killThread
calls with putStrLn calls, and the putStrLn that is triggered by
Snap.Internal.Http.Server.SimpleBackend's runSession (line 163 in
snap-server 0.8.0.1) seems to be the culprit.  Is that a rate limiter,
or is that something else?  Anyhow, I think there's a bug in there
somewhere.  I'll be poking at it a bit more, but that seems to be the
top-level source of the errors.

Gmane