headers
Luke Melia | 6 May 2010 22:29
Favicon
Gravatar

Garbage collection outside of request cycle?

I've been analyzing our Unicorn-powered Rails app's performance, and have found that garbage collection
is a big factor in slow requests.

In the interest of avoiding those performance hits while handling requests, would it be possible to have a
unicorn worker run garbage collection after handling a request and before waiting for the next one? Would
this be a good idea?

Cheers,
Luke
_______________________________________________
Unicorn mailing list - mongrel-unicorn <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
Permalink | Reply |
headers
Eric Wong | 6 May 2010 22:57

Re: Garbage collection outside of request cycle?

Luke Melia <luke <at> lukemelia.com> wrote:
> I've been analyzing our Unicorn-powered Rails app's performance, and
> have found that garbage collection is a big factor in slow requests.
> 
> In the interest of avoiding those performance hits while handling
> requests, would it be possible to have a unicorn worker run garbage
> collection after handling a request and before waiting for the next
> one? Would this be a good idea?

Hi Luke,

I made this for one heavyweight app a while back.

I guess I should throw this into the examples section, but it won't be
the default since it hurts simpler applications that don't generate
much garbage.

==> big_app_gc.rb <==
# This shouldn't hurt overall performance as long as the server cluster
# is at <=50% CPU capacity, and improves the performance of most memory
# intensive requests.  This serves to improve _client-visible_
# performance (possibly at the cost of overall performance).
#
# We'll call GC after each request is been written out to the socket, so
# the client never sees the extra GC hit it. It's ideal to call the GC
# inside the HTTP server (vs middleware or hooks) since the stack is
# smaller at this point, so the GC will both be faster and more
# effective at releasing unused memory.
#
# This monkey patch is _only_ effective for applications that use a lot
(Continue reading)

Permalink | Reply |
headers
Eric Wong | 6 May 2010 23:12

Re: Garbage collection outside of request cycle?

Eric Wong <normalperson <at> yhbt.net> wrote:
> I guess I should throw this into the examples section

Added a few more comments, but the code is still the same:

http://unicorn.bogomips.org/examples/big_app_gc.rb
http://git.bogomips.org/cgit/unicorn.git/commit/?id=510a48dafc5f7e2cb618d785885395c79570821c

--

-- 
Eric Wong
_______________________________________________
Unicorn mailing list - mongrel-unicorn <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
Permalink | Reply |
headers
Luke Melia | 7 May 2010 01:17
Favicon
Gravatar

Re: Garbage collection outside of request cycle?

On May 6, 2010, at 4:57 PM, Eric Wong wrote:

> I made this for one heavyweight app a while back.
> 
> ==> big_app_gc.rb <==
> # This shouldn't hurt overall performance as long as the server cluster
> # is at <=50% CPU capacity, and improves the performance of most memory
> # intensive requests.  This serves to improve _client-visible_
> # performance (possibly at the cost of overall performance).

This is exactly the tradeoff I'm looking for. Our app has a large footprint.

Thanks for the quick response. I'll report back when I have some data.

Cheers,
Luke

_______________________________________________
Unicorn mailing list - mongrel-unicorn <at> rubyforge.org
http://rubyforge.org/mailman/listinfo/mongrel-unicorn
Do not quote signatures (like this one) or top post when replying
Permalink | Reply |
headers
Luke Melia | 13 May 2010 22:29
Favicon
Gravatar

Re: Garbage collection outside of request cycle?

> On May 6, 2010, at 4:57 PM, Eric Wong wrote:
> 
>> ==> big_app_gc.rb <==
>> # This shouldn't hurt overall performance as long as the server cluster
>> # is at <=50% CPU capacity, and improves the performance of most memory
>> # intensive requests.  This serves to improve _client-visible_
>> # performance (possibly at the cost of overall performance).

I thought the list might be interested in how this worked for us. I applied the patch to execute GC between
each request. I'm using NewRelic to measure the app. Prior to the patch our, we spent about 25% of our
aggregate time serving a request in GC and our application was running at around 20-30% CPU load. Our
running app shows up as using ~330MB of memory.

Applying the patch cut the time spent in GC time to nearly zero and as predicted CPU spiked.
Client-perceived responsiveness increased as well. Unfortunately, during our busiest time of the day,
CPU load got so high that nginx locked up, so we rolled back the patch.

I made a simple change to execute GC once every 5 requests and applied it again. Aggregate time spent in GC
reduced to out 10% of total request time. This resulted in a bout a 25% overall improvement in client
response time. Big win! CPU maxes out at about 80% with this configuration..

One other thing I did was force GC to execute before_fork, on the theory that with COW, we would want to fork in
the tidiest state possible. I have not measured this on it's own to evaluate it's impact.

Thanks again for the help and code on this, Eric.

Considering how useful this is, perhaps unicorn should have an after_request hook, to avoid the need to monkey-patch?

Cheers,
Luke
(Continue reading)

Permalink | Reply |
headers
Eric Wong | 14 May 2010 21:02

Re: Garbage collection outside of request cycle?

Luke Melia <luke <at> lukemelia.com> wrote:
> > On May 6, 2010, at 4:57 PM, Eric Wong wrote:
> > 
> >> ==> big_app_gc.rb <== # This shouldn't hurt overall performance as
> >> long as the server cluster # is at <=50% CPU capacity, and improves
> >> the performance of most memory # intensive requests.  This serves
> >> to improve _client-visible_ # performance (possibly at the cost of
> >> overall performance).
> 
> I thought the list might be interested in how this worked for us. I
> applied the patch to execute GC between each request. I'm using
> NewRelic to measure the app. Prior to the patch our, we spent about
> 25% of our aggregate time serving a request in GC and our application
> was running at around 20-30% CPU load. Our running app shows up as
> using ~330MB of memory.

Thanks for the feedback, Luke.

Was the original 30% CPU load during peak traffic or normal traffic?

> Applying the patch cut the time spent in GC time to nearly zero and as
> predicted CPU spiked. Client-perceived responsiveness increased as
> well. Unfortunately, during our busiest time of the day, CPU load got
> so high that nginx locked up, so we rolled back the patch.

Yikes, nginx locking up is rare.  It's worth investigating and fixing
that from the nginx side if you can reproduce it.

> I made a simple change to execute GC once every 5 requests and applied
> it again. Aggregate time spent in GC reduced to out 10% of total
(Continue reading)

Permalink | Reply |

Gmane