RE: Low pause GC for java 1.6

Bill,

As you know, it really depends on the size of your index combined with which features you're using.  There is
really no substitute for having a good load test and monitoring tool and to run multiple tests while trying
different settings.

My guess is that you're experiencing "full" gc's, even with CMS enabled.  This means either your tenured
("old") generation is too small or you have the "-XX:CMSInitiatingOccupancyFraction" set too high (it
starts the CMS too late and runs out of memory before it can finish).  We've found that some of the defaults
the JVM picks and/or the general advice out there doesn't apply to an app like Solr, which is just a
different kind of animal than the typical web frontend you might run in a J2ee container.

Below are the settings i am using as a starting point for our development Solr 4.0 app.  These may or may not
work for you but at least should give you a basic idea of how 1 other installation is configured.

Also, if you're using older grouping patches (I remember you worked on some of these), perhaps you're
hitting some of the scalibility problems that were predicted for some of these?  I'm pretty sure the GA
grouping features in 3.x solved these problems though.

Finally, you probably will get better responses on the users list than the dev list.  Also, other users might
benefit from other answers you get, so perhaps you could cross-post your question.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311

# Basic JVM settings.  I thing the 3g new generation size is bigger than you'd normally have with a typical web
app but for us it makes the old gen fill up slower and have fewer CMS gc's.  Minor ("parnew") gc's are still
fast enough for us, even with a biggish new gen.

Re: Low pause GC for java 1.6

Re: Low pause GC for java 1.6

Out of curiosity, how long does your Solr instance run before you hit a GC 
issue? Minutes, hours, days? If reasonably long, maybe you simply need to 
periodically take each Solr instance out of rotation and shut it down and 
restart the JVM before any of them hit the GC issue.

The first question is whether the GC delay is being caused because your JVM 
heap is too small (thrashing) or too large (long GC elapsed time.)

-- Jack Krupansky

-----Original Message----- 
From: Bill Bell
Sent: Saturday, June 30, 2012 8:48 PM
To: Bill Bell
Cc: dev <at> lucene.apache.org
Subject: Re: Low pause GC for java 1.6

Nothing?

Bill Bell
Sent from mobile

On Jun 29, 2012, at 9:09 PM, Bill Bell <billnbell <at> gmail.com> wrote:

> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>
> We have tried CMS. But we still have. 4 second wait on GC.
>
> What works well for Solr when using 16 GB of RAM?
>
> I have read lots of articles and now just looking for practical advise and 
> examples.
>
> Sent from my Mobile device
> 720-256-8076

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe <at> lucene.apache.org
For additional commands, e-mail: dev-help <at> lucene.apache.org 

Re: Low pause GC for java 1.6

On 6/30/2012 7:48 PM, Bill Bell wrote:
> Nothing?
>
> Bill Bell
> Sent from mobile
>
>
> On Jun 29, 2012, at 9:09 PM, Bill Bell <billnbell <at> gmail.com> wrote:
>
>> We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>>
>> We have tried CMS. But we still have. 4 second wait on GC.
>>
>> What works well for Solr when using 16 GB of RAM?
>>
>> I have read lots of articles and now just looking for practical advise and examples.

Bill, this is what I am using to minimize GC impact on Solr:

-Xms4096M
-Xmx8192M
-XX:NewRatio=1
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled

The key seems to be having a large chunk of memory for the new 
generation (eden), so that it's not forced to heavily use the old 
generation.  With a NewRatio of 1, half the memory gets used for the old 
generation, the other half is divided up among eden and survivor spaces, 
with eden getting most of it.  I don't know what the default is, but 
whatever it is resulted in a pretty small eden space, and I was doing a 
lot of full GCs.

As of this moment, Solr (3.5.0) and Jetty are using a little less than 
5GB of RAM.  At an uptime of 53190.8 seconds (14 hours), there were 229 
young garbage collections that took a total of 30.522 seconds.  There 
were 14 full garbage collections that took a total of 9.351 seconds.  
These numbers come from jstat with a 5000 millisecond interval.

jstat -gc -t -h12 <pid> 5000

I don't think you can specify a NewRatio below 1, so if you end up 
needing more eden space than that can give you, you can set an explicit 
NewSize instead.

Thanks,
Shawn

Re: Low pause GC for java 1.6

Which params do you use now?
Which ones have you tried?

Please move to user <at> 

Otis 
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 

----- Original Message -----
> From: Bill Bell <billnbell <at> gmail.com>
> To: Bill Bell <billnbell <at> gmail.com>
> Cc: "dev <at> lucene.apache.org" <dev <at> lucene.apache.org>
> Sent: Saturday, June 30, 2012 9:48 PM
> Subject: Re: Low pause GC for java 1.6
> 
> Nothing?
> 
> Bill Bell
> Sent from mobile
> 
> 
> On Jun 29, 2012, at 9:09 PM, Bill Bell <billnbell <at> gmail.com> wrote:
> 
>>  We are getting large Solr pauses on Java garbage collection in 1.6 Java.
>> 
>>  We have tried CMS. But we still have. 4 second wait on GC.
>> 
>>  What works well for Solr when using 16 GB of RAM?
>> 
>>  I have read lots of articles and now just looking for practical advise and 
> examples.
>> 
>>  Sent from my Mobile device
>>  720-256-8076
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe <at> lucene.apache.org
> For additional commands, e-mail: dev-help <at> lucene.apache.org
>