Re: preloading / "warming up" the index
Charles Mi <charlesmi <at> gmail.com>
2006-06-01 05:16:32 GMT
Is there a way to preload portions of the other files, particularly .tis,
.frq, .prx into memory? My total index size is roughly 4GB and we have 2GB
memory in the machine... the .tii file is tiny (about 1.5 MB). Basically,
before my server starts accepting and handling queries, I'd like to load as
much of the index into memory as possible, so that Lucene doesn't have to
always hit the disk for each unique keyword, which is an order of magnitude
slower... Does Lucene have this preloading feature?
If not, is there a way to approximate the preloading to get Lucene to cache
some of the disk data into memory? I tried to warm up Lucene by searching
for 1000 random terms... but that doesn't seem to work. After warming up,
searching for those same exact 1000 terms again was very fast....
but querying 1000 *other* terms was just as slow as the warm up query
On 5/31/06, Otis Gospodnetic <otis_gospodnetic <at> yahoo.com> wrote:
> Look in your index directory and look for a .tii file. That file is read
> in RAM (if there is enough of it. If there is not, you will see OOM). What
> Monsur was talking about is related to sorting and warming up of FieldCache
> instances. If you don't sort your results by criteria other than the
> default relevance, you can ignore FieldCache.
> Any query should cause Lucene to read the whole .tii in RAM.
> If you do not see a .tii file in your index directory, and instead see one
> or more .cfs file, you are using the compound index format. Run IndexReader
> as a java app (e.g. java org.apache.lucene....IndexReader/your/index/dir/file(?)) to get a listing
of individual index files inside a