Arv | 18 Aug 00:50 2009
Picon

Maximum number of values in a multi-valued field.


All,
We are considering some new changes to our Solr schema to better support
some new functionality for our application. To that extent, we want to add
an additional field that is multi-valued, but will contain a large number of
values per document. Potentially up to 2000 values on this field per
document.

Questions:
- Is this wise?
- Though we will not be faceting on this field, are there any implications
for performance?
- I understand that the XML in/out will be large, and we may need to stop
this field being sent back on every query, as this field is essentially used
as a filter only.

The reason I am asking is that our instance of Solr currently works
wonderfully and is very fast, and I am wary of doing anything that will
affect this.  So, any pointer on design here will help.

-- Aravind

--

-- 
View this message in context: http://www.nabble.com/Maximum-number-of-values-in-a-multi-valued-field.-tp25015685p25015685.html
Sent from the Solr - User mailing list archive at Nabble.com.

Jason Rutherglen | 18 Aug 01:06 2009
Picon

Re: Maximum number of values in a multi-valued field.

Your term dictionary will grow somewhat, which means the term
index could consume more memory. Because the term dictionary has
grown there could be less performance in looking up terms but
that is unlikely to affect your application. How many unique
terms will there be?

On Mon, Aug 17, 2009 at 3:50 PM, Arv<aravind.naidu <at> vsc.ph> wrote:
>
> All,
> We are considering some new changes to our Solr schema to better support
> some new functionality for our application. To that extent, we want to add
> an additional field that is multi-valued, but will contain a large number of
> values per document. Potentially up to 2000 values on this field per
> document.
>
> Questions:
> - Is this wise?
> - Though we will not be faceting on this field, are there any implications
> for performance?
> - I understand that the XML in/out will be large, and we may need to stop
> this field being sent back on every query, as this field is essentially used
> as a filter only.
>
> The reason I am asking is that our instance of Solr currently works
> wonderfully and is very fast, and I am wary of doing anything that will
> affect this.  So, any pointer on design here will help.
>
> -- Aravind
>
> --
(Continue reading)

Aravind Naidu | 18 Aug 01:15 2009
Picon

Re: Maximum number of values in a multi-valued field.


Hi,
The possibility is that all items in this field could be unique. Let me
clarify.
The main Solr index is a for a list of products. Some products belong to
catalogues.  So, the consideration is to add a multi-valued field to put the
id of the catalogue in each product as a multi-valued field to be used as a
filter.

-- Aravind

Jason Rutherglen-2 wrote:
> 
> Your term dictionary will grow somewhat, which means the term
> index could consume more memory. Because the term dictionary has
> grown there could be less performance in looking up terms but
> that is unlikely to affect your application. How many unique
> terms will there be?
> 
> On Mon, Aug 17, 2009 at 3:50 PM, Arv<aravind.naidu <at> vsc.ph> wrote:
>>
>> All,
>> We are considering some new changes to our Solr schema to better support
>> some new functionality for our application. To that extent, we want to
>> add
>> an additional field that is multi-valued, but will contain a large number
>> of
>> values per document. Potentially up to 2000 values on this field per
>> document.
>>
(Continue reading)

Constantijn Visinescu | 18 Aug 12:15 2009
Picon

Re: Maximum number of values in a multi-valued field.

Hi,

This would also depend on the amount of documents you got in your solr
index.

10k ? 100k? 1m? 10m? 100m?

I'm by no means an expert on solr but i recently had a similar question and
to get my answer i grabbed a new blank solr index, modiefied my schema.xml,
reindexed all my data assuming the worst case.

Make up something like 100k (pick a number that seams realistic and multiply
by 10) different catalog names and assign 2000 to each document randomly.
Let your computer crunch bits for a few hours to rebuild an index (close to
10m documents in my case), copy the index to a server similar to your
production server and see what happens :)

For me there was no noticeable performance difference.

Constantijn Visinescu

On Tue, Aug 18, 2009 at 1:15 AM, Aravind Naidu <aravind.naidu <at> vsc.ph> wrote:

>
> Hi,
> The possibility is that all items in this field could be unique. Let me
> clarify.
> The main Solr index is a for a list of products. Some products belong to
> catalogues.  So, the consideration is to add a multi-valued field to put
> the
(Continue reading)

Shalin Shekhar Mangar | 18 Aug 22:52 2009
Picon

Re: Maximum number of values in a multi-valued field.

On Tue, Aug 18, 2009 at 4:20 AM, Arv <aravind.naidu <at> vsc.ph> wrote:

>
> All,
> We are considering some new changes to our Solr schema to better support
> some new functionality for our application. To that extent, we want to add
> an additional field that is multi-valued, but will contain a large number
> of
> values per document. Potentially up to 2000 values on this field per
> document.
>
> Questions:
> - Is this wise?

Depends :)

>
> - Though we will not be faceting on this field, are there any implications
> for performance?

Should be ok but benchmark it to be sure.

>
> - I understand that the XML in/out will be large, and we may need to stop
> this field being sent back on every query, as this field is essentially
> used
> as a filter only.
>

You can do that with the "fl" request parameter. If you don't need it back
(Continue reading)


Gmane