John Nelson | 27 Jan 17:06 2011

How many databases are too many?

TLDR: Does CouchDB (conspiring with the OS?) handle thousands of databases
well or should I use the tried-and-true-but-aggravating mapping on
[group_id, ...] or is there a better pattern that I am unfamiliar with?

I'm building a web application that is group-oriented, by which I mean that
each user is interested in documents produced only by the group to which
they belong. There is no sharing of information across groups at all.

Since these groups are completely isolated, it seems logical to house each
one there own database but this makes the side of me familiar with RDBMS
very uncomfortable. I found one blog post by John Wood which suggested that
there "appears to be no “penalty” for hosting many databases within the same
CouchDB server." However, in my case, there would be roughly 1,000 - 5,000
groups. From what I understand, each database is stored as the data file
plus various index files. This would mean there are thousands of file
handles -- it seems like a bad idea.

Thanks in advance, Johnny
Zachary Zolton | 27 Jan 19:02 2011
Picon

Re: How many databases are too many?

Johnny,

CouchDB is known to handle thousands of databases per server, but
depending on your usage you may need to do some tweaks for the max
number of open databases/file descriptors:

http://wiki.apache.org/couchdb/Performance#Resource_Limits

Cheers,

Zach

On Thu, Jan 27, 2011 at 10:06 AM, John Nelson
<john.b.nelson@...> wrote:
> TLDR: Does CouchDB (conspiring with the OS?) handle thousands of databases
> well or should I use the tried-and-true-but-aggravating mapping on
> [group_id, ...] or is there a better pattern that I am unfamiliar with?
>
> I'm building a web application that is group-oriented, by which I mean that
> each user is interested in documents produced only by the group to which
> they belong. There is no sharing of information across groups at all.
>
> Since these groups are completely isolated, it seems logical to house each
> one there own database but this makes the side of me familiar with RDBMS
> very uncomfortable. I found one blog post by John Wood which suggested that
> there "appears to be no “penalty” for hosting many databases within the same
> CouchDB server." However, in my case, there would be roughly 1,000 - 5,000
> groups. From what I understand, each database is stored as the data file
> plus various index files. This would mean there are thousands of file
> handles -- it seems like a bad idea.
(Continue reading)

Mikeal Rogers | 27 Jan 19:22 2011
Picon

Re: How many databases are too many?

I know of one server that did over a million, of course not all were
highly active.

On Thursday, January 27, 2011, Zachary Zolton
<zachary.zolton@...> wrote:
> Johnny,
>
> CouchDB is known to handle thousands of databases per server, but
> depending on your usage you may need to do some tweaks for the max
> number of open databases/file descriptors:
>
> http://wiki.apache.org/couchdb/Performance#Resource_Limits
>
>
> Cheers,
>
> Zach
>
> On Thu, Jan 27, 2011 at 10:06 AM, John Nelson
> <john.b.nelson@...> wrote:
>> TLDR: Does CouchDB (conspiring with the OS?) handle thousands of databases
>> well or should I use the tried-and-true-but-aggravating mapping on
>> [group_id, ...] or is there a better pattern that I am unfamiliar with?
>>
>> I'm building a web application that is group-oriented, by which I mean that
>> each user is interested in documents produced only by the group to which
>> they belong. There is no sharing of information across groups at all.
>>
>> Since these groups are completely isolated, it seems logical to house each
>> one there own database but this makes the side of me familiar with RDBMS
(Continue reading)

John Nelson | 27 Jan 19:42 2011

Re: How many databases are too many?

Thanks, Mike and Zach!

That really makes the development of my app impressively easy and will
undoubtedly eliminate many  bugs that would arise from trying to enforce
isolation by other means.

On Thu, Jan 27, 2011 at 1:22 PM, Mikeal Rogers <mikeal.rogers@...>wrote:

> I know of one server that did over a million, of course not all were
> highly active.
>
> On Thursday, January 27, 2011, Zachary Zolton <zachary.zolton@...>
> wrote:
> > Johnny,
> >
> > CouchDB is known to handle thousands of databases per server, but
> > depending on your usage you may need to do some tweaks for the max
> > number of open databases/file descriptors:
> >
> > http://wiki.apache.org/couchdb/Performance#Resource_Limits
> >
> >
> > Cheers,
> >
> > Zach
> >
> > On Thu, Jan 27, 2011 at 10:06 AM, John Nelson
> > <john.b.nelson@...> wrote:
> >> TLDR: Does CouchDB (conspiring with the OS?) handle thousands of
> databases
(Continue reading)

Robert Newson | 27 Jan 19:32 2011
Picon

Re: How many databases are too many?

The point is that idle databases are free, barring the disk space they
consume. They'll be closed and paged out. It's the active set of
databases that matter hence the suggestion to look at your open file
descriptor limit, etc.

B.

On Thu, Jan 27, 2011 at 6:22 PM, Mikeal Rogers
<mikeal.rogers@...> wrote:
> I know of one server that did over a million, of course not all were
> highly active.
>
> On Thursday, January 27, 2011, Zachary Zolton
<zachary.zolton@...> wrote:
>> Johnny,
>>
>> CouchDB is known to handle thousands of databases per server, but
>> depending on your usage you may need to do some tweaks for the max
>> number of open databases/file descriptors:
>>
>> http://wiki.apache.org/couchdb/Performance#Resource_Limits
>>
>>
>> Cheers,
>>
>> Zach
>>
>> On Thu, Jan 27, 2011 at 10:06 AM, John Nelson
>> <john.b.nelson@...> wrote:
>>> TLDR: Does CouchDB (conspiring with the OS?) handle thousands of databases
(Continue reading)


Gmane