headers
Marten Lehmann | 8 Feb 2011 13:22
Picon

NoSQL Storage Backend

Hello,

I found a posting on the mailing list from 2008 "Adding new storage 
backend to Dovecot" while searching for POP3 and IMAP servers, that 
allow to use a NoSQL backend rather than a traditional file system.

The problem with traditional filesystems is, that they are very limited. 
You cannot easily extend it, even with RAID 10 all messages may be lost 
due to a RAID controller failure and just-in-time backups of a heavy 
loaded system with consistently changing messages folders are virtually 
impossible.

Modern object stores like Scality, CouchDB or MongoDB can be very handy, 
as they easily allow to keep an arbitrary amount of copies of each 
message on different servers, you can simply add a storage node and new 
storage is immediately available and mailboxes aren't bound to a certain 
server any longer, but messages are evenly distributed across all 
storage nodes, thus distributing load as well.

Since the original posting was dated 2008 I was in good hope, that 
someone worked on that already, but at least in 
dovecot-2.0.9/src/lib-storage I couldn't find anything relating to NoSQL 
message stores.

Isn't this option of big interest for every bigger mailbox provider? How 
hard would it be to develop? Would it be possible to get this developed 
on a paid basis by the dovecot core team?

Kind regards
Marten
(Continue reading)

Permalink | Reply |
headers
Timo Sirainen | 8 Feb 2011 18:27
Picon
Picon
Favicon

Re: NoSQL Storage Backend

On 8.2.2011, at 14.22, Marten Lehmann wrote:

> I found a posting on the mailing list from 2008 "Adding new storage backend to Dovecot" while searching for
POP3 and IMAP servers, that allow to use a NoSQL backend rather than a traditional file system.

I don't remember any mails about NoSQL backend from 2008, but at the end of 2009 it became something I started
planning on doing. It's listed in http://wiki2.dovecot.org/Roadmap

> Isn't this option of big interest for every bigger mailbox provider? How hard would it be to develop? Would
it be possible to get this developed on a paid basis by the dovecot core team?

It should be pretty easy to develop if you don't care about performance, but since you likely do, it gets more
difficult. The latencies for db lookups are higher than local reads, so it would be good to be able to do
multiple lookups in parallel. I've recently started imapc backend, which has similar problems and I was
planning on solving those problems with it, so it will be easier then for nosql backend. Also it would be
good to design a storage backend for nosql that didn't require any kind of locking.

One guy recently started looking into developing something for this, but I haven't heard from him for some
weeks. I could ask if he's gotten anywhere.. As for me, I'm not entirely sure about my plans for this spring.
There's another company wanting to pay me, but I don't know how much of my time they'll really need.
Permalink | Reply |
headers
Oliver Eales | 8 Feb 2011 14:07

Re: NoSQL Storage Backend

Am 08.02.2011 13:22, schrieb Marten Lehmann:
> Hello,
>
> I found a posting on the mailing list from 2008 "Adding new storage 
> backend to Dovecot" while searching for POP3 and IMAP servers, that 
> allow to use a NoSQL backend rather than a traditional file system.

There has been a recent mail from Timo regarding the future development 
plans for 2.1 where he mentioned a key/value storage backend.
http://dovecot.markmail.org/message/juecu7wm2nwkluji?q=plans+2.1
>
> Isn't this option of big interest for every bigger mailbox provider? 
> How hard would it be to develop? Would it be possible to get this 
> developed on a paid basis by the dovecot core team?
I am also very interested on seeing this implemented, maybe also on a 
paid basis....
I think the core team is Timo :-)
>
Regards,
Oliver
Permalink | Reply |
headers
Jan-Frode Myklebust | 8 Feb 2011 19:21

Re: NoSQL Storage Backend

On Tue, Feb 8, 2011 at 1:22 PM, Marten Lehmann <lehmann <at> cnm.de> wrote:

> Modern object stores like Scality, CouchDB or MongoDB can be very handy, as
> they easily allow to keep an arbitrary amount of copies of each message on
> different servers, you can simply add a storage node and new storage is
> immediately available and mailboxes aren't bound to a certain server any
> longer, but messages are evenly distributed across all storage nodes, thus
> distributing load as well.

The Scality webpages mentions they´ve developed storage connector for dovecot:

http://www.scality.com/storage-solutions/

whatever that means..

  -jf
Permalink | Reply |
headers
Charles Marcus | 8 Feb 2011 20:07

Re: NoSQL Storage Backend

On 2011-02-08 1:21 PM, Jan-Frode Myklebust wrote:
> The Scality webpages mentions they´ve developed storage connector for dovecot:
> 
> http://www.scality.com/storage-solutions/
> 
> whatever that means..

Not so great - they're linking to dovecot.COM instead of dovecot.ORG...

--

-- 

Best regards,

Charles
Permalink | Reply |
headers
Marten Lehmann | 10 Feb 2011 19:26
Picon

Re: NoSQL Storage Backend

>> The Scality webpages mentions they´ve developed storage connector for dovecot:
>>
>> http://www.scality.com/storage-solutions/
>>
>> whatever that means..
>
> Not so great - they're linking to dovecot.COM instead of dovecot.ORG...

I asked them about the dovecot support two days ago but still no reply...
Permalink | Reply |
headers
Marc Villemade | 10 Feb 2011 19:48
Favicon

Re: NoSQL Storage Backend

Hey Marten,

As a disclaimer, I work with Scality.

Thank you guys for letting us know about the typo in the URL for dovecot on the website.
I have tried to reply to the email that sent us the information to thank them, but it was bogus ;(

We have also seen your request for information. Our Account Manager from Germany will be in touch with you
very soon.

Best,

-Marc
 <at> mastachand
http://linkd.in/heve30

On Feb 10, 2011, at 10:26 AM, Marten Lehmann wrote:

>>> The Scality webpages mentions they´ve developed storage connector for dovecot:
>>> 
>>> http://www.scality.com/storage-solutions/
>>> 
>>> whatever that means..
>> 
>> Not so great - they're linking to dovecot.COM instead of dovecot.ORG...
> 
> I asked them about the dovecot support two days ago but still no reply...
Permalink | Reply |
headers
list | 10 Feb 2011 20:02
Favicon

Re: NoSQL Storage Backend

On Thu, 10 Feb 2011 10:48:56 -0800, Marc Villemade
<marc.villemade <at> scality.com> wrote:
> Hey Marten,
> 
> As a disclaimer, I work with Scality.
> 
> Thank you guys for letting us know about the typo in the URL for dovecot
> on the website.
> I have tried to reply to the email that sent us the information to thank
> them, but it was bogus ;(
> 
> We have also seen your request for information. Our Account Manager from
> Germany will be in touch with you very soon.
> 
> Best,
> 
> -Marc
>  <at> mastachand
> http://linkd.in/heve30
> 
> 
> 
> 
> On Feb 10, 2011, at 10:26 AM, Marten Lehmann wrote:
> 
>>>> The Scality webpages mentions they´ve developed storage connector for
>>>> dovecot:
>>>> 
>>>> http://www.scality.com/storage-solutions/
>>>>
(Continue reading)

Permalink | Reply |
headers
Marc Villemade | 10 Feb 2011 21:08
Favicon

Re: NoSQL Storage Backend

On Feb 10, 2011, at 11:02 AM, <list <at> airstreamcomm.net> <list <at> airstreamcomm.net> wrote:

> On Thu, 10 Feb 2011 10:48:56 -0800, Marc Villemade
> <marc.villemade <at> scality.com> wrote:
>> Hey Marten,
>> 
>> As a disclaimer, I work with Scality.
>> 
>> Thank you guys for letting us know about the typo in the URL for dovecot
[.....]
> 
> Marc,
> 
> That must have been us (jesusjone <at> fu.com?), sorry we just wanted to get
> the message over without any sales pitches or something similar coming
> back.  Glad to hear it's fixed.
> 
> Michael
> 

Hey There Michael,

Thanks for letting us know, and no worries about the email, i wasn't expecting an answer with such an email
addy, but felt like trying anyway to thank you, just in case :-D

Best,

-Marc
 <at> mastachand
http://linkd.in/heve30
(Continue reading)

Permalink | Reply |
headers
Giorgio Regni | 10 Feb 2011 12:05
Favicon
Gravatar

Re: NoSQL Storage Backend


Hi, full disclaimer, I am the CTO of Scality. 
We just fixed the link to dovecot.com to dovecot.org on our website, thanks
for pointing that out.

Indeed we have developed a connector to our object storage for dovecot 2.x,
this is aimed at very large scale deployments, millions on users and
petabytes of storage. Our next goal is to also distribute the metadata on
our RING platform, think of it as a nosql database put tailor made for
Dovecot.
This would enable stateless operations on a layer of dovecot IMAP/POP
frontends, although from a performance/caching point of view, a layer of
statefull load balancers would redirect the same user to one dovecot box,
until it dies and then transparently fails over to another box, without
having to move any data around.

As Timo pointed out, focusing on keeping a lightning fast user experience
makes the exercise more difficult as a naive DB layer implementation would
add a lot of latency. Locking is our enemy...

Best regards

Giorgio

tanstaafl_bh wrote:
> 
> On 2011-02-08 1:21 PM, Jan-Frode Myklebust wrote:
>> The Scality webpages mentions they´ve developed storage connector for
>> dovecot:
>> http://www.scality.com/storage-solutions/
(Continue reading)

Permalink | Reply |
headers
Robert Brockway | 8 Feb 2011 21:56

Re: NoSQL Storage Backend

On Tue, 8 Feb 2011, Marten Lehmann wrote:

> The problem with traditional filesystems is, that they are very limited. You 
> cannot easily extend it, even with RAID 10 all messages may be lost due to a 
> RAID controller failure and just-in-time backups of a heavy loaded system 
> with consistently changing messages folders are virtually impossible.

Hi Marten.  There are many options that allow you to snapshop an active 
filesystem and guarantee a consistent backup.  Eg LVM on Linux or many 
other OSes. WAFL on a NetApp, etc.

In my experience a lot of high end mail systems store mail on the 
filesystem in Maildir format.

> Modern object stores like Scality, CouchDB or MongoDB can be very handy, as 
> they easily allow to keep an arbitrary amount of copies of each message on

This is certainly an interesting idea.

> different servers, you can simply add a storage node and new storage is 
> immediately available and mailboxes aren't bound to a certain server any 
> longer, but messages are evenly distributed across all storage nodes, thus 
> distributing load as well.

Being able to easily rebalance load would be useful.  I've known of 
several well known companies with a lot of mail users where mail was 
stored on filesystems across many systems and significant imbalances 
developed over time.  There was no easy way to rebalance them without 
causing an outage for at least some users.
(Continue reading)

Permalink | Reply |
headers
Marc Perkel | 14 Feb 2011 18:46

Re: NoSQL Storage Backend

The way I do MySQL backups is that I do a slave on another server. When 
I need to backup I stop the slave - back it up - then start it again.

On 2/8/2011 12:56 PM, Robert Brockway wrote:
> On Tue, 8 Feb 2011, Marten Lehmann wrote:
>
>> The problem with traditional filesystems is, that they are very 
>> limited. You cannot easily extend it, even with RAID 10 all messages 
>> may be lost due to a RAID controller failure and just-in-time backups 
>> of a heavy loaded system with consistently changing messages folders 
>> are virtually impossible.
>
> Hi Marten.  There are many options that allow you to snapshop an 
> active filesystem and guarantee a consistent backup.  Eg LVM on Linux 
> or many other OSes. WAFL on a NetApp, etc.
>
> In my experience a lot of high end mail systems store mail on the 
> filesystem in Maildir format.
>
>> Modern object stores like Scality, CouchDB or MongoDB can be very 
>> handy, as they easily allow to keep an arbitrary amount of copies of 
>> each message on
>
> This is certainly an interesting idea.
>
>> different servers, you can simply add a storage node and new storage 
>> is immediately available and mailboxes aren't bound to a certain 
>> server any longer, but messages are evenly distributed across all 
>> storage nodes, thus distributing load as well.
>
(Continue reading)

Permalink | Reply |

Gmane