Timo Sirainen | 16 Jan 17:18 2011
Picon
Picon

Smart IMAP proxying with imapc storage

I just committed a very early initial implementation of "imapc" storage
backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1

You can't really do anything except open INBOX and read mails from it,
so it's currently only intended for initial testing. It sucks in many
ways right now, but I'll be improving it.

The idea is that you could set for example:

mail_location = imapc:imap.gmail.com

And then Dovecot could act as a proxy to gmail. It won't actually work
currently with gmail, because there's no SSL support for outgoing
connections.

Currently index files are forcibly disabled, but it would be easy to
enable them, allowing Dovecot to do caching locally to improve
performance. In future perhaps it will also support caching message
headers/bodies to avoid unnecessary traffic.

Besides the mail_location setting, you'll need to also forward the
user's password to imap process in "pass" userdb extra field. How to do
that depends on what passdb/userdb you're using. While testing you could
simply set a static password:

plugin {
  pass = yourpassword
}

(Continue reading)

Ed W | 17 Jan 00:26 2011

Re: Smart IMAP proxying with imapc storage

On 16/01/2011 16:18, Timo Sirainen wrote:
> I just committed a very early initial implementation of "imapc" storage
> backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1

I know "me too" posts aren't that useful, but this does sound like a 
terrific direction.

Thanks for development

Ed W

Timo Sirainen | 28 Jan 19:17 2011
Picon
Picon

Re: Smart IMAP proxying with imapc storage

On Sun, 2011-01-16 at 18:18 +0200, Timo Sirainen wrote:

> I just committed a very early initial implementation of "imapc" storage
> backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1 

This is "fully functional" now, at least in theory. TODO:

 * SSL/TLS support to remote server
 * Don't read entire message bodies into a string in memory
 * Fetch more than one message at a time when it's known that more
messages are going to be fetched
 * Support locally stored indexes
 * Error handling could be better
 * Eventually make lib-storage API more asynchronous -> make imapc more
asynchronous

Some potential (future) uses for imapc include:

 * Locally caching IMAP proxy
  * Maybe even fully transparent proxy to client, so at office it would
be using Dovecot proxy while at home it would connect directly to remote
server. (A bit tricky with SSL connections, but possible I'd think.)
 * Filtering proxy that modifies messages on the fly
 * Fixing Exchange's (and everyone else's) IMAP support by being in
front of it
 * Could be used for sharing mailboxes across multiple Dovecot servers
without shared storage
 * Easy "perfect migration" from any IMAP server to Dovecot with dsync

(Continue reading)

Willie Gillespie | 28 Jan 19:32 2011

Re: Smart IMAP proxying with imapc storage

On 01/28/2011 11:17 AM, Timo Sirainen wrote:
> On Sun, 2011-01-16 at 18:18 +0200, Timo Sirainen wrote:
>
>> I just committed a very early initial implementation of "imapc" storage
>> backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1
>
> This is "fully functional" now, at least in theory. TODO:
>
>   * SSL/TLS support to remote server
>   * Don't read entire message bodies into a string in memory
>   * Fetch more than one message at a time when it's known that more
> messages are going to be fetched
>   * Support locally stored indexes
>   * Error handling could be better
>   * Eventually make lib-storage API more asynchronous ->  make imapc more
> asynchronous
>
> Some potential (future) uses for imapc include:
>
>   * Locally caching IMAP proxy
>    * Maybe even fully transparent proxy to client, so at office it would
> be using Dovecot proxy while at home it would connect directly to remote
> server. (A bit tricky with SSL connections, but possible I'd think.)
>   * Filtering proxy that modifies messages on the fly
>   * Fixing Exchange's (and everyone else's) IMAP support by being in
> front of it
>   * Could be used for sharing mailboxes across multiple Dovecot servers
> without shared storage
>   * Easy "perfect migration" from any IMAP server to Dovecot with dsync

(Continue reading)

Timo Sirainen | 31 Jan 03:05 2011
Picon
Picon

Re: Smart IMAP proxying with imapc storage

On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
> > I just committed a very early initial implementation of "imapc" storage
> > backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1 
> 
> This is "fully functional" now, at least in theory. TODO:
> 
>  * Don't read entire message bodies into a string in memory
>  * Fetch more than one message at a time when it's known that more
> messages are going to be fetched

The above features are done now. So the performance shouldn't be too
horrible anymore.

>  * SSL/TLS support to remote server
>  * Support locally stored indexes
>  * Error handling could be better
>  * Eventually make lib-storage API more asynchronous -> make imapc more
> asynchronous

These are not. Another TODO item is:

 * Add support for multiple connections to IMAP server. This is required
for e.g. THREAD command to work correctly.

Timo Sirainen | 31 Jan 18:30 2011
Picon
Picon

Re: Smart IMAP proxying with imapc storage

On Mon, 2011-01-31 at 04:05 +0200, Timo Sirainen wrote:
> On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
> > > I just committed a very early initial implementation of "imapc" storage
> > > backend to v2.1 hg: http://hg.dovecot.org/dovecot-2.1 
> > 
> >  * SSL/TLS support to remote server

Implemented. I tested with gmail:

mail_location = imapc:imap.gmail.com
plugin {
  pass = secret
  imapc_user = user <at> gmail.com
  imapc_port = 993
  imapc_ssl_ca_dir = /etc/ssl/certs

  # If you want to use STARTTLS (with port 143), set:
  #imapc_ssl_starttls = yes
}

So in above config /etc/ssl/certs/ contains all the valid CA certs. The
code requires a valid certificate to exist, but doesn't currently verify
that the name actually matches imap.gmail.com. I really need to put that
code there soon so I won't forget it..

Timo Sirainen | 1 Feb 18:20 2011
Picon
Picon

Re: Smart IMAP proxying with imapc storage

imapc settings have moved away from plugin {} section and mail_location.
Now instead use:

mail_location = imapc:

imapc_host = imap.example.com
#imapc_port = 143 # default

#imapc_user = %u # default
imapc_password = secret

imapc_ssl = no # or imaps or starttls
imapc_ssl_ca_dir = /etc/ssl/certs

Note the imapc_password change also. If passdb/userdb returned
userdb_pass/pass previously, return now instead
userdb_imapc_password/imapc_password.

Willie Gillespie | 1 Feb 20:25 2011

Re: Smart IMAP proxying with imapc storage

On 02/01/2011 10:20 AM, Timo Sirainen wrote:
> imapc settings have moved away from plugin {} section and mail_location.
>
> #imapc_user = %u # default
> imapc_password = secret

Is there a way to pass through a password like is done with the user name?

Timo Sirainen | 1 Feb 21:22 2011
Picon
Picon

Re: Smart IMAP proxying with imapc storage

On Tue, 2011-02-01 at 12:25 -0700, Willie Gillespie wrote:
> On 02/01/2011 10:20 AM, Timo Sirainen wrote:
> > imapc settings have moved away from plugin {} section and mail_location.
> >
> > #imapc_user = %u # default
> > imapc_password = secret
> 
> Is there a way to pass through a password like is done with the user name?

Depends on what passdb and userdb is used. With sql, ldap, passwd-file,
checkpassword it's possible by returning userdb_imapc_password=%w and
using userdb prefetch to pass it through. This works of course only with
plaintext authentication.

Mike Korizek | 6 Feb 00:23 2011
Picon

Re: Smart IMAP proxying with imapc storage

On 01/-10/-28163 08:59 PM, Timo Sirainen wrote:
> imapc settings have moved away from plugin {} section and mail_location.
> Now instead use:
> 
> mail_location = imapc:
> 
> imapc_host = imap.example.com
> #imapc_port = 143 # default
> 
> #imapc_user = %u # default
> imapc_password = secret
> 
> imapc_ssl = no # or imaps or starttls
> imapc_ssl_ca_dir = /etc/ssl/certs
> 
> Note the imapc_password change also. If passdb/userdb returned
> userdb_pass/pass previously, return now instead
> userdb_imapc_password/imapc_password.
I get the following error in the log:
Feb  6 00:17:44 hostname dovecot: auth:
static(user.name <at> domain.ch,127.0.0.1): No password
Feb  6 00:17:44 hostname dovecot: imap-login: Login:
user=<user.name <at> domain.ch>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1,
mpid=20283, secured
Feb  6 00:17:44 hostname dovecot: master: Error: service(imap): child
20283 killed with signal 11 (core dumps disabled)

dovecot -n
# 2.1.UNSTABLE (4e4c7f982fd5): /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-27-generic i686 Ubuntu 10.04.1 LTS
(Continue reading)

Timo Sirainen | 7 Feb 12:54 2011
Picon
Picon

Re: Smart IMAP proxying with imapc storage

On 6.2.2011, at 1.23, Mike Korizek wrote:

> Feb  6 00:17:44 hostname dovecot: master: Error: service(imap): child
> 20283 killed with signal 11 (core dumps disabled)
..
> # 2.1.UNSTABLE (4e4c7f982fd5): /usr/local/etc/dovecot/dovecot.conf

4e4c7f982fd5 was a buggy version that always crashed at startup. I fixed it a couple of days ago.

Chris Davies | 19 Feb 00:03 2011
Picon

Re: Smart IMAP proxying with imapc storage

Timo Sirainen <tss <at> iki.fi> wrote:
> * Fixing Exchange's (and everyone else's) IMAP support by being in
> front of it

Davmail makes a nice job of this. I see no reason why you couldn't
inspect its code, should you want to do so.

Chris

Timo Sirainen | 19 Feb 07:19 2011
Picon
Picon

Re: Smart IMAP proxying with imapc storage

On 19.2.2011, at 1.03, Chris Davies wrote:

> Timo Sirainen <tss <at> iki.fi> wrote:
>> * Fixing Exchange's (and everyone else's) IMAP support by being in
>> front of it
> 
> Davmail makes a nice job of this. I see no reason why you couldn't
> inspect its code, should you want to do so.

Interesting project. However, I find it doubtful that it can implement a fully compliant IMAP server in
just 1700 lines of code. A quick look shows that its LIST/LSUB handling code has been written to just handle
a couple of IMAP clients, not a full correct support. I don't see it handling FETCH BODY[HEADER.FIELDS (x y
z)] at all. Probably other things too.

Timo Sirainen | 28 Feb 17:20 2011
Picon
Picon

Re: Smart IMAP proxying with imapc storage

On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:

> Some potential (future) uses for imapc include:
> 
>  * Locally caching IMAP proxy
>   * Maybe even fully transparent proxy to client, so at office it would
> be using Dovecot proxy while at home it would connect directly to remote
> server. (A bit tricky with SSL connections, but possible I'd think.)
>  * Filtering proxy that modifies messages on the fly
>  * Fixing Exchange's (and everyone else's) IMAP support by being in
> front of it

Anyone have good ideas what this thing could be called? :) Calling it
"proxy" would confuse it with the dummy proxying feature of Dovecot. And
calling it "smart proxy" isn't very clear either what it does,
especially since I don't think it even is a proxy really.

So far the best I've come up with is a buzzwordy "IMAP accelerator".

Willie Gillespie | 28 Feb 17:31 2011

Re: Smart IMAP proxying with imapc storage

On 2/28/2011 9:20 AM, Timo Sirainen wrote:
> On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
>> Some potential (future) uses for imapc include:
>>
>>   * Locally caching IMAP proxy
>>    * Maybe even fully transparent proxy to client, so at office it would
>> be using Dovecot proxy while at home it would connect directly to remote
>> server. (A bit tricky with SSL connections, but possible I'd think.)
>>   * Filtering proxy that modifies messages on the fly
>>   * Fixing Exchange's (and everyone else's) IMAP support by being in
>> front of it
>
> Anyone have good ideas what this thing could be called? :) Calling it
> "proxy" would confuse it with the dummy proxying feature of Dovecot. And
> calling it "smart proxy" isn't very clear either what it does,
> especially since I don't think it even is a proxy really.
>
> So far the best I've come up with is a buzzwordy "IMAP accelerator".

Accelerator would work since a good use could be at the edge of a 
company branch network... especially if it had a "prefetch = yes" option 
and was given sufficient disk space.

Ed W | 28 Feb 18:56 2011

Re: Smart IMAP proxying with imapc storage

On 28/02/2011 16:20, Timo Sirainen wrote:
> On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
>
>> Some potential (future) uses for imapc include:
>>
>>   * Locally caching IMAP proxy
>>    * Maybe even fully transparent proxy to client, so at office it would
>> be using Dovecot proxy while at home it would connect directly to remote
>> server. (A bit tricky with SSL connections, but possible I'd think.)
>>   * Filtering proxy that modifies messages on the fly
>>   * Fixing Exchange's (and everyone else's) IMAP support by being in
>> front of it
> Anyone have good ideas what this thing could be called? :) Calling it
> "proxy" would confuse it with the dummy proxying feature of Dovecot. And
> calling it "smart proxy" isn't very clear either what it does,
> especially since I don't think it even is a proxy really.
>
> So far the best I've come up with is a buzzwordy "IMAP accelerator".
>

"Cache" is a good word for this kind of application?  Perhaps something 
like "IMAP Cache"?

Ed W

Charles Marcus | 28 Feb 21:23 2011

Re: Smart IMAP proxying with imapc storage

On 2011-02-28 11:20 AM, Timo Sirainen wrote:
> On Fri, 2011-01-28 at 20:17 +0200, Timo Sirainen wrote:
> 
>> Some potential (future) uses for imapc include:
>>
>>  * Locally caching IMAP proxy
>>   * Maybe even fully transparent proxy to client, so at office it would
>> be using Dovecot proxy while at home it would connect directly to remote
>> server. (A bit tricky with SSL connections, but possible I'd think.)
>>  * Filtering proxy that modifies messages on the fly
>>  * Fixing Exchange's (and everyone else's) IMAP support by being in
>> front of it
> 
> Anyone have good ideas what this thing could be called? :) Calling it
> "proxy" would confuse it with the dummy proxying feature of Dovecot. And
> calling it "smart proxy" isn't very clear either what it does,
> especially since I don't think it even is a proxy really.
> 
> So far the best I've come up with is a buzzwordy "IMAP accelerator".

IMAP collector? proxyfetch? proxyconnect?

--

-- 

Best regards,

Charles


Gmane