headers
Andrew Tamm | 13 Jun 21:22

x-imapbase timestamp field being rewritten

I'm having a problem with various users having repeated downloads of the 
messages in their mailboxes.  Most, but not all, are using Mac Mail.app, 
although I'm not sure if this makes a difference.

The problem occurs intermittently, although it does seem to appear in 
clusters, i.e. multiple users will be affected at the same time.

For some reason, the X-IMAP field (in particular the first number, the 
time stamp) is being rewritten to the current time instead of its 
original value.  This causes all the messages to have new UIDL values, 
and so the POP client thinks that these are new messages.  The field in 
question is:

X-IMAP: 1213219525 0000000002

Although if the data is not stored in a placeholder message, the header 
is named X-IMAPbase.

Does anyone have any idea why that number would be changing?
_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
Permalink | Reply |
headers
Joel Reicher | 14 Jun 14:48

Re: x-imapbase timestamp field being rewritten

> I'm having a problem with various users having repeated downloads of the 
> messages in their mailboxes.  Most, but not all, are using Mac Mail.app, 
> although I'm not sure if this makes a difference.
> 
> The problem occurs intermittently, although it does seem to appear in 
> clusters, i.e. multiple users will be affected at the same time.
> 
> For some reason, the X-IMAP field (in particular the first number, the 

Are you using unix format mailboxes? Changing the format might fix this
problem, and will almost certainly make your server more efficient.

> time stamp) is being rewritten to the current time instead of its 
> original value.  This causes all the messages to have new UIDL values, 
> and so the POP client thinks that these are new messages.  The field in 

Is POP the only method of access to these mailboxes? Are none of these
Mac clients configured to use IMAP and you're not running a webmail
system either (which would almost certainly use IMAP)?

Maybe something else has accessed the mailbox in the middle of a user's
session; are the users getting any errors about sessions disconnecting?

> question is:
> 
> X-IMAP: 1213219525 0000000002
> 
> Although if the data is not stored in a placeholder message, the header 
> is named X-IMAPbase.
(Continue reading)

Permalink | Reply |
headers
Andrew Tamm | 15 Jun 23:19

Re: x-imapbase timestamp field being rewritten

Joel Reicher wrote:

> 
> Are you using unix format mailboxes? Changing the format might fix this
> problem, and will almost certainly make your server more efficient.
> 

yes, the mailboxes are in unix flat file format.  This was necessitated by some users still using
pine/mutt/mail to 
access their mail.

>> time stamp) is being rewritten to the current time instead of its 
>> original value.  This causes all the messages to have new UIDL values, 
>> and so the POP client thinks that these are new messages.  The field in 
> 
> Is POP the only method of access to these mailboxes? Are none of these
> Mac clients configured to use IMAP and you're not running a webmail
> system either (which would almost certainly use IMAP)?
> 

For the users in question, POP clients are the only means of access.  There are a few that have multiple
clients 
connecting and checking the mail, but there are several notable examples of people who have never used
command line 
programs and only have one POP client configued.

> Maybe something else has accessed the mailbox in the middle of a user's
> session; are the users getting any errors about sessions disconnecting?
> 
>> question is:
(Continue reading)

Permalink | Reply |
headers
Joel Reicher | 18 Jun 02:24

Re: x-imapbase timestamp field being rewritten

> >> question is:
> >>
> >> X-IMAP: 1213219525 0000000002
> >>
> >> Although if the data is not stored in a placeholder message, the header 
> >> is named X-IMAPbase.
> > 
> > By "placeholder message" do you mean what is described here?
> > 
> > http://www.washington.edu/imap/IMAP-FAQs/index.html#6.14
> > 
> > If so, then it is correct that this change over time.
> > 
> Yes, that is the "placeholder" message I was referring to.  In mailboxes that
>  have never been empty when the POP client 
> connected, the X-IMAP info is stored in the first message in the mailbox, whi
> ch seems to be moved to the next message 
> when that first message is deleted.

http://www.washington.edu/imap/IMAP-FAQs/index.html#6.15

> Why is that correct to change over time?  Do you mean the second field of the
>  X-IMAP header to change? It was my 
> understanding that the second field is a strict counter of the last assigned 
> X-UID number, and the first field was a 
> time stamp of when the "DO NOT DELETE" message was created.  The POP server i
> s using the two to create a single UIDL 
> (hex representations of both field concacted).  Wouldnt changing the first fi
> eld (which is what we are experiencing) 
> cause the same message to have two different UIDLs and thus confuse the clien
(Continue reading)

Permalink | Reply |
headers
Andrew Daviel | 4 Jul 21:00

FYI: procmail/dmail rule


FYI

We run procmail rules for server-side spam filtering, using dmail as the 
delivery agent for MIX format. Generally this works well.

I just realized that for years I've had the wrong rule.
I had
  :0
  |dmail
as the final rule to deliver to INBOX, but it should have been
  :0 w
  |dmail

If dmail failed, the mail was lost instead of being bounced or requeued.
As I found out when I fixed up a user's mailbox as root, leaving a 
.mixnnn file with the wrong ownership. Duh.

(still somewhat shocked by Mark's dismissal ... I hope imapd
can continue as an open source project. To say nothing of Mark's work 
with the IETF developing the imap standard, and the ongoing standards 
work for mobile devices)
Permalink | Reply |
headers
Jeroen van Aart | 7 Jul 20:37

Re: FYI: procmail/dmail rule

Andrew Daviel wrote:
> We run procmail rules for server-side spam filtering, using dmail as the 
> delivery agent for MIX format. Generally this works well.

Wouldn't spam filtering at the smtp level be more effective and cause 
less traffic? It's important to be able to reject a message asap, 
preferably before any data goes through (grey listing, rejecting certain 
IP-addresses, etc.). So you have the least possible waste of resources.

With procmail the spam email already has been accepted, handled by MTAs 
and is about to be delivered to the recipient, at which time it decides 
to reject it. That kills one of the main benefits of filtering, decrease 
of useless traffic.

Regards,
Jeroen

_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
Permalink | Reply |
headers
Andrew Daviel | 8 Jul 00:28

Re: FYI: procmail/dmail rule

On Mon, 7 Jul 2008, Jeroen van Aart wrote:

> Andrew Daviel wrote:
>> We run procmail rules for server-side spam filtering, using dmail as the 
>> delivery agent for MIX format. Generally this works well.
>
> Wouldn't spam filtering at the smtp level be more effective and cause less 
> traffic? It's important to be able to reject a message asap, preferably 
> before any data goes through (grey listing, rejecting certain IP-addresses, 
> etc.). So you have the least possible waste of resources.

Yes, we do that too - RBL at RCPT time plus spamassassin milter at DATA 
time. Currently it looks like 80% is rejected at RCPT, 10% at DATA, and
3% filtered on delivery, leaving 7% legitimate mail. The RBL 
rejected number seems on a linear rise over the last year :-(

Mail scored over one threshold is deemed "definitely spam" and rejected 
by SMTP DATA response.
Mail scored over a lower threshold is deemed "possible spam" and 
accepted, allowing users to filter it, adjust whitelists etc.
They can also have multiple inboxes for mailing lists etc. if they wish.
Mail scored below the low threshold is deemed "ham" (sic).

One problem with filtering up front is that messages must be scored 
before they are cloned, meaning that a user's personal whitelist cannot 
be applied if there is more than one recipient. Just in case we have some 
spam-lovers, or commercial newsletters that trip the threshold...
Permalink | Reply |
headers
Oscar del Rio | 14 Jun 16:56

Re: x-imapbase timestamp field being rewritten

Andrew Tamm wrote:

> For some reason, the X-IMAP field (in particular the first number, the 
> time stamp) is being rewritten to the current time instead of its 
> original value.  This causes all the messages to have new UIDL values, 
> and so the POP client thinks that these are new messages.  The field in 
> question is:
> 
> X-IMAP: 1213219525 0000000002

> Does anyone have any idea why that number would be changing?

SPAM with bogus headers?

We used to have this problem when using Unix mailbox format.
Every once in a while a SPAM comes with bogus headers
(X-UID, X-IMAPbase, etc) and the POP3/IMAP server has to
reindex all messages.

POP3 clients will see every message as new;
IMAP clients will reload all headers - they don't see duplicates but
a delay in "refreshing" the Inbox.

The fix is to either switch to mix format and/or filter out incoming
emails with bogus headers

Our procmail filters remove the following headers
^(X-UID|X-IMAPbase|X-IMAP|X-Status|Status|X-Keywords)
_______________________________________________
Imap-uw mailing list
(Continue reading)

Permalink | Reply |
headers
Joel Reicher | 18 Jun 11:14

Re: x-imapbase timestamp field being rewritten

> The fix is to either switch to mix format and/or filter out incoming
> emails with bogus headers
> 
> Our procmail filters remove the following headers
> ^(X-UID|X-IMAPbase|X-IMAP|X-Status|Status|X-Keywords)

Don't tmail and dmail take care of this by writing valid headers on
delivery, even to a UNIX-format maildrop?

tmail at least overwrites X-UIDs; I just tested that case.

Cheers,

	- Joel
_______________________________________________
Imap-uw mailing list
Imap-uw <at> u.washington.edu
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
Permalink | Reply |

Gmane