headers
George Michaelson | 3 Jul 2009 01:05
Picon
Favicon

IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

I would like to suggest that IMAP extensions could usefully be  
specified to permit MUA to signal to the MS over IMAP, that the  
message in question is, or is not spam. I would also like to suggest  
that an IMAP extension pair to signal that the specified address is to  
be white, or black listed, would be equally useful.

This is because a large number of people now depend on IMAP backed MS  
in places like google, or corporate maildrops, but also have in-client  
spam tagging. Where the spam filtering is not sufficiently complete to  
prevent significant leakage, and mis-attribution across the MUA/MS  
dialogue, the lack of an in-protocol method to pass the information  
means we are all using what I would call ad-hoc methods to fix this  
situation up.

If it was possible to communicate this 'first class' in IMAP, I think  
that it would significantly enhance the user experience.

As an example, at the moment if I wish to inform google that I have  
known spam in local folders, I have to go to the web interface and  
manually tag. If there was an IMAP extension, I could review my local  
baysian junk folder, remove all non-spam (and flag the senders as  
white-listed if need be), and request the rest to be flagged as spam  
back on the IMAP backed MS.

Likewise for false positives and blacklisted senders.

I am told that the IMAPEXT WG is closed. I mail this, in the hope that  
somebody can help decide if this is a viable topic for consideration.

Many thanks for guidance and advice thus far from Lisa Dusseault.
(Continue reading)

Permalink | Reply |
headers
Iljitsch van Beijnum | 5 Jul 2009 21:03
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On 3 jul 2009, at 1:05, George Michaelson wrote:

> As an example, at the moment if I wish to inform google that I have  
> known spam in local folders, I have to go to the web interface and  
> manually tag. If there was an IMAP extension, I could review my  
> local baysian junk folder, remove all non-spam (and flag the senders  
> as white-listed if need be), and request the rest to be flagged as  
> spam back on the IMAP backed MS.

Doesn't moving the spam messages to the spam folder accomplish this  
already? That's what my client does.

But if you want this, I'd say that it needs to be a fractional thing,  
not a binary spam/no spam indication. For instance, the server could  
give something a spam score of 2 and the client also 2 and together  
that would be 4 so the message is presumed to be spam (assuming the  
spam threshold is 3), but in a binary system no spam OR no spam = no  
spam.
Permalink | Reply |
headers
Ned Freed | 6 Jul 2009 02:40

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


> On 3 jul 2009, at 1:05, George Michaelson wrote:

> > As an example, at the moment if I wish to inform google that I have
> > known spam in local folders, I have to go to the web interface and
> > manually tag. If there was an IMAP extension, I could review my
> > local baysian junk folder, remove all non-spam (and flag the senders
> > as white-listed if need be), and request the rest to be flagged as
> > spam back on the IMAP backed MS.

> Doesn't moving the spam messages to the spam folder accomplish this
> already? That's what my client does.

Please explain how moving a message to a random, arbtirarily named folder tells
the server anything useful. People use all sorts of different naming schemes
for folders, and it's common for clients to provide lots of flexibility in this
area.

The obvious way to make this work in IMAP is with a standardized folder
annotation saying "messages put in this folder were flagged as spam". Then
moves of messages to the folder with this annotation provide the server with
the necessary information that this message is considered to be spam by the
user. And by the same token, moving a message from this folder can be construed
as the message not being considered spam after all.

It isn't perfect, but as long as this is taken as an indicator, not as an
absolute statement, it should get the job done.

> But if you want this, I'd say that it needs to be a fractional thing,
> not a binary spam/no spam indication. For instance, the server could
(Continue reading)

Permalink | Reply |
headers
Arnt Gulbrandsen | 6 Jul 2009 19:50
Picon
Favicon
Gravatar

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Ned Freed writes:
> The obvious way to make this work in IMAP is with a standardized 
> folder annotation saying "messages put in this folder were flagged as 
> spam".

I think what Thunderbird and Apple Mail do is to set specific message 
flags on messages in whatever folders they happen to be in: $Junk if 
the user said a particular message is spam, $autojunk if tb 
autonomously decided that message is spam, $nonjunk and $autononjunk 
for the opposite. (I may be wrong about the exact flag names.)

This was specified in some expired draft, which IMO now has been frozen 
by deployment. Apple Mail and Thunderbird add up to >10% for me. Time 
for an informational RFC, perhaps. "Widely-used IMAP flag names"?

Arnt
Permalink | Reply |
headers
Lyndon Nerenberg | 6 Jul 2009 19:53
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> The obvious way to make this work in IMAP is with a standardized
> folder
> annotation saying "messages put in this folder were flagged as spam".
> Then
> moves of messages to the folder with this annotation provide the
> server with
> the necessary information that this message is considered to be spam
> by the
> user. And by the same token, moving a message from this folder can be
> construed
> as the message not being considered spam after all.

This is just trying to turn a folder into a keyword, but without full
keyword semantics. The most obvious flaw is that it constrains the
client to storing all the spam in one bucket. It doesn't allow for (say)
sorting junk by type into different folders (perhaps on different
servers). A message's 'state of spamminess' is an attribute of the
message, properly denoted by keywords that stick to (and with) the
message.

If you want this functionality (and I think it's a useful service for
servers to provide) you need to implement two keywords: $spam and
$nospam (these are just example names). Setting $spam on a message would
instruct the server to pass the message off to the filtering component
as spam; setting $nospam would instruct the server to pass the message
off to the filtering component as valid (non-spam). The two keywords are
exclusive, and it would be up to the server to decide what action to
take if the client tries to light up both at once (e.g. unset the other
flag and take the corresponding action, reject the command, or silently
(Continue reading)

Permalink | Reply |
headers
Nicolas Williams | 6 Jul 2009 19:44
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Sun, Jul 05, 2009 at 05:40:31PM -0700, Ned Freed wrote:
> 
> >On 3 jul 2009, at 1:05, George Michaelson wrote:
> 
> >> As an example, at the moment if I wish to inform google that I have
> >> known spam in local folders, I have to go to the web interface and
> >> manually tag. If there was an IMAP extension, I could review my
> >> local baysian junk folder, remove all non-spam (and flag the senders
> >> as white-listed if need be), and request the rest to be flagged as
> >> spam back on the IMAP backed MS.
> 
> >Doesn't moving the spam messages to the spam folder accomplish this
> >already? That's what my client does.
> 
> Please explain how moving a message to a random, arbtirarily named
> folder tells the server anything useful. People use all sorts of
> different naming schemes for folders, and it's common for clients to
> provide lots of flexibility in this area.

In particular the word 'spam' is really an English-language word.  L10N
would demand that the name of this folder not be 'spam' in all locales,
or at least that clients know how to map that folder name to the user's
language(s).  But even if we go for the latter, having special folder
names means denying or complicating the use of those names in other
languages for other purposes; it's bad enough we have one well-known
folder name (INBOX), let's have no more.

> The obvious way to make this work in IMAP is with a standardized
> folder annotation saying "messages put in this folder were flagged as
> spam". Then moves of messages to the folder with this annotation
(Continue reading)

Permalink | Reply |
headers
Michael Kirkham | 6 Jul 2009 19:26
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On Sun, 5 Jul 2009, Ned Freed wrote:

> Please explain how moving a message to a random, arbtirarily named 
> folder tells the server anything useful.

Seems to me this is up to the server configuration and software running on 
the server.  I don't see why you need a specific contract between IMAP 
client and server for flagging spam, and it's of limited usefulness.  For 
one thing, it was pretty painless for me to set up cron jobs to suck in 
whatever messages I save to certain folders (that could have been named 
anything) for training my spam filter.  I also have other cron jobs set up 
to do other things with email completely unrelated to spam that also 
simply involve saving to a particular folder (e.g. a tickler file for 
email--I move messages to folders named for a month or day, and cron jobs 
automatiacally reinsert them back in my inbox later).  That kind of thing 
isn't addressed by having something spam-specific, but as far as my email 
client is concerned, they're flagged the same way.

Any additional detail or instructions could likely be provided by adding 
headers to the message being saved, if the client allows it.

Just my two (not seeing the point) cents.

--
Michael Kirkham
President & CEO
Muonics, Inc.
http://www.muonics.com/
(Continue reading)

Permalink | Reply |
headers
Tony Finch | 6 Jul 2009 22:25
Picon
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Mon, 6 Jul 2009, Michael Kirkham wrote:
> On Sun, 5 Jul 2009, Ned Freed wrote:
>
> > Please explain how moving a message to a random, arbtirarily named folder
> > tells the server anything useful.
>
> Seems to me this is up to the server configuration and software running on the
> server.  I don't see why you need a specific contract between IMAP client and
> server for flagging spam

So that clients can have user interfaces that expose the functionality
in a more friendly manner.

Tony.
--

-- 
f.anthony.n.finch  <dot <at> dotat.at>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.
Permalink | Reply |
headers
Nicolas Williams | 6 Jul 2009 22:31
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Mon, Jul 06, 2009 at 09:25:32PM +0100, Tony Finch wrote:
> On Mon, 6 Jul 2009, Michael Kirkham wrote:
> > On Sun, 5 Jul 2009, Ned Freed wrote:
> >
> > > Please explain how moving a message to a random, arbtirarily named folder
> > > tells the server anything useful.
> >
> > Seems to me this is up to the server configuration and software running on the
> > server.  I don't see why you need a specific contract between IMAP client and
> > server for flagging spam
> 
> So that clients can have user interfaces that expose the functionality
> in a more friendly manner.

More specifically: so that users and their clients don't need to know
specific details of the server's configuration.
Permalink | Reply |
headers
Ned Freed | 6 Jul 2009 20:07

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


> On Sun, 5 Jul 2009, Ned Freed wrote:

> > Please explain how moving a message to a random, arbtirarily named
> > folder tells the server anything useful.

> Seems to me this is up to the server configuration and software running on
> the server.  I don't see why you need a specific contract between IMAP
> client and server for flagging spam,

The primary reason is so that the server can use the information to
automaticallly refine it's filtering policies on a per-user basis.

Other uses include, but are not limited to, setting expiration and arhival
policies.

> and it's of limited usefulness.

On the contrary, such a mechanism would be HUGELY useful. We have any number of
customers begging for it, as a matter of fact.

> For
> one thing, it was pretty painless for me to set up cron jobs to suck in
> whatever messages I save to certain folders (that could have been named
> anything) for training my spam filter.

Are you serious? Cron jobs? Do you have any idea how minute a fraction of the
users of email even know what a cron job is, let alone know how to set one up?

The minute you start talking about setting up your own cron jobs, it  pretty
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 6 Jul 2009 20:07

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


> On Sun, 5 Jul 2009, Ned Freed wrote:

> > Please explain how moving a message to a random, arbtirarily named
> > folder tells the server anything useful.

> Seems to me this is up to the server configuration and software running on
> the server.  I don't see why you need a specific contract between IMAP
> client and server for flagging spam,

The primary reason is so that the server can use the information to
automaticallly refine it's filtering policies on a per-user basis.

Other uses include, but are not limited to, setting expiration and arhival
policies.

> and it's of limited usefulness.

On the contrary, such a mechanism would be HUGELY useful. We have any number of
customers begging for it, as a matter of fact.

> For
> one thing, it was pretty painless for me to set up cron jobs to suck in
> whatever messages I save to certain folders (that could have been named
> anything) for training my spam filter.

Are you serious? Cron jobs? Do you have any idea how minute a fraction of the
users of email even know what a cron job is, let alone know how to set one up?

The minute you start talking about setting up your own cron jobs, it  pretty
(Continue reading)

Permalink | Reply |
headers
Iljitsch van Beijnum | 6 Jul 2009 21:55
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On 6 jul 2009, at 20:07, Ned Freed wrote:

>> it was pretty painless for me to set up cron jobs to suck in
>> whatever messages I save to certain folders (that could have been  
>> named
>> anything) for training my spam filter.

> Are you serious? Cron jobs? Do you have any idea how minute a  
> fraction of the
> users of email even know what a cron job is, let alone know how to  
> set one up?

Hey, everyone needs a hobby.

I have my procmail set up to check whether I have an IMAP session with  
the server. If so, I let the client do the mail filtering, if not, the  
server does it. That way, the unread counters for folders get updated  
when the client is connected, but the client doesn't have to move  
large volume mailing list traffic to the correct folder, costing me  
expensive wireless data traffic and/or slowing things down.

About folders for spam: Apple's Mail lets the user select "use this  
folder for... {sent, drafts, junk, trash}, which is nice if you used a  
different client before that uses a different folder naming scheme for  
these things. Works well enough although a little more automation  
wouldn't hurt.

The point that working with folders can be done with existing software  
(I later train spamassassin with what Mail put in the "junk" folder,  
even though SA doesn't know about IMAP) is an important one. Requiring
(Continue reading)

Permalink | Reply |
headers
Michael Kirkham | 6 Jul 2009 20:56
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On Mon, 6 Jul 2009, Ned Freed wrote:

> The primary reason is so that the server can use the information to
> automaticallly refine it's filtering policies on a per-user basis.

Which is what my setup does (assuming I use the folders I set aside for 
flagging spam/ham).

> Other uses include, but are not limited to, setting expiration and arhival
> policies.

But I don't see what these things have to do with IMAP specifically. 
Consider Pine, for example, which saves a special message (which Pine 
hides from the user) to folders it interacts with, having some 
Pine-specific details about the folder/messages in it.  If what you want 
is to be able to set folder-specific policies, perhaps what you should be 
arguing for is standardizing an extension to the mbox (or other mail 
storage) format that does something similar.  A standard set of header 
fields in this hidden message, plus whatever client-specific stuff some 
other client might want to add, etc.

> On the contrary, such a mechanism would be HUGELY useful. We have any 
> number of customers begging for it, as a matter of fact.

I think you misunderstood what I was pointing at not being useful.  If 
these customers don't understand what a cron job is, they probably haven't 
much clue about IMAP either, other than plugging in the settings their IT 
person tells them to so their email just works.
(Continue reading)

Permalink | Reply |
headers
Iljitsch van Beijnum | 6 Jul 2009 21:55
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On 6 jul 2009, at 20:07, Ned Freed wrote:

>> it was pretty painless for me to set up cron jobs to suck in
>> whatever messages I save to certain folders (that could have been  
>> named
>> anything) for training my spam filter.

> Are you serious? Cron jobs? Do you have any idea how minute a  
> fraction of the
> users of email even know what a cron job is, let alone know how to  
> set one up?

Hey, everyone needs a hobby.

I have my procmail set up to check whether I have an IMAP session with  
the server. If so, I let the client do the mail filtering, if not, the  
server does it. That way, the unread counters for folders get updated  
when the client is connected, but the client doesn't have to move  
large volume mailing list traffic to the correct folder, costing me  
expensive wireless data traffic and/or slowing things down.

About folders for spam: Apple's Mail lets the user select "use this  
folder for... {sent, drafts, junk, trash}, which is nice if you used a  
different client before that uses a different folder naming scheme for  
these things. Works well enough although a little more automation  
wouldn't hurt.

The point that working with folders can be done with existing software  
(I later train spamassassin with what Mail put in the "junk" folder,
(Continue reading)

Permalink | Reply |
headers
Alexey Melnikov | 6 Jul 2009 23:06
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Iljitsch van Beijnum wrote:
 [...]

> About folders for spam: Apple's Mail lets the user select "use this  
> folder for... {sent, drafts, junk, trash}, which is nice if you used 
> a  different client before that uses a different folder naming scheme 
> for  these things. Works well enough although a little more 
> automation  wouldn't hurt.
>
> The point that working with folders can be done with existing 
> software  (I later train spamassassin with what Mail put in the "junk" 
> folder,  even though SA doesn't know about IMAP) is an important one. 
> Requiring  keywords that are hardcoded in clients and servers means 
> deployment  will be extremely slow, if it ever happens.

I don't think your argument in favour of folders (mailboxes) is valid. 
An IMAP server don't need to treat $Junk/$NotJunk any specially, it just 
needs to be able to store arbitrary IMAP keywords. Most servers already do.
A recent Thunderbird can also be configured to set arbitrary IMAP 
keywords. (It also recognized $Junk/$NonJunk natively, but this is 
besides my point).
Permalink | Reply |
headers
Alexey Melnikov | 6 Jul 2009 23:06
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


Iljitsch van Beijnum wrote:
 [...]

> About folders for spam: Apple's Mail lets the user select "use this  
> folder for... {sent, drafts, junk, trash}, which is nice if you used 
> a  different client before that uses a different folder naming scheme 
> for  these things. Works well enough although a little more 
> automation  wouldn't hurt.
>
> The point that working with folders can be done with existing 
> software  (I later train spamassassin with what Mail put in the "junk" 
> folder,  even though SA doesn't know about IMAP) is an important one. 
> Requiring  keywords that are hardcoded in clients and servers means 
> deployment  will be extremely slow, if it ever happens.

I don't think your argument in favour of folders (mailboxes) is valid. 
An IMAP server don't need to treat $Junk/$NotJunk any specially, it just 
needs to be able to store arbitrary IMAP keywords. Most servers already do.
A recent Thunderbird can also be configured to set arbitrary IMAP 
keywords. (It also recognized $Junk/$NonJunk natively, but this is 
besides my point).
Permalink | Reply |
headers
Lyndon Nerenberg | 6 Jul 2009 19:53
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> The obvious way to make this work in IMAP is with a standardized
> folder
> annotation saying "messages put in this folder were flagged as spam".
> Then
> moves of messages to the folder with this annotation provide the
> server with
> the necessary information that this message is considered to be spam
> by the
> user. And by the same token, moving a message from this folder can be
> construed
> as the message not being considered spam after all.

This is just trying to turn a folder into a keyword, but without full
keyword semantics. The most obvious flaw is that it constrains the
client to storing all the spam in one bucket. It doesn't allow for (say)
sorting junk by type into different folders (perhaps on different
servers). A message's 'state of spamminess' is an attribute of the
message, properly denoted by keywords that stick to (and with) the
message.

If you want this functionality (and I think it's a useful service for
servers to provide) you need to implement two keywords: $spam and
$nospam (these are just example names). Setting $spam on a message would
instruct the server to pass the message off to the filtering component
as spam; setting $nospam would instruct the server to pass the message
off to the filtering component as valid (non-spam). The two keywords are
exclusive, and it would be up to the server to decide what action to
take if the client tries to light up both at once (e.g. unset the other
(Continue reading)

Permalink | Reply |
headers
Iljitsch van Beijnum | 7 Jul 2009 09:21
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On 6 jul 2009, at 19:53, Lyndon Nerenberg wrote:

> A message's 'state of spamminess' is an attribute of the
> message, properly denoted by keywords that stick to (and with) the
> message.

That's nice in the abstract, but why do we have servers classify  
messages as spam to begin with? In order to not have to look at them,  
or to be able to look at them when it suits us to clean out the spam  
rather than in between mail that we actually wanted to receive.

So moving spam to a separate folder is not something that happens  
because of lack of a better mechanism, but the thing that we want to  
happen 98% of the time anyway.

> $spam and $nospam

There are more states than that:

1. automatically flagged as spam
2. automatically flagged as non-spam
3. manually flagged as spam
4. manually flagged as non-spam

Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3  
should trigger retraining of baysian filters etc.
Permalink | Reply |
headers
Lyndon Nerenberg | 7 Jul 2009 19:31
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Tue, 2009-07-07 at 09:21 +0200, Iljitsch van Beijnum wrote:
> 3. manually flagged as spam
> 4. manually flagged as non-spam
> 
> Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3  
> should trigger retraining of baysian filters etc.

The keywords I'm describing are intended to tell the server to process
the flagged message through its filters (i.e. train on the message). It
isn't relevant to the proposal whether a human sets them, just what the
server's behaviour is in their presence.

--lyndon
Permalink | Reply |
headers
Ned Freed | 7 Jul 2009 15:54

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

> On 6 jul 2009, at 19:53, Lyndon Nerenberg wrote:

> > A message's 'state of spamminess' is an attribute of the
> > message, properly denoted by keywords that stick to (and with) the
> > message.

> That's nice in the abstract, but why do we have servers classify
> messages as spam to begin with? In order to not have to look at them,
> or to be able to look at them when it suits us to clean out the spam
> rather than in between mail that we actually wanted to receive.

Actually, the primary reason servers classify messages as spam is so they
can avoid having to deal with them. If the overhead could be ignored no
server operator would bother - spam filteringh is expensive and difficult.
The reason it's done is the alternate of receiving and dealing with the
those spam messages, with all the ramificaations, is even more expensive
and difficult.

This more than anything is why a score and not a yes/no verdict is important. A
score lets you do stuff like reject messages clearly found to be spam at the
SMTP level, while letting likely but less certain to be spam messages through
to specially designated spam folders or whatever.

> So moving spam to a separate folder is not something that happens
> because of lack of a better mechanism, but the thing that we want to
> happen 98% of the time anyway.

Sometimes yes, sometimes no.

> > $spam and $nospam
(Continue reading)

Permalink | Reply |
headers
Dave Cridland | 6 Jul 2009 22:16

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Mon Jul  6 18:53:41 2009, Lyndon Nerenberg wrote:
> On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> > The obvious way to make this work in IMAP is with a standardized
> > folder
> > annotation saying "messages put in this folder were flagged as  
> spam".
> > Then
> > moves of messages to the folder with this annotation provide the
> > server with
> > the necessary information that this message is considered to be  
> spam
> > by the
> > user. And by the same token, moving a message from this folder  
> can be
> > construed
> > as the message not being considered spam after all.
> 
> This is just trying to turn a folder into a keyword, but without  
> full
> keyword semantics.

+1.

It's fundamentally no different from clients trying to replace the  
\Deleted keyword with a Trash mailbox, and has all the related  
issues, which we presumably all know. (Or else can find the attempt  
at documenting them I once tried).

This is compounded by the state being a tristate rather than binary -  
messages are "Ham", or "Spam", or else "The user has not expressed
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 6 Jul 2009 20:00

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

> On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> > The obvious way to make this work in IMAP is with a standardized
> > folder
> > annotation saying "messages put in this folder were flagged as spam".
> > Then
> > moves of messages to the folder with this annotation provide the
> > server with
> > the necessary information that this message is considered to be spam
> > by the
> > user. And by the same token, moving a message from this folder can be
> > construed
> > as the message not being considered spam after all.

> This is just trying to turn a folder into a keyword, but without full
> keyword semantics.

Actually, it's a hack t work around the lack of client support, which is why
I didn't propose a keyword-based mechanism.

> The most obvious flaw is that it constrains the
> client to storing all the spam in one bucket.

If there's a reason multiple folders can't be marked this way I sure don't
know what it is.

> It doesn't allow for (say)
> sorting junk by type into different folders (perhaps on different
> servers). A message's 'state of spamminess' is an attribute of the
> message, properly denoted by keywords that stick to (and with) the
> message.
(Continue reading)

Permalink | Reply |
headers
Lyndon Nerenberg | 6 Jul 2009 20:26
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Mon, 2009-07-06 at 11:00 -0700, Ned Freed wrote:
> If there's a reason multiple folders can't be marked this way I sure
> don't
> know what it is.

You're right -- I didn't parse what you were saying correctly the first
time around.

But if your concern is adoption rates, a keyword-only solution will get
wider take-up than something that requires support for annotations. (And
from a purely engineering standpoint, the simplest solution wins.)

--lyndon
Permalink | Reply |
headers
Lyndon Nerenberg | 6 Jul 2009 20:26
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On Mon, 2009-07-06 at 11:00 -0700, Ned Freed wrote:
> If there's a reason multiple folders can't be marked this way I sure
> don't
> know what it is.

You're right -- I didn't parse what you were saying correctly the first
time around.

But if your concern is adoption rates, a keyword-only solution will get
wider take-up than something that requires support for annotations. (And
from a purely engineering standpoint, the simplest solution wins.)

--lyndon
Permalink | Reply |
headers
Ned Freed | 7 Jul 2009 16:22

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

> On Mon, 2009-07-06 at 11:00 -0700, Ned Freed wrote:
> > If there's a reason multiple folders can't be marked this way I sure
> > don't
> > know what it is.

> You're right -- I didn't parse what you were saying correctly the first
> time around.

> But if your concern is adoption rates, a keyword-only solution will get
> wider take-up than something that requires support for annotations. (And
> from a purely engineering standpoint, the simplest solution wins.)

My concern isn't adoption rate. Rather, my concern is, as someone who provides
server software, how I can offer a better user experience in regards to spam.

Suppose I implement spam filter retraining based on setting of flags. how well
this works, or whether it works at all, depends entirely on the client people
use. And past experience on client uptake of new features like this is best
described as "lousy".

Now suppose I implement this with folder annotations. I'm in business
immediately. Things just work no matter what client people use.

There is, of course, the issue of setting the annotation on the proper folder
and  having a client to do that. But this isn't as much of a problem as it
sounds. Since the annotation only has to be set once, it can be done through a
simple web page, or better still, make it an option in the webmail interface.
(Everyone has a webmail interface these days.) And in many cases even this is
unnecessary, because many service providers predesignate a folder as the spam
folder.
(Continue reading)

Permalink | Reply |
headers
Cyrus Daboo | 7 Jul 2009 17:15
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Hi Ned,

--On July 7, 2009 7:22:42 AM -0700 Ned Freed <ned.freed <at> mrochek.com> wrote:

> Now suppose I implement this with folder annotations. I'm in business
> immediately. Things just work no matter what client people use.

Actually you need to consider what the user experience is likely to be on 
the client side. There are several things to consider: for spam messages 
users will typically want to delete them from their regular mailbox. With 
ham messages they will typically want to leave the message in place (or 
file them away after processing). In each case I certainly would not want 
the spam messages or an extra copy of the ham messages to reside on my 
server consuming quota or other resources, or being indexed as part of a 
server-wide search etc.

With that in mind lets look at how each solution might work client-side:

1) Keywords: for a spam message the user sets the spam keyword, then sets 
the deleted flag (and then may explicitly expunge if the client has that as 
an option as opposed to hide-deleted-messages with lazy expunge). For a ham 
message the user just sets the ham keyword. i.e. two user operations for 
spam, one for ham.

2) Mailbox (I am going to assume that the client has both a move and copy 
command available - the former implicitly setting the delete flag after the 
IMAP COPY command): for a spam message the user moves the message to the 
spam mailbox. For a ham message the user copies the message to the ham 
mailbox. NB both of these assume that the server does not actually store 
the spam/ham messages in the spam/ham mailboxes (or perhaps it garbage
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 7 Jul 2009 17:43

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

> Hi Ned,

> --On July 7, 2009 7:22:42 AM -0700 Ned Freed <ned.freed <at> mrochek.com> wrote:

> > Now suppose I implement this with folder annotations. I'm in business
> > immediately. Things just work no matter what client people use.

> Actually you need to consider what the user experience is likely to be on
> the client side. There are several things to consider: for spam messages
> users will typically want to delete them from their regular mailbox. With
> ham messages they will typically want to leave the message in place (or
> file them away after processing). In each case I certainly would not want
> the spam messages or an extra copy of the ham messages to reside on my
> server consuming quota or other resources, or being indexed as part of a
> server-wide search etc.

> With that in mind lets look at how each solution might work client-side:

> 1) Keywords: for a spam message the user sets the spam keyword, then sets
> the deleted flag (and then may explicitly expunge if the client has that as
> an option as opposed to hide-deleted-messages with lazy expunge). For a ham
> message the user just sets the ham keyword. i.e. two user operations for
> spam, one for ham.

> 2) Mailbox (I am going to assume that the client has both a move and copy
> command available - the former implicitly setting the delete flag after the
> IMAP COPY command): for a spam message the user moves the message to the
> spam mailbox. For a ham message the user copies the message to the ham
> mailbox. NB both of these assume that the server does not actually store
> the spam/ham messages in the spam/ham mailboxes (or perhaps it garbage
(Continue reading)

Permalink | Reply |
headers
Arnt Gulbrandsen | 6 Jul 2009 22:56
Picon
Favicon
Gravatar

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Lyndon Nerenberg writes:
> But if your concern is adoption rates, a keyword-only solution will 
> get wider take-up than something that requires support for 
> annotations.

Given that two significant clients (say two of the top five) use the 
keyword approach today, I think it's safe to say «has received wider 
take-up», not «will get wider».

Arnt
Permalink | Reply |
headers
Alexey Melnikov | 6 Jul 2009 20:29
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Lyndon Nerenberg wrote:

>But if your concern is adoption rates, a keyword-only solution will get
>wider take-up than something that requires support for annotations. (And
>from a purely engineering standpoint, the simplest solution wins.)
>  
>
+1.
Permalink | Reply |
headers
Arnt Gulbrandsen | 6 Jul 2009 22:56
Picon
Favicon
Gravatar

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


Lyndon Nerenberg writes:
> But if your concern is adoption rates, a keyword-only solution will 
> get wider take-up than something that requires support for 
> annotations.

Given that two significant clients (say two of the top five) use the 
keyword approach today, I think it's safe to say «has received wider 
take-up», not «will get wider».

Arnt
Permalink | Reply |
headers
Alexey Melnikov | 6 Jul 2009 20:29
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


Lyndon Nerenberg wrote:

>But if your concern is adoption rates, a keyword-only solution will get
>wider take-up than something that requires support for annotations. (And
>from a purely engineering standpoint, the simplest solution wins.)
>  
>
+1.
Permalink | Reply |
headers
Arnt Gulbrandsen | 6 Jul 2009 19:50
Picon
Favicon
Gravatar

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


Ned Freed writes:
> The obvious way to make this work in IMAP is with a standardized 
> folder annotation saying "messages put in this folder were flagged as 
> spam".

I think what Thunderbird and Apple Mail do is to set specific message 
flags on messages in whatever folders they happen to be in: $Junk if 
the user said a particular message is spam, $autojunk if tb 
autonomously decided that message is spam, $nonjunk and $autononjunk 
for the opposite. (I may be wrong about the exact flag names.)

This was specified in some expired draft, which IMO now has been frozen 
by deployment. Apple Mail and Thunderbird add up to >10% for me. Time 
for an informational RFC, perhaps. "Widely-used IMAP flag names"?

Arnt
Permalink | Reply |
headers
Alexey Melnikov | 6 Jul 2009 20:14
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Arnt Gulbrandsen wrote:

> Ned Freed writes:
>
>> The obvious way to make this work in IMAP is with a standardized 
>> folder annotation saying "messages put in this folder were flagged as 
>> spam".
>
> I think what Thunderbird and Apple Mail do is to set specific message 
> flags on messages in whatever folders they happen to be in: $Junk if 
> the user said a particular message is spam, $autojunk if tb 
> autonomously decided that message is spam, $nonjunk and $autononjunk 
> for the opposite. (I may be wrong about the exact flag names.)

Something like that, yes.

> This was specified in some expired draft,

Which was mine.

> which IMO now has been frozen by deployment. Apple Mail and 
> Thunderbird add up to >10% for me. Time for an informational RFC, 
> perhaps. "Widely-used IMAP flag names"?

Yes. I think even more important to create the IANA registry for keywords.
Do you want to help finish my draft?
Permalink | Reply |
headers
Alexey Melnikov | 6 Jul 2009 20:14
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


Arnt Gulbrandsen wrote:

> Ned Freed writes:
>
>> The obvious way to make this work in IMAP is with a standardized 
>> folder annotation saying "messages put in this folder were flagged as 
>> spam".
>
> I think what Thunderbird and Apple Mail do is to set specific message 
> flags on messages in whatever folders they happen to be in: $Junk if 
> the user said a particular message is spam, $autojunk if tb 
> autonomously decided that message is spam, $nonjunk and $autononjunk 
> for the opposite. (I may be wrong about the exact flag names.)

Something like that, yes.

> This was specified in some expired draft,

Which was mine.

> which IMO now has been frozen by deployment. Apple Mail and 
> Thunderbird add up to >10% for me. Time for an informational RFC, 
> perhaps. "Widely-used IMAP flag names"?

Yes. I think even more important to create the IANA registry for keywords.
Do you want to help finish my draft?
Permalink | Reply |
headers
George Michaelson | 6 Jul 2009 00:08
Picon
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On 06/07/2009, at 5:03 AM, Iljitsch van Beijnum wrote:

> On 3 jul 2009, at 1:05, George Michaelson wrote:
>
>> As an example, at the moment if I wish to inform google that I have  
>> known spam in local folders, I have to go to the web interface and  
>> manually tag. If there was an IMAP extension, I could review my  
>> local baysian junk folder, remove all non-spam (and flag the  
>> senders as white-listed if need be), and request the rest to be  
>> flagged as spam back on the IMAP backed MS.
>
> Doesn't moving the spam messages to the spam folder accomplish this  
> already? That's what my client does.
>

I think thats a side effect. you depend on the server having a rule  
which says that the folder of a specific name defines spam, and that  
moving mails to and from it makes things happen.

I'm not driven to side ffects. I like first class behaviour. I think  
that is fine, in its own way, but I'd like it to have a way to say it  
without doing that.

for instance. what if I run TWO email accounts, off different imap  
servers. I'd like the JOIN of their state to be used to mark senders  
white or black listed sometimes.

> But if you want this, I'd say that it needs to be a fractional  
> thing, not a binary spam/no spam indication. For instance, the
(Continue reading)

Permalink | Reply |
headers
Dave Cridland | 6 Jul 2009 10:24

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
> On 3 jul 2009, at 1:05, George Michaelson wrote:
> 
>> As an example, at the moment if I wish to inform google that I  
>> have  known spam in local folders, I have to go to the web  
>> interface and  manually tag. If there was an IMAP extension, I  
>> could review my  local baysian junk folder, remove all non-spam  
>> (and flag the senders  as white-listed if need be), and request  
>> the rest to be flagged as  spam back on the IMAP backed MS.
> 
> Doesn't moving the spam messages to the spam folder accomplish this  
>  already? That's what my client does.
> 
> 
There's also a proposal (expired?) to use keywords to signal  
spamminess.

> But if you want this, I'd say that it needs to be a fractional  
> thing,  not a binary spam/no spam indication. For instance, the  
> server could  give something a spam score of 2 and the client also  
> 2 and together  that would be 4 so the message is presumed to be  
> spam (assuming the  spam threshold is 3), but in a binary system no  
> spam OR no spam = no  spam.

Of course, if both client and server use precisely the same criteria,  
you've simply halved your threshold.

There's two cases:

1) The server has some feedback-based spam detection mechanism, like
(Continue reading)

Permalink | Reply |
headers
Alexey Melnikov | 6 Jul 2009 12:27
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Dave Cridland wrote:

> On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
>
>> On 3 jul 2009, at 1:05, George Michaelson wrote:
>>
>>> As an example, at the moment if I wish to inform google that I  
>>> have  known spam in local folders, I have to go to the web  
>>> interface and  manually tag. If there was an IMAP extension, I  
>>> could review my  local baysian junk folder, remove all non-spam  
>>> (and flag the senders  as white-listed if need be), and request  the 
>>> rest to be flagged as  spam back on the IMAP backed MS.
>>
>> Doesn't moving the spam messages to the spam folder accomplish this  
>>  already? That's what my client does.
>
> There's also a proposal (expired?) to use keywords to signal  spamminess.

To expand on this: I believe the proposal is implemented by several 
major email clients (Thunderbird is one of them).

[For people not familiar with IMAP: IMAP keywords is a basic IMAP 
feature implemented by majority of IMAP servers I've seen.]
Permalink | Reply |
headers
Ned Freed | 6 Jul 2009 19:39

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


> Dave Cridland wrote:

> > On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
> >
> >> On 3 jul 2009, at 1:05, George Michaelson wrote:
> >>
> >>> As an example, at the moment if I wish to inform google that I
> >>> have  known spam in local folders, I have to go to the web
> >>> interface and  manually tag. If there was an IMAP extension, I
> >>> could review my  local baysian junk folder, remove all non-spam
> >>> (and flag the senders  as white-listed if need be), and request  the
> >>> rest to be flagged as  spam back on the IMAP backed MS.
> >>
> >> Doesn't moving the spam messages to the spam folder accomplish this
> >>  already? That's what my client does.
> >
> > There's also a proposal (expired?) to use keywords to signal  spamminess.

> To expand on this: I believe the proposal is implemented by several
> major email clients (Thunderbird is one of them).

> [For people not familiar with IMAP: IMAP keywords is a basic IMAP
> feature implemented by majority of IMAP servers I've seen.]

The problem with using keywords for this is that you are now dependence on your
client supporting that particular set of keywords and their associated
semantics. Had we standardize such a keyword set 10 or so years ago this
approach miight have had a chance, but the client base is now too large and
too difficult to update.
(Continue reading)

Permalink | Reply |
headers
George Michaelson | 7 Jul 2009 00:51
Picon
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

>
>
> This is why I think folder annotations are a better bet. You mark  
> the spam
> folder and use that marking to attach meaning to the movement of  
> messages into
> and out of that folder. It's not perfect, but it has the key  
> advantage that it
> works with any client (OK, any client that can use folders) once the  
> annotation
> is in place.
>
> 				Ned

I have two IMAP servers, one google, one not.

how do I synchronize state between them. After all, If I (and I do)  
sometimes have to send from one, to reply to mails from the other  
(mail clients are bad at selecting sender account) it would be useful  
to have at least white and blacklist sender synchronization between  
them. Or, I could be sharing explicit spam tagging information to  
improve BOTH of their tracking. Do I really have to pass the entire  
messages between imap servers?

An explicit extension/flag/keyword would be unquestionably more  
efficient use of the network.

And, removing the mails from the spam folder in your methodology  
implies saying 'its not spam' -so the server I move them FROM thinks  
they've been un-tagged.
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 7 Jul 2009 02:42

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

> >
> >
> > This is why I think folder annotations are a better bet. You mark
> > the spam
> > folder and use that marking to attach meaning to the movement of
> > messages into
> > and out of that folder. It's not perfect, but it has the key
> > advantage that it
> > works with any client (OK, any client that can use folders) once the
> > annotation
> > is in place.
> >
> > 				Ned

> I have two IMAP servers, one google, one not.

That's a problem in and of itself, because last time I looked gmail is rather
limited in its support of IMAP.

> how do I synchronize state between them.

By "state" you appear to be referring to spam filtering state. If so, the short
answer is you can't possibly do it.

> After all, If I (and I do)
> sometimes have to send from one, to reply to mails from the other
> (mail clients are bad at selecting sender account) it would be useful
> to have at least white and blacklist sender synchronization between
> them.
(Continue reading)

Permalink | Reply |
headers
George Michaelson | 7 Jul 2009 03:28
Picon
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

hang on.

The MS has the message. It presumably has some held state of it,  
specific to its implementation. I don't want deep knowledge of that  
model, I really just want to be able to override it with my explicit  
knowledge.

I simply want to pass the toggle:
	
	If you class this as spam, please class as ham
or
	if you class this as ham, please class as spam

what it means, semantically, is of course MS specific. I don't care  
how its held, or what the effect on the weighting is, in the re-class  
as spam. In the class as ham, I care quite a lot that its sender is  
removed from hard blocks of course.

Yes, I see that moving this state between servers requires the entire  
message flow. But, if we forget that for now, and just consider that I  
have some mail in my local client cache, which I feel is now worthy of  
a state change, why do I have to pass full-state baysian model  
information? I just want to tell an MS "change your own flags to match  
my flags" in the hard sense: yes-or-no, not some qualitative weighting  
model. How it acts, is up to it. I don't see why I need a complex flag  
model to say what I think, as the consumer of the mail.

If I moved off SPAM/HAM to white/black on sender, would that be easier  
for you to consider?
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 7 Jul 2009 16:42

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

> hang on.

> The MS has the message. It presumably has some held state of it,
> specific to its implementation.

Actually, that's highly unlikely. There is some amounts of state IMAP requires
that's associated with the message - flags/keywords, annotations, some remnants
of the envelope. But beyond that there's little point in keeping a bunch of
implementation-specific state. Why? Because there's no way to expose such state
through the IMAP protocol. So when messages are FETCHed the state stays behind.
And more to the point, when a client moves a message, implementation-specific
state tends to get lost, because the client doesn't know how to access it in
order to move it.

The result is that in most csaes IMAP servers don't store much beyond the
things the standard requires them to store. The differences between servers lie
in how the standard stuff is stored and to what extent metainformation, like
body structured, is cached instead of being recomputed.

> I don't want deep knowledge of that
> model, I really just want to be able to override it with my explicit
> knowledge.

> I simply want to pass the toggle:
	
> 	If you class this as spam, please class as ham
> or
> 	if you class this as ham, please class as spam

This is exactly what we're trying to accomplish. The question is what mechanism
(Continue reading)

Permalink | Reply |
headers
George Michaelson | 8 Jul 2009 01:42
Picon
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

I have no desire to argue against my own proposal! sorry if I  
misunderstood the tenor of some comments to this suggestion.

The original posting asked about two things

1) spam/ham

2) white/black

so far, people seem to be focussing on 1). I acknowledge problems with  
1), I appreciate that people are trying to address 1) as a possibility.

I asked about 2). I asked you if, ignoring 1), it was possible or  
plausible to send white/black information about sender, if 1) was hard  
or intractable.

Does that make more sense? I do realize white/black listing is a  
different PROBLEM to spam/ham marking, but neither of them currently  
have support directly in imap, and I imagine both of them would use  
IMAP extensions to pass information client-server.

Sorry if I'm being unclear. I'm not subscribed to the IMAPEXT, or  
other lists, I have very little context for this beyond wanting a  
facility to cover a problem I see as a mail consumer. I'm not in the  
business of mail system development.

I appreciate people spending time discussing this. I still think there  
is a tractable problem here, some class of IMAP extension will permit  
better information flow from the client side.
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 8 Jul 2009 05:02

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

> I have no desire to argue against my own proposal! sorry if I
> misunderstood the tenor of some comments to this suggestion.

> The original posting asked about two things

> 1) spam/ham

> 2) white/black

> so far, people seem to be focussing on 1). I acknowledge problems with
> 1), I appreciate that people are trying to address 1) as a possibility.

> I asked about 2). I asked you if, ignoring 1), it was possible or
> plausible to send white/black information about sender, if 1) was hard
> or intractable.

Of course it's possible, but this has no business being in IMAP. IMAP is a
message access protocol, not an address handling protocol. It is also already
quite complex, and extending its function to include what amountss to a
specialized address book capability is a really bad idea IMO.

In fact a pretty good case can be made that it's inappropriate to use 
IMAP for spam/ham classification, but at the spam/ham classification is
intrinsicly tied to messages in the store. The same cannot be said for
address white and blacklisting.

> Does that make more sense? I do realize white/black listing is a
> different PROBLEM to spam/ham marking, but neither of them currently
> have support directly in imap, and I imagine both of them would use
> IMAP extensions to pass information client-server.
(Continue reading)

Permalink | Reply |
headers
George Michaelson | 9 Jul 2009 00:27
Picon
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

>>
>
> The place where I'd like to see white and black listing done is in  
> CARDDAV, not
> IMAP.
>
> 				Ned

so there's only one real question: if it was standardized, would the  
likes of google implement it? How about the client side?

because, unless its in gmail AND in the Apple Mail client,  
Thunderbird, Outlook, its not useful.

-G
Permalink | Reply |
headers
Ned Freed | 9 Jul 2009 02:30

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


> >>
> >
> > The place where I'd like to see white and black listing done is in
> > CARDDAV, not
> > IMAP.
> >
> > 				Ned

> so there's only one real question: if it was standardized, would the
> likes of google implement it? How about the client side?

This is actually one of the reasons why CARDDAV is a better choice to piggyback
on.

RIght now proprietary environments like Exchange/Outlook and Lotus Notes
have a significant advantage over standards-based solutions: They provide
a fully integrated mail/calendar/addressbook client experience.

One reason standards-based solutions have a problem with this has been the lack
of open protocols for accessing and manipulating calendar and addressbook
information.

But this is changing. CALDAV addresses the calendar access issue, CARDDAV does
the same for address books. Once all the pieces are in place, and assuming we
get the details right, this will allow standards-based clients and servers all
to interoperate.

Uptake isn't going to be fast because these are complex protocols and
getting the integration right isn't easy. But for once the incentives
(Continue reading)

Permalink | Reply |
headers
Cyrus Daboo | 9 Jul 2009 04:50
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Hi Ned,

--On July 8, 2009 5:30:06 PM -0700 Ned Freed <ned.freed <at> mrochek.com> wrote:

> I note, however, that now that CALDAV is complete we see Apple,
> Thunderbird, and gmail all either supporting it or working on supporting
> it.

For a more comprehensive list of implementations see: 
<http://caldav.calconnect.org/implementations.html>.

> It's a little much to expect Outlook to shift from it's currrent
> proprietary way of doing all this, but even Microsoft may eventually have
> no choice but to support the emerging standards in this area.

In the meantime the folks at <http://zideone.com> have a plugin for Outlook 
that does CalDAV and CardDAV.

--

-- 
Cyrus Daboo
Permalink | Reply |
headers
Anil SRIVASTAVA | 8 Jul 2009 05:50
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On 2009-07-07/20:02 [-0700], ned.freed <at> mrochek.com [Ned Freed] wrote:

> 
> > I have no desire to argue against my own proposal! sorry if I
> > misunderstood the tenor of some comments to this suggestion.
> 
> > The original posting asked about two things
> 
> > 1) spam/ham
> 
> > 2) white/black
> 
> > so far, people seem to be focussing on 1). I acknowledge problems with
> > 1), I appreciate that people are trying to address 1) as a possibility.
> 
> > I asked about 2). I asked you if, ignoring 1), it was possible or
> > plausible to send white/black information about sender, if 1) was hard
> > or intractable.
> 
> Of course it's possible, but this has no business being in IMAP. IMAP is 
> a message access protocol, not an address handling protocol. It is also 
> already quite complex, and extending its function to include what 
> amountss to a specialized address book capability is a really bad idea 
> IMO.
> 
> In fact a pretty good case can be made that it's inappropriate to 
> useIMAP for spam/ham classification, but at the spam/ham classification 
> is intrinsicly tied to messages in the store. The same cannot be said 
> for address white and blacklisting.
>
(Continue reading)

Permalink | Reply |
headers
Alexey Melnikov | 8 Jul 2009 12:08
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

Ned Freed wrote:

>> I have no desire to argue against my own proposal! sorry if I
>> misunderstood the tenor of some comments to this suggestion.
>
>> The original posting asked about two things
>
>> 1) spam/ham
>
>> 2) white/black
>
>> so far, people seem to be focussing on 1). I acknowledge problems with
>> 1), I appreciate that people are trying to address 1) as a possibility.
>
>> I asked about 2). I asked you if, ignoring 1), it was possible or
>> plausible to send white/black information about sender, if 1) was hard
>> or intractable.
>
> Of course it's possible, but this has no business being in IMAP. IMAP 
> is a
> message access protocol, not an address handling protocol. It is also 
> already
> quite complex, and extending its function to include what amountss to a
> specialized address book capability is a really bad idea IMO.
>
> In fact a pretty good case can be made that it's inappropriate to use 
> IMAP for spam/ham classification, but at the spam/ham classification is
> intrinsicly tied to messages in the store. The same cannot be said for
> address white and blacklisting.
(Continue reading)

Permalink | Reply |
headers
Iljitsch van Beijnum | 8 Jul 2009 08:08
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On 8 jul 2009, at 5:02, Ned Freed wrote:

> The place where I'd like to see white and black listing done is in  
> CARDDAV, not
> IMAP.

I never felt comfortable uploading all my contact info to Google. (May  
be somewhat strange since they do have all the mail sent to one of my  
email addresses.)

So I think white/blacklists shouldn't be complete contact info, but  
rather, just lists of hashes of email addresses.
Permalink | Reply |
headers
Ned Freed | 8 Jul 2009 08:19

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


> On 8 jul 2009, at 5:02, Ned Freed wrote:

> > The place where I'd like to see white and black listing done is in
> > CARDDAV, not
> > IMAP.

> I never felt comfortable uploading all my contact info to Google. (May
> be somewhat strange since they do have all the mail sent to one of my
> email addresses.)

That's fine, but this doesn't translate into a need to put an addressbook
capability in IMAP.

> So I think white/blacklists shouldn't be complete contact info, but
> rather, just lists of hashes of email addresses.

So use CARDDAV to only upload the stuff that's needed to Google. Problem
solved.

I'll also note that addressbook programs have had facilities for exporting
subsets of the information they contain pretty much from the start. So doing
this  is hardly entering new territory.

And most users want an authoritative address book with all their contact
information in one place. I know I do - it's much too confusing otherwise.

				Ned
Permalink | Reply |
headers
Alexey Melnikov | 8 Jul 2009 12:08
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


Ned Freed wrote:

>> I have no desire to argue against my own proposal! sorry if I
>> misunderstood the tenor of some comments to this suggestion.
>
>> The original posting asked about two things
>
>> 1) spam/ham
>
>> 2) white/black
>
>> so far, people seem to be focussing on 1). I acknowledge problems with
>> 1), I appreciate that people are trying to address 1) as a possibility.
>
>> I asked about 2). I asked you if, ignoring 1), it was possible or
>> plausible to send white/black information about sender, if 1) was hard
>> or intractable.
>
> Of course it's possible, but this has no business being in IMAP. IMAP 
> is a
> message access protocol, not an address handling protocol. It is also 
> already
> quite complex, and extending its function to include what amountss to a
> specialized address book capability is a really bad idea IMO.
>
> In fact a pretty good case can be made that it's inappropriate to use 
> IMAP for spam/ham classification, but at the spam/ham classification is
> intrinsicly tied to messages in the store. The same cannot be said for
> address white and blacklisting.
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 8 Jul 2009 20:54

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


> Ned Freed wrote:

> >> I have no desire to argue against my own proposal! sorry if I
> >> misunderstood the tenor of some comments to this suggestion.
> >
> >> The original posting asked about two things
> >
> >> 1) spam/ham
> >
> >> 2) white/black
> >
> >> so far, people seem to be focussing on 1). I acknowledge problems with
> >> 1), I appreciate that people are trying to address 1) as a possibility.
> >
> >> I asked about 2). I asked you if, ignoring 1), it was possible or
> >> plausible to send white/black information about sender, if 1) was hard
> >> or intractable.
> >
> > Of course it's possible, but this has no business being in IMAP. IMAP
> > is a
> > message access protocol, not an address handling protocol. It is also
> > already
> > quite complex, and extending its function to include what amountss to a
> > specialized address book capability is a really bad idea IMO.
> >
> > In fact a pretty good case can be made that it's inappropriate to use
> > IMAP for spam/ham classification, but at the spam/ham classification is
> > intrinsicly tied to messages in the store. The same cannot be said for
> > address white and blacklisting.
(Continue reading)

Permalink | Reply |
headers
Alexey Melnikov | 19 Jul 2009 20:38
Favicon

Comments on draft-melnikov-sieve-external-lists-02.txt


Hi Ned,
I've changed the subject and moving my reply to the Sieve mailing list, 
where I think it belongs.

Ned Freed wrote:
 [...]

>> There is also a Sieve extension for making use of whitelists/blacklists
>> during mail delivery. See draft-melnikov-sieve-external-lists-02.txt.
>> It can work with CARDDAV, LDAP and other things.
>
> Interestingly, this extension in its current form is not compatible with
> at least one of the proposals that was made here. The issue is one I've
> raised before - the fact that :list is separate argument and not a 
> match type.
>
> Where there is clearly some utility in being able to say stuff along the
> lines of:
>
>    header :contains :list "subject" "tag:dirty-word-list"
>
> The problem is that in order for this to work the list has to be 
> enumerable.

This wasn't the intent. I was certainly aware of the need not to 
retrieve the whole membership list.
But you are saying that my syntax wouldn't work in this case.

> Not all lists are enumerable, and even some that are are so large even
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 19 Jul 2009 20:46

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


> I've changed the subject and moving my reply to the Sieve mailing list,
> where I think it belongs.

Agreed.

> >> There is also a Sieve extension for making use of whitelists/blacklists
> >> during mail delivery. See draft-melnikov-sieve-external-lists-02.txt.
> >> It can work with CARDDAV, LDAP and other things.
> >
> > Interestingly, this extension in its current form is not compatible with
> > at least one of the proposals that was made here. The issue is one I've
> > raised before - the fact that :list is separate argument and not a
> > match type.
> >
> > Where there is clearly some utility in being able to say stuff along the
> > lines of:
> >
> >    header :contains :list "subject" "tag:dirty-word-list"
> >
> > The problem is that in order for this to work the list has to be
> > enumerable.

> This wasn't the intent. I was certainly aware of the need not to
> retrieve the whole membership list.

> But you are saying that my syntax wouldn't work in this case.

It's not a question of syntax, but semantics. When a list isn't enumerable it
typically only supports some specific sort of matching that's intrinsic to the
(Continue reading)

Permalink | Reply |
headers
Alexey Melnikov | 19 Jul 2009 21:25
Favicon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


Ned Freed wrote:
 [...]

> There are basically two ways to address one. One is to say "some 
> combinations
> of :list and variaus MATCH-TYPES" are going to fail in some cases".
> Alternately, we can simply say "the MATCH-TYPE is implied by the list 
> itself,
> and :list is therefore best thought of as a MATCH-TYPE".
>
> I strongly prefer the latter. The former has least astonishment violation
> written all over it. It also fails to allow for lists whose underlying 
> match
> criteria do not align with existing Sieve match types, and there are a 
> LOT of
> those; prefix match, suffix match, IP address pattern matches, etc. 
> And before
> anyone suggests defining new match types for these, allow me to point 
> out that
> in some cases the ynderlying match types allowed from some sources are 
> open
> ended and extensible.

Would we need to have :list-is, :list-matches, :list-contains, or can we 
get away with just :list, which would check for the exact match?
Permalink | Reply |
headers
Ned Freed | 19 Jul 2009 22:13

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


> Would we need to have :list-is, :list-matches, :list-contains, or can we
> get away with just :list, which would check for the exact match?

Making the match type part of the list argument is just moving the information
around and not changing the underlying semantics. Having any specification at
aall of the match type other than "it's matching a list" is exactly what we
don't want and which cannot be supported. 

Perhaps some specific examples will make this clear. Suppose I have a list of
whitelisted addresses, but because of privacy concerns this list contains the 
SHA-1 hash of each address. The only match type such a list would support is
:is. And it's an inherent characteristic of the list that that's the only
match type.

Now suppose I  have a list of half a million or so strings I don't want to see
as part of a subject line. I've used some algorithm or other to turn this into
a data structure that will tell me if any of those substrings appear in a
single  pass over the subject. The implied match type for such a list is
:contains.

Now consider a list consisting of IP addresses and net masks. I want to check
and see if a given IP address matches anything on the list. (Assuming the list
content is properly normalized, this can be done for IPv4 with no more than 32
exact match comparisons, 128 for IPv6. Of course other, more sophisticated
approaches are also possible.) This list has an effective match type that
corresponds to nothing we currently have in sieve, yet it is a perfectly
reasonable sort of list to want to match against.

Now think for a moment what's involved with having a list that support :matches
(Continue reading)

Permalink | Reply |
headers
Дилян Палаузов | 19 Jul 2009 21:41

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


Hello,

> Would we need to have :list-is, :list-matches, :list-contains, or can we 
> get away with just :list, which would check for the exact match?

As far as I understood Ned, it is the list who decides on the exact 
match type : it is not up to the script to prescribe the matching. So 
"just :list".

Any use cases where :list-is, :list-contains ... make sense and "just 
:list" would be insufficient?

Със здраве,
	дилян
Permalink | Reply |
headers
Alexey Melnikov | 20 Jul 2009 11:41
Favicon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


Дилян Палаузов wrote:

> Hello,
>
>> Would we need to have :list-is, :list-matches, :list-contains, or can 
>> we get away with just :list, which would check for the exact match?
>
> As far as I understood Ned, it is the list who decides on the exact 
> match type : it is not up to the script to prescribe the matching. So 
> "just :list".
>
> Any use cases where :list-is, :list-contains ... make sense and "just 
> :list" would be insufficient?

Ned has answered this nicely. I was thinking about substring matches 
(kind of simplified :matches), but this can be thought of as an 
implementation details. I.e. we can consider an external list to be 
black box.
Permalink | Reply |
headers
Barry Leiba | 21 Jul 2009 23:19
Picon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


> Ned has answered this nicely. I was thinking about substring matches (kind
> of simplified :matches), but this can be thought of as an implementation
> details. I.e. we can consider an external list to be black box.

I agree with Ned completely, and Alexey sums it up perfectly: an
external list is a black box.  It was the lack of that semantic that
made me resistant to external lists before.  If :list is a match type,
that major issue goes away for me (though I still have others, but I
think they can be resolved).

Barry
Permalink | Reply |
headers
Barry Leiba | 22 Jul 2009 07:55
Picon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


In my previous message I said that I had other issues with the
document.  Here they are:

Making :list a match type sorts out my issues with section 2.2, so that's fine.

But the same issues affect section 2.3.  If the list can't be
enumerated, you can't redirect to it.  Section 2.3 needs to make it
clear that whether an external list can do this or not depends upon
the list, and it needs to say what happens if it can't (ignore, or
throw runtime error seem to be the two sensible choices, and I think I
prefer the latter).

Section 2.4 says that the syntax of the list name is a URI, but it
doesn't talk about the semantics.  Along with the change to 2.2,
section 2.4 needs to change to (1) note that some URI types make sense
and some don't (Alexey already has this on his to-do list, and (2)
explain that the URI has to resolve to something that can be used as a
list in the manner specified in 2.2 and 2.3... and talk a little about
what that means.  Specifically, it has to be something that can be
given a string and can determine whether that string "is a member"...
and that it MAY also be something that can return its members as a
list or through an iterator.

Section 3, Security Considerations, needs some work, I think.  First,
I don't like that there's some normative language there that doesn't
appear elsewhere.  I prefer to have the normative parts of Security
Considerations be elsewhere in the document, in the parts that one
expects to be normative.
(Continue reading)

Permalink | Reply |
headers
Alexey Melnikov | 22 Jul 2009 12:38
Favicon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


Barry Leiba wrote:

>In my previous message I said that I had other issues with the
>document.  Here they are:
>
>Making :list a match type sorts out my issues with section 2.2, so that's fine.
>
>But the same issues affect section 2.3.  If the list can't be
>enumerated, you can't redirect to it.
>
:list is not a match type in this case, again it is a black box.

>Section 2.3 needs to make it
>clear that whether an external list can do this or not depends upon
>the list, and it needs to say what happens if it can't (ignore, or
>throw runtime error seem to be the two sensible choices, and I think I
>prefer the latter).
>  
>
We should discuss this in Stockholm (and on the mailing list). I wasn't 
thinking about being able to use the header test without being able to 
reference the same list in redirect.

>Section 2.4 says that the syntax of the list name is a URI, but it
>doesn't talk about the semantics.  Along with the change to 2.2,
>section 2.4 needs to change to (1) note that some URI types make sense
>and some don't (Alexey already has this on his to-do list,
>
Right.
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 22 Jul 2009 16:24

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


> Barry Leiba wrote:

> >In my previous message I said that I had other issues with the
> >document.  Here they are:
> >
> >Making :list a match type sorts out my issues with section 2.2, so that's fine.
> >
> >But the same issues affect section 2.3.  If the list can't be
> >enumerated, you can't redirect to it.
> >
> :list is not a match type in this case, again it is a black box.

> >Section 2.3 needs to make it
> >clear that whether an external list can do this or not depends upon
> >the list, and it needs to say what happens if it can't (ignore, or
> >throw runtime error seem to be the two sensible choices, and I think I
> >prefer the latter).
> >
> >
> We should discuss this in Stockholm (and on the mailing list). I wasn't
> thinking about being able to use the header test without being able to
> reference the same list in redirect.

Well, I can sort of see an argument that you might expect to be able to
reference any list that you used in an _address_ test in redirect, but
header? Who says the lists we'll be checking headers against are of
addresses or anything that correlates to addresses?

And in fact there are even lists of addresses (hashed lists are the obvious
(Continue reading)

Permalink | Reply |
headers
robertburrelldonkin | 23 Jul 2009 23:06
Picon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt

(thanks for kicking this off: the timing is fortuitous since i planned to start implementing something
similar soon. i'll start a new threads for any longer comments.) 

On Wed, Jul 22, 2009 at 3:24 PM, Ned Freed<ned.freed <at> mrochek.com> wrote:
>
>
>> Barry Leiba wrote:

<snip>

>> >Section 2.3 needs to make it
>> >clear that whether an external list can do this or not depends upon
>> >the list, and it needs to say what happens if it can't (ignore, or
>> >throw runtime error seem to be the two sensible choices, and I think I
>> >prefer the latter).
>> >
>> >
>> We should discuss this in Stockholm (and on the mailing list). I wasn't
>> thinking about being able to use the header test without being able to
>> reference the same list in redirect.
>
> Well, I can sort of see an argument that you might expect to be able to
> reference any list that you used in an _address_ test in redirect, but
> header? Who says the lists we'll be checking headers against are of
> addresses or anything that correlates to addresses?
>
> And in fact there are even lists of addresses (hashed lists are the obvious
> example) where you won't be able to reference them in redirect.
>
> The opposite is going to be true as well. There are plenty of use cases for
(Continue reading)

Permalink | Reply |
headers
Alexey Melnikov | 22 Jul 2009 22:18
Favicon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


Ned Freed wrote:
 [...]

>> We should discuss this in Stockholm (and on the mailing list). I wasn't
>> thinking about being able to use the header test without being able to
>> reference the same list in redirect.
>
> Well, I can sort of see an argument that you might expect to be able to
> reference any list that you used in an _address_ test in redirect, but
> header? Who says the lists we'll be checking headers against are of
> addresses or anything that correlates to addresses?
>
> And in fact there are even lists of addresses (hashed lists are the 
> obvious
> example) where you won't be able to reference them in redirect.
>
> The opposite is going to be true as well. There are plenty of use 
> cases for a
> list you want people to be able to redirect to but not test the 
> content of in
> any way.

I was originally thinking about email only lists, but you and Barry 
convinced me that other types of lists are also useful.

So yes, I will update the draft.
Permalink | Reply |
headers
Barry Leiba | 22 Jul 2009 15:46
Picon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


>> But the same issues affect section 2.3.  If the list can't be
>> enumerated, you can't redirect to it.
>>
> :list is not a match type in this case, again it is a black box.
...
> So I would like to correct your statement. There is no need whatsoever to be
> able to enumerate list membership in Sieve in order to redirect.
> The only requirement is that whoever does mailing list expansion needs to be
> able to do that. Such entity doesn't have to be the Sieve engine.

Two things:
1. When I say that the list must be enumerable, I don't necessarily
mean "by the Sieve engine."  It's possible for the list not to be
enumerable at all, even by the list handler.  Examples are where the
list stores hashes of email addresses (to preserve privacy, perhaps)
and where the list stores mailbox patterns (all members of department
Z have email addresses that start with "dz", so the list of department
Z members is querieable ("Is X a member?"), but not enumerable).

2. Ah, so you're suggesting that the Sieve engine, which knows how to
do a redirect, should pass this on to the list handler, which now also
has to know how to do a redirect.  I'm not sure that's a good idea,
but I could perhaps be convinced.  I'm curious about what Ned thinks
here.

>> Section 2.3 needs to make it
>> clear that whether an external list can do this or not depends upon
>> the list, and it needs to say what happens if it can't (ignore, or
>> throw runtime error seem to be the two sensible choices, and I think I
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 22 Jul 2009 16:49

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


> >> But the same issues affect section 2.3.  If the list can't be
> >> enumerated, you can't redirect to it.
> >>
> > :list is not a match type in this case, again it is a black box.
> ...
> > So I would like to correct your statement. There is no need whatsoever to be
> > able to enumerate list membership in Sieve in order to redirect.
> > The only requirement is that whoever does mailing list expansion needs to be
> > able to do that. Such entity doesn't have to be the Sieve engine.

> Two things:
> 1. When I say that the list must be enumerable, I don't necessarily
> mean "by the Sieve engine."  It's possible for the list not to be
> enumerable at all, even by the list handler.  Examples are where the
> list stores hashes of email addresses (to preserve privacy, perhaps)
> and where the list stores mailbox patterns (all members of department
> Z have email addresses that start with "dz", so the list of department
> Z members is querieable ("Is X a member?"), but not enumerable).

Exactly so. And by the same token, a list you can redirect to may not
necessarily be queryable in any other way.

> 2. Ah, so you're suggesting that the Sieve engine, which knows how to
> do a redirect, should pass this on to the list handler, which now also
> has to know how to do a redirect.  I'm not sure that's a good idea,
> but I could perhaps be convinced.  I'm curious about what Ned thinks
> here.

Handling redirect :list by throwing the problem over the fence to a list
(Continue reading)

Permalink | Reply |
headers
Robert Burrell Donkin | 23 Jul 2009 23:18
Picon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


On Wed, Jul 22, 2009 at 3:49 PM, Ned Freed<ned.freed <at> mrochek.com> wrote:
>
>
>> >> But the same issues affect section 2.3.  If the list can't be
>> >> enumerated, you can't redirect to it.
>> >>
>> > :list is not a match type in this case, again it is a black box.
>> ...
>> > So I would like to correct your statement. There is no need whatsoever to be
>> > able to enumerate list membership in Sieve in order to redirect.
>> > The only requirement is that whoever does mailing list expansion needs to be
>> > able to do that. Such entity doesn't have to be the Sieve engine.
>
>> Two things:
>> 1. When I say that the list must be enumerable, I don't necessarily
>> mean "by the Sieve engine."  It's possible for the list not to be
>> enumerable at all, even by the list handler.  Examples are where the
>> list stores hashes of email addresses (to preserve privacy, perhaps)
>> and where the list stores mailbox patterns (all members of department
>> Z have email addresses that start with "dz", so the list of department
>> Z members is querieable ("Is X a member?"), but not enumerable).
>
> Exactly so. And by the same token, a list you can redirect to may not
> necessarily be queryable in any other way.

+1

>> 2. Ah, so you're suggesting that the Sieve engine, which knows how to
>> do a redirect, should pass this on to the list handler, which now also
(Continue reading)

Permalink | Reply |
headers
Barry Leiba | 22 Jul 2009 18:18
Picon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt

Attached is my proposed -03 version (attached as "-03-pre01"), which
has NOT been posted as an I-D at this point.  I've sent Alexey my
edits, and the edit token is back with him.

I've made a bunch of changes, so forget DIFFs and please read the
whole thing (it's short) and comment.

Should this next version be renamed to
"draft-ietf-sieve-external-lists-00"?  In other words, should the WG
adopt it?  I think it should.

Barry


Sieve Working Group                                          A. Melnikov
Internet-Draft                                             Isode Limited
Intended status: Standards Track                           July 22, 2009
Expires: January 23, 2010

                Sieve Extension: Externally Stored Lists
                 draft-melnikov-sieve-external-lists-03

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
(Continue reading)

Permalink | Reply |
headers
Alexey Melnikov | 22 Jul 2009 22:50
Favicon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


Barry Leiba wrote:
 [...]

>Should this next version be renamed to
>"draft-ietf-sieve-external-lists-00"?  In other words, should the WG
>adopt it?  I think it should.
>  
>
Yes. This document is in our charter and my understanding is that the WG 
already made a decision to accept it.
Permalink | Reply |
headers
Alexey Melnikov | 22 Jul 2009 12:42
Favicon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


Alexey Melnikov wrote:

> Barry Leiba wrote:
>
>> In my previous message I said that I had other issues with the
>> document.  Here they are:
>>
>> Making :list a match type sorts out my issues with section 2.2, so 
>> that's fine.
>>
>> But the same issues affect section 2.3.  If the list can't be
>> enumerated, you can't redirect to it.
>
> :list is not a match type in this case, again it is a black box.

So I would like to correct your statement. There is no need whatsoever 
to be able to enumerate list membership in Sieve in order to redirect.
The only requirement is that whoever does mailing list expansion needs 
to be able to do that. Such entity doesn't have to be the Sieve engine.

>> Section 2.3 needs to make it
>> clear that whether an external list can do this or not depends upon
>> the list, and it needs to say what happens if it can't (ignore, or
>> throw runtime error seem to be the two sensible choices, and I think I
>> prefer the latter).
>
> We should discuss this in Stockholm (and on the mailing list). I 
> wasn't thinking about being able to use the header test without being 
> able to reference the same list in redirect.
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 20 Jul 2009 03:12

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


> Hello,

> > Would we need to have :list-is, :list-matches, :list-contains, or can we
> > get away with just :list, which would check for the exact match?

> As far as I understood Ned, it is the list who decides on the exact
> match type : it is not up to the script to prescribe the matching. So
> "just :list".

> Any use cases where :list-is, :list-contains ... make sense and "just
> :list" would be insufficient?

Sure, you can have lists that support more than one kind of matching, although
I expect such cases to be the exception rather than the rule. In such cases I
would propose that the type of matching be embedded in the list name somehow.

This then gets to the idea of using URLs for list names. A tag: URL is of
course completely capable of handling additional embedded material. Other
URL schemes, not so much. In fact I'm not entirely sure how this is supposed to
work with LDAP URLs with reasonable efficiency.

Consider the fairly typical case of whitelisting based on a personal address
book in LDAP. Assuming everyone is OK with users being able to construct
and perform their own LDAP searches and have no problem knowing and hard coding
the base DN of their entries (both very big ifs), this would call for something
like:

   if address :list "from"
      "ldap:///cn=me,o=pab?mail?sub?(objectclass=regularentry)" {
(Continue reading)

Permalink | Reply |
headers
Robert Burrell Donkin | 23 Jul 2009 23:48
Picon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


2009/7/20 Ned Freed <ned.freed <at> mrochek.com>:
>
>
>> Hello,
>
>> > Would we need to have :list-is, :list-matches, :list-contains, or can we
>> > get away with just :list, which would check for the exact match?
>
>> As far as I understood Ned, it is the list who decides on the exact
>> match type : it is not up to the script to prescribe the matching. So
>> "just :list".
>
>> Any use cases where :list-is, :list-contains ... make sense and "just
>> :list" would be insufficient?
>
> Sure, you can have lists that support more than one kind of matching,
> although
> I expect such cases to be the exception rather than the rule. In such cases
> I
> would propose that the type of matching be embedded in the list name
> somehow.
>
> This then gets to the idea of using URLs for list names. A tag: URL is of
> course completely capable of handling additional embedded material. Other
> URL schemes, not so much. In fact I'm not entirely sure how this is supposed
> to
> work with LDAP URLs with reasonable efficiency.
>
> Consider the fairly typical case of whitelisting based on a personal address
(Continue reading)

Permalink | Reply |
headers
Ned Freed | 24 Jul 2009 14:55

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


> 2009/7/20 Ned Freed <ned.freed <at> mrochek.com>:
> >
> >
> >> Hello,
> >
> >> > Would we need to have :list-is, :list-matches, :list-contains, or can we
> >> > get away with just :list, which would check for the exact match?
> >
> >> As far as I understood Ned, it is the list who decides on the exact
> >> match type : it is not up to the script to prescribe the matching. So
> >> "just :list".
> >
> >> Any use cases where :list-is, :list-contains ... make sense and "just
> >> :list" would be insufficient?
> >
> > Sure, you can have lists that support more than one kind of matching,
> > although
> > I expect such cases to be the exception rather than the rule. In such cases
> > I
> > would propose that the type of matching be embedded in the list name
> > somehow.
> >
> > This then gets to the idea of using URLs for list names. A tag: URL is of
> > course completely capable of handling additional embedded material. Other
> > URL schemes, not so much. In fact I'm not entirely sure how this is supposed
> > to
> > work with LDAP URLs with reasonable efficiency.
> >
> > Consider the fairly typical case of whitelisting based on a personal address
(Continue reading)

Permalink | Reply |
headers
Robert Burrell Donkin | 24 Jul 2009 16:11
Picon

Re: Comments on draft-melnikov-sieve-external-lists-02.txt


(apologies for messing up the reply-to foo first time 8-)

On Fri, Jul 24, 2009 at 1:55 PM, Ned Freed<ned.freed <at> mrochek.com> wrote:
>
>> 2009/7/20 Ned Freed <ned.freed <at> mrochek.com>:
>> >
>> >
>> >> Hello,
>> >
>> >> > Would we need to have :list-is, :list-matches, :list-contains, or can we
>> >> > get away with just :list, which would check for the exact match?
>> >
>> >> As far as I understood Ned, it is the list who decides on the exact
>> >> match type : it is not up to the script to prescribe the matching. So
>> >> "just :list".
>> >
>> >> Any use cases where :list-is, :list-contains ... make sense and "just
>> >> :list" would be insufficient?
>> >
>> > Sure, you can have lists that support more than one kind of matching,
>> > although
>> > I expect such cases to be the exception rather than the rule. In such cases
>> > I
>> > would propose that the type of matching be embedded in the list name
>> > somehow.
>> >
>> > This then gets to the idea of using URLs for list names. A tag: URL is of
>> > course completely capable of handling additional embedded material. Other
>> > URL schemes, not so much. In fact I'm not entirely sure how this is supposed
(Continue reading)

Permalink | Reply |
headers
Anil SRIVASTAVA | 8 Jul 2009 05:50
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On 2009-07-07/20:02 [-0700], ned.freed <at> mrochek.com [Ned Freed] wrote:

> 
> > I have no desire to argue against my own proposal! sorry if I
> > misunderstood the tenor of some comments to this suggestion.
> 
> > The original posting asked about two things
> 
> > 1) spam/ham
> 
> > 2) white/black
> 
> > so far, people seem to be focussing on 1). I acknowledge problems with
> > 1), I appreciate that people are trying to address 1) as a possibility.
> 
> > I asked about 2). I asked you if, ignoring 1), it was possible or
> > plausible to send white/black information about sender, if 1) was hard
> > or intractable.
> 
> Of course it's possible, but this has no business being in IMAP. IMAP is 
> a message access protocol, not an address handling protocol. It is also 
> already quite complex, and extending its function to include what 
> amountss to a specialized address book capability is a really bad idea 
> IMO.
> 
> In fact a pretty good case can be made that it's inappropriate to 
> useIMAP for spam/ham classification, but at the spam/ham classification 
> is intrinsicly tied to messages in the store. The same cannot be said 
> for address white and blacklisting.
(Continue reading)

Permalink | Reply |
headers
Lyndon Nerenberg | 6 Jul 2009 20:02
Picon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On Mon, 2009-07-06 at 10:39 -0700, Ned Freed wrote:
> The problem with using keywords for this is that you are now
> dependence on your
> client supporting that particular set of keywords and their associated
> semantics. Had we standardize such a keyword set 10 or so years ago
> this
> approach miight have had a chance, but the client base is now too
> large and
> too difficult to update.

Is it really too late? We're making an request to process a particular
message through the site's "learning" filters, i.e. this is explicit
request to have the server take action on the message. I don't that
behaviour is implied by any of the current schemes floating around.

I don't see how it could hurt to standardize a pair of new keywords (as
I suggested in my previous message) with this behaviour documented. It
lets the user request specific identified behaviour from the server; the
existing keywords are all private switches for client behaviour, in my
experience, so the new keyword behaviour would be complimentary to
what's already out there.

--lyndon
Permalink | Reply |
headers
Ned Freed | 6 Jul 2009 20:35

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


> On Mon, 2009-07-06 at 10:39 -0700, Ned Freed wrote:
> > The problem with using keywords for this is that you are now
> > dependence on your
> > client supporting that particular set of keywords and their associated
> > semantics. Had we standardize such a keyword set 10 or so years ago
> > this
> > approach miight have had a chance, but the client base is now too
> > large and
> > too difficult to update.

> Is it really too late? We're making an request to process a particular
> message through the site's "learning" filters, i.e. this is explicit
> request to have the server take action on the message. I don't that
> behaviour is implied by any of the current schemes floating around.

Well, you have two choices: Try to extend an existing ad-hoc keyword
usage scheme that has some deployment but doesn't have quite these semaantics
currently, or start from scratch with a fresh keyword set that has zero
deployment and is unlikely to displace any of the ad-hoc schemes.

This choice is why I view using keywords as at a minimum suboptimal.

> I don't see how it could hurt to standardize a pair of new keywords (as
> I suggested in my previous message) with this behaviour documented. It
> lets the user request specific identified behaviour from the server; the
> existing keywords are all private switches for client behaviour, in my
> experience, so the new keyword behaviour would be complimentary to
> what's already out there.
(Continue reading)

Permalink | Reply |
headers
Iljitsch van Beijnum | 6 Jul 2009 10:37
Favicon

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On 6 jul 2009, at 10:24, Dave Cridland wrote:

>> But if you want this, I'd say that it needs to be a fractional  
>> thing,  not a binary spam/no spam indication. For instance, the  
>> server could  give something a spam score of 2 and the client also  
>> 2 and together  that would be 4 so the message is presumed to be  
>> spam (assuming the  spam threshold is 3), but in a binary system no  
>> spam OR no spam = no  spam.

> The notion of using two spam filters in concert to attempt to make  
> an overall decision is basically flawed, because it fits into one or  
> other case above - if you do go to the effort of having a range  
> based spamminess from one, you'd have to use it as mere input into  
> the other's decision process, since combining them naïvely would  
> produce poorly weighted results.

Right. The filtering would have to be complimentary. So perhaps there  
should be an indication of the type of filtering done, too. The reason  
to combine server and client side approaches is that servers can  
easily do blacklists, and running any type of filtering on the server  
gives it the opportunity to reject the message during the SMTP  
session, creating a good error message for real senders but no useless  
bounces to innocent third parties in the case of spoofed senders. On  
the client is useful because the client can typically better afford to  
run CPU intensive stuff like baysian filters and the client probably  
has a better list of previous correspondents that it can use to  
whitelist.
Permalink | Reply |
headers
Dave Cridland | 6 Jul 2009 11:05

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Mon Jul  6 09:37:06 2009, Iljitsch van Beijnum wrote:
> 
> On 6 jul 2009, at 10:24, Dave Cridland wrote:
> 
>>> But if you want this, I'd say that it needs to be a fractional   
>>> thing,  not a binary spam/no spam indication. For instance, the   
>>> server could  give something a spam score of 2 and the client  
>>> also  2 and together  that would be 4 so the message is presumed  
>>> to be  spam (assuming the  spam threshold is 3), but in a binary  
>>> system no  spam OR no spam = no  spam.
> 
>> The notion of using two spam filters in concert to attempt to make  
>>  an overall decision is basically flawed, because it fits into one  
>> or  other case above - if you do go to the effort of having a  
>> range  based spamminess from one, you'd have to use it as mere  
>> input into  the other's decision process, since combining them  
>> naïvely would  produce poorly weighted results.
> 
> Right. The filtering would have to be complimentary. So perhaps  
> there  should be an indication of the type of filtering done, too.  
> The reason  to combine server and client side approaches is that  
> servers can  easily do blacklists, and running any type of  
> filtering on the server  gives it the opportunity to reject the  
> message during the SMTP  session, creating a good error message for  
> real senders but no useless  bounces to innocent third parties in  
> the case of spoofed senders. On  the client is useful because the  
> client can typically better afford to  run CPU intensive stuff like  
> baysian filters and the client probably  has a better list of  
> previous correspondents that it can use to  whitelist.
(Continue reading)

Permalink | Reply |
headers
Dave Cridland | 6 Jul 2009 11:05

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing


On Mon Jul  6 09:37:06 2009, Iljitsch van Beijnum wrote:
> 
> On 6 jul 2009, at 10:24, Dave Cridland wrote:
> 
>>> But if you want this, I'd say that it needs to be a fractional   
>>> thing,  not a binary spam/no spam indication. For instance, the   
>>> server could  give something a spam score of 2 and the client  
>>> also  2 and together  that would be 4 so the message is presumed  
>>> to be  spam (assuming the  spam threshold is 3), but in a binary  
>>> system no  spam OR no spam = no  spam.
> 
>> The notion of using two spam filters in concert to attempt to make  
>>  an overall decision is basically flawed, because it fits into one  
>> or  other case above - if you do go to the effort of having a  
>> range  based spamminess from one, you'd have to use it as mere  
>> input into  the other's decision process, since combining them  
>> naïvely would  produce poorly weighted results.
> 
> Right. The filtering would have to be complimentary. So perhaps  
> there  should be an indication of the type of filtering done, too.  
> The reason  to combine server and client side approaches is that  
> servers can  easily do blacklists, and running any type of  
> filtering on the server  gives it the opportunity to reject the  
> message during the SMTP  session, creating a good error message for  
> real senders but no useless  bounces to innocent third parties in  
> the case of spoofed senders. On  the client is useful because the  
> client can typically better afford to  run CPU intensive stuff like  
> baysian filters and the client probably  has a better list of  
> previous correspondents that it can use to  whitelist.
(Continue reading)

Permalink | Reply |
headers
Sam Johnston | 6 Jul 2009 11:27
Favicon
Gravatar

Re: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

On Mon, Jul 6, 2009 at 11:05 AM, Dave Cridland <dave <at> cridland.net> wrote:

I'm not suggesting that clients must not do spam filtering - obviously they do, and it's often useful - but I think we should aim to make server-side filtering the best we can. A very simple uniform feedback mechanism of whether the user thinks a message is spam or not makes this very much easier.

Agreed - I use the Gmail web interface exclusively now (after OS X Mail lost its mind a few years ago and forgot what was read, unread, labeled, etc.) but if the "Report Spam" button were available in mail clients then I might consider moving back. Being able to report that messages are not spam is useful too, as is being able to black/whitelist certain senders (albeit less so).

I'm sure I'm not alone in using a laptop and a mobile client (many have desktop and/or home clients in the mix too) so whatever can possibly be centralised should be.

Another thing to consider is that it's not always black and white - certain types of spam also fall into the phishing category and should be specifically investigated (Gmail has a "Report Phishing" menu item for this which allows them to label phishing messages accordingly) and Postini categorises spam so you can lower or raise the threshold depending on whether its type... that way accountants, lawers, etc. can avoid losing good mail (and others can avoid receiving it).

Sam

_______________________________________________
Apps-Discuss mailing list
Apps-Discuss <at> ietf.org
https://www.ietf.org/mailman/listinfo/apps-discuss
Permalink | Reply |

Gmane