Extensions to SMTP

Murray S. Kucherawy | 16 Nov 2011 07:00

FW: I-D Action: draft-kucherawy-received-state-00.txt

As previously discussed.  Feedback would be quite welcome.

-MSK

From: internet-drafts <at> ietf.org <internet-drafts <at> ietf.org>
Subject: I-D Action: draft-kucherawy-received-state-00.txt
Date: 2011-11-16 05:59:27 GMT

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title           : Indicating Email Handling States in Trace Fields
	Author(s)       : D. Crocker
                          Murray S. Kucherawy
	Filename        : draft-kucherawy-received-state-00.txt
	Pages           : 7
	Date            : 2011-11-15

   This memo registers a trace field clause for use in indicating
   transitions between handling queues, including enacting inter-host
   message transitions.  This might include message quarantining,
   mailing list moderation, timed delivery, or other similar causes.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-kucherawy-received-state-00.txt

(Continue reading)

headers

Hector Santos | 16 Nov 2011 09:23

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


Hi,

I read the document.  My comments.

I guess the irony here is that when I got your post, I was actually 
working precisely at the moment, on a new ID clause algorithm because 
of a new change in our receiver model would lose the uniqueness of the 
ID.  :)

Anyway, after reading the memo and its intent, I immediately saw it 
may present a "chicken and egg" design implementation issue for 
receivers models that will record the Received field into a temporary 
spool file and append the data payload into it.  They either have to 
split the capturing of information, do the STATE processing result, 
and then merge it for a final spool file.

For example, for our receiver, when a new mail transaction is started 
(at MAIL), a new file is created:

     hMailFile - holds the final result merging of meta headers and email.

This file would have the ENVELOPE records as the client goes thru MAIL 
to RCPT, etc.  When the DATA is reached before the 354 responses is 
issued:

    1.0 - Received Line is written to hMailFile
    2.0 - Any Received-SPF field is written (from MAIL/RCPT shims) to 
hMailFile

(Continue reading)

headers

Hector Santos | 16 Nov 2011 09:42

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

This is a follow up to my post, after thinking about this, the I-D is 
assuming that middle ware that touches the message and happens to add 
another Received: trace line would be appropriate for the proposed 
state clause.  In that case, it would be easier to implement.

I don't think it correct to mix up two different functionalities with 
the MTA hop Received hop stamping, so maybe the I-D should make a 
point of stating that the focus is with additional Received lines 
after the MTA required HOP Received field stamping.  Reading the doc, 
especially with the usage of the terms like "transient" and 
"transition," it would applicable to hop to hop required Received 
trace fields.  It could because the SMTP could be doing all the mail 
filtering.  While we do add more Received lines, there are done with 
different processes as the message is passed off to each. I have 
trouble with the idea the SMTP would add a 2nd receive line so I would 
probably be one that will add the state clause to the initial Received 
line.

--

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: hector <at> jabber.isdg.net

Murray S. Kucherawy wrote:
> As previously discussed.  Feedback would be quite welcome.
> 
> -MSK

(Continue reading)

headers

Bill McQuillan | 16 Nov 2011 10:44

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


The issue I have trouble with is the fact that a Received: field 
is added as an *event* and doesn't correspond to a *state*.

I guess it would help if it were made clear whether the "state"
is being entered into or being exited when the Received: field is
added.

I note that currently Received: fields are commonly added where 
the From and By values are the same which I presume is used to 
indicate some sort of internal queue transition. I would hope 
that this option would allow me to get a better picture of the 
processing steps involved.

--

-- 
Bill McQuillan <McQuilWP <at> pobox.com>

headers

Tony Finch | 16 Nov 2011 12:55

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


I agree with Bill McQuillan's remarks. If the server knows at the time it
receives the message that there is going to be a delay then it can include
the new clause in its usual Received: header. Otherwise it'll have to add
a new header. Would it be OK to edit the previously-added header?

The examples all have syntax errors. Received: sub-fields have a
prescribed order.

I think held-for would be a better (more descriptive) keyword than state.

Tony.
--

-- 
f.anthony.n.finch  <dot <at> dotat.at>  http://dotat.at/
East Sole, Lundy, Fastnet: Southeasterly 4 or 5, veering southwesterly 5 or 6
later, occasionally 7 later except Lundy. Moderate, becoming moderate or
rough. Rain or showers. Moderate or good.

headers

Murray S. Kucherawy | 18 Nov 2011 05:43

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: Tony Finch [mailto:fanf2 <at> hermes.cam.ac.uk] On Behalf Of Tony Finch
> Sent: Wednesday, November 16, 2011 3:55 AM
> To: Murray S. Kucherawy
> Cc: SMTP Discussion
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> I agree with Bill McQuillan's remarks. If the server knows at the time
> it receives the message that there is going to be a delay then it can
> include the new clause in its usual Received: header. Otherwise it'll
> have to add a new header. Would it be OK to edit the previously-added
> header?

I think RFC5321 says clearly that you're not supposed to do that ever.  Adding another one is more appropriate.

> The examples all have syntax errors. Received: sub-fields have a
> prescribed order.

Ah right.  Will fix those.

> I think held-for would be a better (more descriptive) keyword than
> state.

Hmm, maybe.  Anyone else have an opinion on the name?

-MSK

headers

Hector Santos | 18 Nov 2011 11:21

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

Murray S. Kucherawy wrote:

>> I agree with Bill McQuillan's remarks. If the server knows at the time
>> it receives the message that there is going to be a delay then it can
>> include the new clause in its usual Received: header. Otherwise it'll
>> have to add a new header. Would it be OK to edit the previously-added
>> header?
> 
> I think RFC5321 says clearly that you're not supposed to do that ever.  
> Adding another one is more appropriate.

I feel like chop liver here.

Of course, you can edit it.  It depends on when the message become, oh 
whats a good word, "live" again for the next process, or outside the 
network when you can't touch it again.

Read my initial response. Its a chicken and egg redesign issue - A.K.A 
Code Change.

Half the problem with many proposals is that they are based on one's 
idea of how code is written with their own software.  Not always the case.

Again, depending on your ware or the mode of the mail processor, the 
initial Received may be added immediately.  This memo is written more 
like its geared towards a non-SMTP receiver mail processors, where it 
would be a 2nd header.

For a SMTP receiver MTA that may be doing rejection at the transport

(Continue reading)

headers

Dave CROCKER | 19 Nov 2011 09:18

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


On 11/16/2011 7:55 PM, Tony Finch wrote:
> I think held-for would be a better (more descriptive) keyword than state.

state, or transition, or something that generic, is just that:  generic.  It 
pertains to an abstraction.

"held for" says "held" which is a very specific semantic that might not apply 
for some uses of this field.

d/

--

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

headers

Tony Finch | 19 Nov 2011 14:57

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

Dave CROCKER <dhc <at> dcrocker.net> wrote:
> On 11/16/2011 7:55 PM, Tony Finch wrote:
> > I think held-for would be a better (more descriptive) keyword than state.
>
> state, or transition, or something that generic, is just that:
> generic.  It pertains to an abstraction.
>
> "held for" says "held" which is a very specific semantic that might not
> apply for some uses of this field.

But the purpose of this field is to explain why a message was deliberately
delayed. The other Received: fields are similarly specific.

There's no need for multiple layers of abstraction and genericity here,
since the Received: field extension mechanism already accommodates that.

Tony.
--

-- 
f.anthony.n.finch  <dot <at> dotat.at>  http://dotat.at/
Portland, Plymouth: Southeast 5 or 6 decreasing 3 or 4. Moderate or rough in
Portland, rough or very rough in Plymouth. Mainly fair. Moderate or good.

headers

Murray S. Kucherawy | 18 Nov 2011 05:36

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


Hello Bill,

> -----Original Message-----
> From: owner-ietf-smtp <at> mail.imc.org [mailto:owner-ietf-smtp <at> mail.imc.org] On Behalf Of Bill McQuillan
> Sent: Wednesday, November 16, 2011 1:44 AM
> To: SMTP Discussion
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> The issue I have trouble with is the fact that a Received: field is
> added as an *event* and doesn't correspond to a *state*.
> 
> I guess it would help if it were made clear whether the "state"
> is being entered into or being exited when the Received: field is
> added.

The introduction says:

   This memo registers a new optional clause that can be used in trace
   fields to indicate that a message entered such a special processing
   queue for some period.  This allows analysis to reveal that the cause
   for a time gap in trace fields was an imposed delay rather than one
   caused by transient technical difficulties.

It sounds like if we changed "entered" to "has entered", that would resolve it for you?

> I note that currently Received: fields are commonly added where the
> From and By values are the same which I presume is used to indicate
> some sort of internal queue transition. I would hope that this option
> would allow me to get a better picture of the processing steps

(Continue reading)

headers

Bill McQuillan | 19 Nov 2011 08:21

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

On Thu, 2011-11-17, Murray S. Kucherawy wrote:

>> I guess it would help if it were made clear whether the "state"
>> is being entered into or being exited when the Received: field is
>> added.

> The introduction says:

>    This memo registers a new optional clause that can be used in trace
>    fields to indicate that a message entered such a special processing
>    queue for some period.  This allows analysis to reveal that the cause
>    for a time gap in trace fields was an imposed delay rather than one
>    caused by transient technical difficulties.

> It sounds like if we changed "entered" to "has entered", that would resolve it for you?

What I am trying to get clear is whether the date-time on the 
Received: field is the instant that the delaying state *started* or 
the instant that the delaying state *ended*.

Perhaps I'm being pedantic, but there may be other developers
that are as obtuse as I am. I don't feel that the language:
"message entered such a special processing queue for some
period" is completely clear on that point. 

Perhaps: "message is *beginning* a period of special processing"?

Or: "...*ending*...".

(Continue reading)

headers

Dave CROCKER | 19 Nov 2011 09:14

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

On 11/19/2011 3:21 PM, Bill McQuillan wrote:
> What I am trying to get clear is whether the date-time on the
> Received: field is the instant that the delaying state *started* or
> the instant that the delaying state *ended*.
>
> Perhaps I'm being pedantic,

You are, but this is a spec and it's kinda expected.

That said, the Received semantics have always been quite vague about 
distinctions such as entering into vs. departing from.  I think the general 
belief is that it's applied when the message "enters into" but I would /never/ 
want to have any system action be based on that assumption.

Hence the challenge for the current work is whether to stay with that vagueness 
or attempt to impose much more precise sequence definitions than (I believe) has 
previously applied.

I'm not a fan of false precision and I /am/ a fan of flexibility -- don't 
constrain things unless there's a strong need -- and I don't see the clear and 
immediate benefit of our imposing the precision that you are (implicitly?) 
asking for.

I think it's fine that you are asking about this; I'm merely expressing a 
preference for coarse-grained precision.[1]

d/

[1]  The first message system I specified displayed an "incorporating new mail"

(Continue reading)

headers

Bill McQuillan | 19 Nov 2011 20:49

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


On Sat, 2011-11-19, Dave CROCKER wrote:

> On 11/19/2011 3:21 PM, Bill McQuillan wrote:
>> What I am trying to get clear is whether the date-time on the
>> Received: field is the instant that the delaying state *started* or
>> the instant that the delaying state *ended*.
>>
>> Perhaps I'm being pedantic,

> You are, but this is a spec and it's kinda expected.

> That said, the Received semantics have always been quite vague about
> distinctions such as entering into vs. departing from.  I think the general
> belief is that it's applied when the message "enters into" but I would /never/
> want to have any system action be based on that assumption.

> Hence the challenge for the current work is whether to stay with that vagueness
> or attempt to impose much more precise sequence definitions than (I believe) has
> previously applied.

> I'm not a fan of false precision and I /am/ a fan of flexibility -- don't
> constrain things unless there's a strong need -- and I don't see the clear and
> immediate benefit of our imposing the precision that you are (implicitly?)
> asking for.

> I think it's fine that you are asking about this; I'm merely expressing a
> preference for coarse-grained precision.[1]

Since the motivation for the "state" sub-field is to help figure

(Continue reading)

headers

Murray S. Kucherawy | 20 Nov 2011 07:09

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: owner-ietf-smtp <at> mail.imc.org [mailto:owner-ietf-smtp <at> mail.imc.org] On Behalf Of Bill McQuillan
> Sent: Saturday, November 19, 2011 11:50 AM
> To: SMTP Discussion
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> Since the motivation for the "state" sub-field is to help figure out
> why gaps in the Received: date-time stamp sequence occured, I think
> that knowing for sure whether the named "state" is responsible for the
> gap BEFORE the current Received: date-time stamp or AFTER that date-
> time stamp is crucial.

I'm guessing my earlier suggestion was inadequate.  So how's this:

3.  New Trace Clause

   This memo creates a new trace field clause, called "state", which can
   be used to indicate the nature of a delay imposed on relaying of a
   message toward its recipient(s).  It is followed by a single keyword
   that provides that detail.  An MTA or other handling agent that
   determines a message is about to enter a state other than normal
   queueing of messages for delivery SHOULD generate a trace field
   including one of these clauses.  That is, the presence of this clause
   on a trace field is an indication of the entry of the message into
   that state; a later trace field added would indicate its departure
   from that state.

...and the rest unchanged.

(Continue reading)

headers

Hector | 20 Nov 2011 08:35

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


Murray S. Kucherawy wrote:
>> -----Original Message-----
>> From: owner-ietf-smtp <at> mail.imc.org [mailto:owner-ietf-smtp <at> mail.imc.org] On Behalf Of Bill McQuillan
>> Sent: Saturday, November 19, 2011 11:50 AM
>> To: SMTP Discussion
>> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
>>
>> Since the motivation for the "state" sub-field is to help figure out
>> why gaps in the Received: date-time stamp sequence occured, I think
>> that knowing for sure whether the named "state" is responsible for the
>> gap BEFORE the current Received: date-time stamp or AFTER that date-
>> time stamp is crucial.
> 
> I'm guessing my earlier suggestion was inadequate.  So how's this:
> 
> 3.  New Trace Clause
> 
>    This memo creates a new trace field clause, called "state", which can
>    be used to indicate the nature of a delay imposed on relaying of a
>    message toward its recipient(s).  It is followed by a single keyword
>    that provides that detail.  An MTA or other handling agent that
>    determines a message is about to enter a state other than normal
>    queueing of messages for delivery SHOULD generate a trace field
>    including one of these clauses.  That is, the presence of this clause
>    on a trace field is an indication of the entry of the message into
>    that state; a later trace field added would indicate its departure
>    from that state.

-1

(Continue reading)

headers

Bill McQuillan | 20 Nov 2011 10:05

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


On Sat, 2011-11-19, Hector wrote:

> Murray S. Kucherawy wrote:
>>> -----Original Message-----
>>> From: owner-ietf-smtp <at> mail.imc.org [mailto:owner-ietf-smtp <at> mail.imc.org] On Behalf Of Bill McQuillan
>>> Sent: Saturday, November 19, 2011 11:50 AM
>>> To: SMTP Discussion
>>> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
>>>
>>> Since the motivation for the "state" sub-field is to help figure out
>>> why gaps in the Received: date-time stamp sequence occured, I think
>>> that knowing for sure whether the named "state" is responsible for the
>>> gap BEFORE the current Received: date-time stamp or AFTER that date-
>>> time stamp is crucial.
>> 
>> I'm guessing my earlier suggestion was inadequate.  So how's this:
>> 
>> 3.  New Trace Clause
>> 
>>    This memo creates a new trace field clause, called "state", which can
>>    be used to indicate the nature of a delay imposed on relaying of a
>>    message toward its recipient(s).  It is followed by a single keyword
>>    that provides that detail.  An MTA or other handling agent that
>>    determines a message is about to enter a state other than normal
>>    queueing of messages for delivery SHOULD generate a trace field
>>    including one of these clauses.  That is, the presence of this clause
>>    on a trace field is an indication of the entry of the message into
>>    that state; a later trace field added would indicate its departure
>>    from that state.

(Continue reading)

headers

Murray S. Kucherawy | 20 Nov 2011 20:03

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: owner-ietf-smtp <at> mail.imc.org [mailto:owner-ietf-smtp <at> mail.imc.org] On Behalf Of Bill McQuillan
> Sent: Sunday, November 20, 2011 1:06 AM
> To: SMTP Discussion
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> While I would *encourage* an MTA to add this sub-field I have to agree
> with Hector that it does not qualify for SHOULD. Other than that, the
> paragraph satisfies my concerns about the meaning of the sub-field.

This document doesn't say that it updates RFC5321 or RFC5322.  It's a standalone document.  Thus, the SHOULD
here only applies to MTAs that decide to participate.

Put another way, you can't claim compliance with this document unless you apply the SHOULD, but extant
implementations are otherwise completely unaffected.

headers

Murray S. Kucherawy | 9 Jan 2012 23:45

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


Hi all,

In response to feedback here, I posted an -01 version of this document over a month ago.  It includes text
addressing Bill's question about what exactly is going on when a state change is indicated.

Further review and comments welcome, which will guide me toward selecting next steps (if any).

Cheers,
-MSK

headers

John Levine | 10 Jan 2012 05:08

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


In the first paragraph of section 3, I'd say MAY rather than SHOULD,
both because it conflicts with the OPTIONAL at the end of the section,
and just because the world has gotten along without this feature for
30 years so its practical necessity remains debatable.

Other than that it seems mostly harmless.  Has anyone implemented it yet?

R's,
John

headers

Murray S. Kucherawy | 10 Jan 2012 05:17

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: John Levine [mailto:johnl <at> taugh.com]
> Sent: Monday, January 09, 2012 8:08 PM
> To: ietf-smtp <at> imc.org
> Cc: Murray S. Kucherawy
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> In the first paragraph of section 3, I'd say MAY rather than SHOULD,
> both because it conflicts with the OPTIONAL at the end of the section,
> and just because the world has gotten along without this feature for
> 30 years so its practical necessity remains debatable.

The SHOULD only applies to people that are implementing this, meaning if you claim to be compliant with
this, then you'll always add it unless you have a very good (operational) reason why you don't.  If it was
SHOULD and also "Updates RFC5321", I'd agree with you, but this is meant as an independent and optional extension.

But since you're the second person to mention that, I guess some wordsmithing is in order to make that clear.

> Other than that it seems mostly harmless.  Has anyone implemented it
> yet?

I've floated the idea to some open source and commercial MTAs and mailbox providers, and one popular MLM. 
We'll see what traction it gets.

headers

John Levine | 10 Jan 2012 06:13

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


>But since you're the second person to mention that, I guess some
>wordsmithing is in order to make that clear.

In that case, suggest changing the OPTIONAL to something like "RECOMMENDED
if the message has been held."

Other nits: if this is really about deliberate delays, I'd suggest
using a term like "queued" or "delay" or "hold" rather than "state".
I can think of states that might be interesting but don't involve a
delay, e.g. characterized as spam, or downcoded from 8 bits to 6 bits.

Also, it might be a good idea to have a state name for no delay, e.g
"queued none", both for the benefit of dorky software that always wants
to put the same set of clauses in its Received: headers, and to provide
an explicit way to say hey, this delay wasn't my idea.

R's,
John

headers

Murray S. Kucherawy | 10 Jan 2012 07:50

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: John Levine [mailto:johnl <at> taugh.com]
> Sent: Monday, January 09, 2012 9:13 PM
> To: ietf-smtp <at> imc.org
> Cc: Murray S. Kucherawy
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> In that case, suggest changing the OPTIONAL to something like
> "RECOMMENDED if the message has been held."

But RECOMMENDED is the same as SHOULD, to which you originally objected.  The "if the message has been held"
is implicit in the idea that this is only used if some kind of administrative hold has been imposed (rather
than caused by a transport anomaly).

I just changed it to MAY, on the grounds that it will probably be user/admin pressure that gets the feature
added rather than something normative (e.g., nobody requires this, but all the cool kids do it).

> Other nits: if this is really about deliberate delays, I'd suggest
> using a term like "queued" or "delay" or "hold" rather than "state".
> I can think of states that might be interesting but don't involve a
> delay, e.g. characterized as spam, or downcoded from 8 bits to 6 bits.

Those aren't queueing states though; they're metadata about the message.  If I saw "state downcode", I'd
conclude that the time gap implied by the delta between adjacent Received fields indicated the amount of
time that host spent downcoding the message.

> Also, it might be a good idea to have a state name for no delay, e.g
> "queued none", both for the benefit of dorky software that always wants
> to put the same set of clauses in its Received: headers, and to provide

(Continue reading)

headers

Dave CROCKER | 10 Jan 2012 17:02

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

On 1/9/2012 10:50 PM, Murray S. Kucherawy wrote:
> I just changed it to MAY, on the grounds that it will probably be user/admin
> pressure that gets the feature added rather than something normative (e.g.,
> nobody requires this, but all the cool kids do it).

Mumble.  Although I can see the merit in choosing MAY, I suggest you revert to 
SHOULD.

This is a point of continuing confusion.  Does the normative language in an 
extension spec mean "relative to the overall service" or "relative to /this/ 
specification?

And then there is the usual debate about degree of importance that guides 
must/may/should.

My take:

      If we think use of the STATE clause is nice but not all that important, 
then the normative word should be MAY.  If we think this can have significant 
benefit and especially when widely adopted, the word should be SHOULD.

We could argue that email has done well enough without it for many years, so 
this should be MAY.  We could also argue that obscure delays are a significant 
problem and that the current scale of Internet mail and challenges in diagnosing 
problems means that it's important to guide potential implementers about the 
importance of implementing this.

I prefer SHOULD.

(Continue reading)

headers

Murray S. Kucherawy | 10 Jan 2012 19:02

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: Dave CROCKER [mailto:dhc <at> dcrocker.net]
> Sent: Tuesday, January 10, 2012 8:02 AM
> To: Murray S. Kucherawy
> Cc: ietf-smtp <at> imc.org
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> > Those aren't queueing states though; they're metadata about the
> > message.
> 
> I think I don't understand what this means.

The example given was "spam", which strikes me as a label attached to a message somehow (perhaps as
metadata) and not a phase of message handling enroute to its destination.

> Use of the clause requires a new Received header field.  The condition
> the 'none' value seems intended for is for typical header fields as are
> generated today.  Either these should get meaningful labels or the
> existing label 'other'
> should suffice.

I think "other" implies the message is entering some kind of queue state that will introduce an atypical
delay, where "none" is an explicit statement that this is not the case.  They are semantically different.

headers

Dave CROCKER | 10 Jan 2012 19:15

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

On 1/10/2012 10:02 AM, Murray S. Kucherawy wrote:
>>> Those aren't queueing states though; they're metadata about the message.
>>
>> I think I don't understand what this means.
>
> The example given was "spam", which strikes me as a label attached to a
> message somehow (perhaps as metadata) and not a phase of message handling
> enroute to its destination.

This suggests a confusion about the clause.  Does it provide a label to explain 
the specific Received field or does it label the message?  I think that having 
it used as a label for the message is just plain wrong.  Put those somewhere else.

>> Use of the clause requires a new Received header field.  The condition the
>> 'none' value seems intended for is for typical header fields as are
>> generated today.  Either these should get meaningful labels or the existing
>> label 'other' should suffice.
>
> I think "other" implies the message is entering some kind of queue state that
> will introduce an atypical delay, where "none" is an explicit statement that
> this is not the case.  They are semantically different.

OK.  Putting this simplistically, we already have lots of Received: fields being 
generated and we should have a clause value that covers this 
probably-uninteresting set?  I suggest "normal" or somesuch, not "none".  The 
message, /is/ after all, making a transition.  Whatever state or queue it just 
entered, it does exist.

d/

(Continue reading)

headers

Murray S. Kucherawy | 10 Jan 2012 19:21

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: Dave CROCKER [mailto:dhc <at> dcrocker.net]
> Sent: Tuesday, January 10, 2012 10:15 AM
> To: Murray S. Kucherawy
> Cc: ietf-smtp <at> imc.org
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> This suggests a confusion about the clause.  Does it provide a label to
> explain the specific Received field or does it label the message?  I
> think that having it used as a label for the message is just plain
> wrong.  Put those somewhere else.

+1.

> OK.  Putting this simplistically, we already have lots of Received:
> fields being generated and we should have a clause value that covers
> this probably-uninteresting set?  I suggest "normal" or somesuch, not
> "none".  The message, /is/ after all, making a transition.  Whatever
> state or queue it just entered, it does exist.

John suggested what's essentially a no-op state name to accommodate those implementations that, in
supporting this, will always want to put some kind of state clause down, and that seems a decent idea to me.

"normal" would be fine with me too.

-MSK

headers

Dave CROCKER | 10 Jan 2012 20:01

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

On 1/10/2012 10:21 AM, Murray S. Kucherawy wrote:
>> OK.  Putting this simplistically, we already have lots of Received: fields
>> being generated and we should have a clause value that covers this
>> probably-uninteresting set?  I suggest "normal" or somesuch, not "none".
>> The message, /is/ after all, making a transition.  Whatever state or queue
>> it just entered, it does exist.
>
> John suggested what's essentially a no-op state name to accommodate those
> implementations that, in supporting this, will always want to put some kind
> of state clause down, and that seems a decent idea to me.
>
> "normal" would be fine with me too.

Doing what sales folk call "selling past the sale" I'll note that there is no 
such thing as a no-op, since the presence of the Received: field means that some 
sort of 'op' took place.  So the question is what the "nature" of the op is.  A 
label like "normal" therefore can cover the existing header fields with a 
generic term for the nature.

d/

--

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

headers

Rolf E. Sonneveld | 10 Jan 2012 21:28

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


On 1/10/12 8:01 PM, Dave CROCKER wrote:
>
>
>
> On 1/10/2012 10:21 AM, Murray S. Kucherawy wrote:
>>> OK.  Putting this simplistically, we already have lots of Received: 
>>> fields
>>> being generated and we should have a clause value that covers this
>>> probably-uninteresting set?  I suggest "normal" or somesuch, not 
>>> "none".
>>> The message, /is/ after all, making a transition.  Whatever state or 
>>> queue
>>> it just entered, it does exist.
>>
>> John suggested what's essentially a no-op state name to accommodate 
>> those
>> implementations that, in supporting this, will always want to put 
>> some kind
>> of state clause down, and that seems a decent idea to me.
>>
>> "normal" would be fine with me too.
>
>
> Doing what sales folk call "selling past the sale" I'll note that 
> there is no such thing as a no-op, since the presence of the Received: 
> field means that some sort of 'op' took place.  So the question is 
> what the "nature" of the op is. 

The "nature" of the op (or no-op) is 'received'. I.e. the message was

(Continue reading)

headers

Hector Santos | 10 Jan 2012 19:02

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


+1 Dave.

My Take:

1) This should be a MAY implement
2) It can only be a SHOULD with a presumption of other ideas already
    existing.

Its like I MAY build a house, and if I do, I SHOULD do certain things 
expected in building a house.

This should not say I SHOULD build a house.

Dave CROCKER wrote:
> 
> 
> 
> On 1/9/2012 10:50 PM, Murray S. Kucherawy wrote:
>> I just changed it to MAY, on the grounds that it will probably be 
>> user/admin
>> pressure that gets the feature added rather than something normative 
>> (e.g.,
>> nobody requires this, but all the cool kids do it).
> 
> Mumble.  Although I can see the merit in choosing MAY, I suggest you 
> revert to SHOULD.
> 
> This is a point of continuing confusion.  Does the normative language in 
> an extension spec mean "relative to the overall service" or "relative to

(Continue reading)

headers

John Levine | 10 Jan 2012 18:26

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


>> In that case, suggest changing the OPTIONAL to something like
>> "RECOMMENDED if the message has been held."
>
>But RECOMMENDED is the same as SHOULD, to which you originally objected.

Quoting Abraham Lincoln*, when a judge pointed out that he was arguing
the opposite position from one in a case earlier in the day, "Your honor,
this morning I was wrong."

It seems reasonable to use MUSTard that assumes that the reader is going
to implement this spec, in which case it's SHOULD and RECOMMENDED.  Or not,
in which case it's MAY and OPTIONAL.  It wasn't clear to me that you're
only expected to add the clause when you queue the message for longer
than normal, but that should be easy enough to fix.

>> Other nits: if this is really about deliberate delays, I'd suggest
>> using a term like "queued" or "delay" or "hold" rather than "state".
>> I can think of states that might be interesting but don't involve a
>> delay, e.g. characterized as spam, or downcoded from 8 bits to 6 bits.
>
>Those aren't queueing states though; they're metadata about the message.
>If I saw "state downcode", I'd conclude that the time gap implied by the
>delta between adjacent Received fields indicated the amount of time that
>host spent downcoding the message.

Well, yeah, but there are a lot of kind of states other than queueing
states.  That's why I'd suggest a keyword that tells you this is about
queues or delays or something.

(Continue reading)

headers

Dave CROCKER | 10 Jan 2012 18:53

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

On 1/10/2012 9:26 AM, John Levine wrote:
>> Those aren't queueing states though; they're metadata about the message.
>> >If I saw "state downcode", I'd conclude that the time gap implied by the
>> >delta between adjacent Received fields indicated the amount of time that
>> >host spent downcoding the message.
> Well, yeah, but there are a lot of kind of states other than queueing
> states.  That's why I'd suggest a keyword that tells you this is about
> queues or delays or something.

This puts the specific semantic into the definition of the clause, rather than 
the value used in the clause.

The problem with this is lack of flexibility.

For example, it could be noteworthy to record a processing step that introduces 
no interesting delay.

If the new clause is only for delays, then we can't extend use of the clause to 
cover extra processing steps.  If the label is for more general use, then we can.

d/
--

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

headers

John Levine | 10 Jan 2012 20:10

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


>If the new clause is only for delays, then we can't extend use of the clause to 
>cover extra processing steps.  If the label is for more general use,
>then we can.

That's OK, but now I'm wondering if the syntax should allow a list of
states rather than just one

Received: blah blah STATE DOWNCODED-TO-7,MODERATED blah blah

I suppose you could use two received headers for two states, but that
seems unlike the way people use them now.

R's,
John

headers

Murray S. Kucherawy | 10 Jan 2012 20:18

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: owner-ietf-smtp <at> mail.imc.org [mailto:owner-ietf-smtp <at> mail.imc.org] On Behalf Of John Levine
> Sent: Tuesday, January 10, 2012 11:11 AM
> To: ietf-smtp <at> imc.org
> Cc: dcrocker <at> bbiw.net
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> >If the new clause is only for delays, then we can't extend use of the
> >clause to cover extra processing steps.  If the label is for more
> >general use, then we can.
> 
> That's OK, but now I'm wondering if the syntax should allow a list of
> states rather than just one
> 
> Received: blah blah STATE DOWNCODED-TO-7,MODERATED blah blah

I don't think we're talking about this kind of metadata here.  Really this is meant to talk about a delay
introduced while waiting for something to happen, usually operator intervention of some kind
(quarantine, list moderation).  An automatic step like downcoding or (for another example) virus
scrubbing only introduces a compute delay and a bit of I/O, not an enforced usually-human-caused delay
like the other cases.

headers

Tony Finch | 11 Jan 2012 17:12

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


Dave CROCKER <dhc <at> dcrocker.net> wrote:
> On 1/10/2012 9:26 AM, John Levine wrote:
> >
> > Well, yeah, but there are a lot of kind of states other than queueing
> > states.  That's why I'd suggest a keyword that tells you this is about
> > queues or delays or something.
>
> This puts the specific semantic into the definition of the clause, rather than
> the value used in the clause.

Just like the other Received: header clauses.

> The problem with this is lack of flexibility.

Which is why the Received: header can be extended with other clauses for
other purposes. We don't need multiple layers of genericity here.

Tony.
--

-- 
f.anthony.n.finch  <dot <at> dotat.at>  http://dotat.at/
Lundy, Fastnet, Irish Sea, Shannon: Southwest, veering west or northwest, 4 or
5, increasing 6 or 7 for a time in Irish Sea and Shannon. Slight or moderate,
occasionally rough in Fastnet and Shannon. Occasional rain. Moderate or good.

headers

Hector Santos | 11 Jan 2012 22:57

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


Tony Finch wrote:

>> The problem with this is lack of flexibility.
> 
> Which is why the Received: header can be extended with other clauses for
> other purposes. We don't need multiple layers of genericity here.

+1, ideal, but will most likely required the larger code change, in 
logic flow and when to add the Received since it has to be at the top 
for the current hop.

--

-- 
Sincerely

Hector Santos
http://www.santronics.com
jabber: hector <at> jabber.isdg.net

headers

Murray S. Kucherawy | 10 Jan 2012 18:59

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: John Levine [mailto:johnl <at> taugh.com]
> Sent: Tuesday, January 10, 2012 9:26 AM
> To: ietf-smtp <at> imc.org
> Cc: Murray S. Kucherawy
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> It seems reasonable to use MUSTard that assumes that the reader is
> going to implement this spec, in which case it's SHOULD and
> RECOMMENDED.  Or not, in which case it's MAY and OPTIONAL.

Then given this and Dave's post, I'm inclined to go back to SHOULD.

> It wasn't
> clear to me that you're only expected to add the clause when you queue
> the message for longer than normal, but that should be easy enough to
> fix.

The second paragraph of the Introduction section is meant to make this plain, i.e., that you do this for
anything other than queue-for-later-when-transport-fails.  Can you suggest wording tweaks?

> Well, yeah, but there are a lot of kind of states other than queueing
> states.  That's why I'd suggest a keyword that tells you this is about
> queues or delays or something.

I've never heard "spam" (your example) used as part of a state machine.  It's essentially a label attached to
a message, not a phase of handling while a message is in transit.

-MSK

(Continue reading)

headers

Tony Finch | 11 Jan 2012 17:05

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


John Levine <johnl <at> taugh.com> wrote:
>
> Other nits: if this is really about deliberate delays, I'd suggest
> using a term like "queued" or "delay" or "hold" rather than "state".
> I can think of states that might be interesting but don't involve a
> delay, e.g. characterized as spam, or downcoded from 8 bits to 6 bits.

I agree and I have previously made the same suggestion.

Tony.
--

-- 
f.anthony.n.finch  <dot <at> dotat.at>  http://dotat.at/
Tyne, Dogger, Fisher, German Bight, Humber: West or southwest, veering
northwest later, 4 or 5, increasing 6 to gale 8, occasionally severe gale 9 in
Fisher, perhaps severe gale 9 later in Tyne, Dogger and German Bight. Moderate
or rough, occasionally very rough. Rain or squally showers. Moderate or good.

headers

Dave CROCKER | 19 Nov 2011 06:45

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

On 11/16/2011 5:44 PM, Bill McQuillan wrote:
>
>
> The issue I have trouble with is the fact that a Received: field
> is added as an *event* and doesn't correspond to a *state*.
>
> I guess it would help if it were made clear whether the "state"
> is being entered into or being exited when the Received: field is
> added.

In my brain, I keep using the word 'transition' and note that your next 
paragraph did too.  But that's a word that we tend to use about states...

A discussion about the difference between states and events might be fun, if 
rather computer sciency.

Received fields are typically added to mark a change from one 'place' to 
another.  Queue, state, responsible party, i don't care.

Let's decide on the word we like and shift the spec to use it.  It won't change 
the details of the spec, but might make it's nature more intuitive to some readers.

> I note that currently Received: fields are commonly added where
> the From and By values are the same which I presume is used to
> indicate some sort of internal queue transition. I would hope
> that this option would allow me to get a better picture of the
> processing steps involved.

That's exactly the intent.

(Continue reading)

headers

Robert A. Rosenberg | 12 Jan 2012 06:20

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

At 10:15 -0800 on 01/10/2012, Dave CROCKER wrote about Re: FW: I-D 
Action: draft-kucherawy-received-state-00.txt:

>On 1/10/2012 10:02 AM, Murray S. Kucherawy wrote:
>>>>  Those aren't queueing states though; they're metadata about the message.
>>>
>>>  I think I don't understand what this means.
>>
>>  The example given was "spam", which strikes me as a label attached to a
>>  message somehow (perhaps as metadata) and not a phase of message handling
>>  enroute to its destination.
>
>This suggests a confusion about the clause.  Does it provide a label 
>to explain
>the specific Received field or does it label the message?  I think that having
>it used as a label for the message is just plain wrong.  Put those 
>somewhere else.

That depends on the meaning of the SPAM state. If it means checking 
if the message IS spam that is different from it meaning that the 
message has been identified AS spam. You are treating as the latter 
when the intent might be the former (ie: The check might introduce a 
delay in processing the message so you are marking it to show the 
reason for the delay).

headers

Murray S. Kucherawy | 12 Jan 2012 07:30

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: Robert A. Rosenberg [mailto:hal9001 <at> panix.com]
> Sent: Wednesday, January 11, 2012 9:20 PM
> To: dcrocker <at> bbiw.net
> Cc: Murray S. Kucherawy; ietf-smtp <at> imc.org; Dave CROCKER
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
>  
> That depends on the meaning of the SPAM state. If it means checking if
> the message IS spam that is different from it meaning that the message
> has been identified AS spam. You are treating as the latter when the
> intent might be the former (ie: The check might introduce a delay in
> processing the message so you are marking it to show the reason for the
> delay).

I don't know what it means for a message to be in a "spam" state.

A message in a "hold for moderation" state means it's stuck in a queue until a moderator does something with it.

A message in a "quarantine" state means it's quarantined, inaccessible, until an operator brings it out or
destroys it.

A message in a "timed" state means it's held in a queue until a certain release time arrives.

All of these are states that cause processing delays, but ultimately from which the message will be released.

If a "spam" state were to exist, what are the conditions under which it would be released for processing and delivery?

-MSK

(Continue reading)

headers

Robert A. Rosenberg | 12 Jan 2012 20:11

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


At 22:30 -0800 on 01/11/2012, Murray S. Kucherawy wrote about Re: FW: 
I-D Action: draft-kucherawy-received-state-00.txt:

>  > -----Original Message-----
>>  From: Robert A. Rosenberg [mailto:hal9001 <at> panix.com]
>>  Sent: Wednesday, January 11, 2012 9:20 PM
>>  To: dcrocker <at> bbiw.net
>>  Cc: Murray S. Kucherawy; ietf-smtp <at> imc.org; Dave CROCKER
>>  Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
>> 
>>  That depends on the meaning of the SPAM state. If it means checking if
>>  the message IS spam that is different from it meaning that the message
>>  has been identified AS spam. You are treating as the latter when the
>>  intent might be the former (ie: The check might introduce a delay in
>>  processing the message so you are marking it to show the reason for the
>>  delay).
>
>I don't know what it means for a message to be in a "spam" state.
>
>A message in a "hold for moderation" state means it's stuck in a 
>queue until a moderator does something with it.
>
>A message in a "quarantine" state means it's quarantined, 
>inaccessible, until an operator brings it out or destroys it.
>
>A message in a "timed" state means it's held in a queue until a 
>certain release time arrives.
>
>All of these are states that cause processing delays, but ultimately

(Continue reading)

headers

Carl S. Gutekunst | 12 Jan 2012 20:47

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt

Robert A. Rosenberg wrote:
>> If a "spam" state were to exist, what are the conditions under which 
>> it would be released for processing and delivery?
>>
>
> I was thinking of checking the message to see if it were possible 
> spam. You might have some cases where your processing can not do an 
> immediate SPAM/HAM determination but have some delay in making that 
> decision. Thus the spam state while the message is delayed while the 
> extended check is being done. 

Offline or deferred processing for potential threats or detecting policy 
violations is pretty common. But the keyword "spam" is a judgment, not a 
reason. How about verbs like "study", "analyze", or even "filter"?

("Weigh" and "adjudge" are arguable more accurate, except no one would 
have idea what they mean. Which is heavier, ham or spam?)

<csg>

headers

Murray S. Kucherawy | 12 Jan 2012 21:40

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: Carl S. Gutekunst [mailto:csg <at> alameth.org]
> Sent: Thursday, January 12, 2012 11:47 AM
> To: Robert A. Rosenberg
> Cc: Murray S. Kucherawy; ietf-smtp <at> imc.org
> Subject: Re: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> Offline or deferred processing for potential threats or detecting
> policy violations is pretty common. But the keyword "spam" is a
> judgment, not a reason. How about verbs like "study", "analyze", or
> even "filter"?

I was thinking about "content-eval" as a generic, and then the implementation could use a comment after
that to be more specific, e.g. "state content-eval (spam checking)".  The ABNF already allows this.

headers

Murray S. Kucherawy | 12 Jan 2012 21:39

RE: FW: I-D Action: draft-kucherawy-received-state-00.txt


> -----Original Message-----
> From: Robert A. Rosenberg [mailto:hal9001 <at> panix.com]
> Sent: Thursday, January 12, 2012 11:11 AM
> To: Murray S. Kucherawy
> Cc: ietf-smtp <at> imc.org
> Subject: RE: FW: I-D Action: draft-kucherawy-received-state-00.txt
> 
> I was thinking of checking the message to see if it were possible spam.
> You might have some cases where your processing can not do an immediate
> SPAM/HAM determination but have some delay in making that decision.
> Thus the spam state while the message is delayed while the extended
> check is being done.

OK, this makes more sense.  What I'd seen so far suggested using "state spam" to indicate that the agent
adding the field tagged it as spam.  The intent here is to show a transition into a processing state.  The
particular impact I'm hoping to enable is the ability to see why there was a delay between timestamps.

If however there are extant or possible implementations that actually queue a message for spam evaluation
by some other agent, and that's a totally separate step (even by separate software) than the basic message
intake, then it makes sense to denote that as a state in this context.

My own experience with various MTA implementations is that spam, virus, and other content evaluation is
all part of the intake step, before it hits a queue where prolonged delay is likely.  That's why this is
giving me quite a bit of pause.

headers

Carl S. Gutekunst | 12 Jan 2012 21:53

Re: FW: I-D Action: draft-kucherawy-received-state-00.txt


Murray S. Kucherawy wrote:
> If however there are extant or possible implementations that actually queue a message for spam
evaluation by some other agent, and that's a totally separate step (even by separate software) than the
basic message intake, then it makes sense to denote that as a state in this context.
>
> My own experience with various MTA implementations is that spam, virus, and other content evaluation is
all part of the intake step, before it hits a queue where prolonged delay is likely.  That's why this is
giving me quite a bit of pause.
>   
Shoveling the message off to another service for evaluation is indeed 
rare among stand-alone MTAs, but pretty common in cloud-based 
implementations, especially those that use a forest of single-purpose 
blades. There are frontends to SpamAssassin that support this as well.

<csg>