headers
River Tarnell | 1 Jan 2012 02:47

Invalid 431 response when server is paused

Hi,

While working on some news software I noticed something odd from a peer:

Jan 01 01:36:18 INFO:     feeder: unit0[2001:470:933e::5]:nntp: expected 
msg-id <4efed3fb$0$30390$a8266bb1 <at> newsreader.readnews.com> but got 431 
Expiring  process 3697

This server is running:

200 news.unit0.net InterNetNews server INN 2.5.2 ready (transit mode)

The problem seems to be the code for handling a paused server state in 
innd/nc.c:

    } else if (Mode == OMpaused) {
        cp->Check_deferred++;
        xasprintf(&buff, "%d %s", NNTP_FAIL_CHECK_DEFER, ModeReason);

ModeReason is set to "Expiring process %ld" while server expiry is 
running.

I tested this on my own server and was able to reproduce the problem:

isis# ctlinnd pause 'testing'
Ok
harmony% telnet isis 433
Trying 2a01:348:65::3...
Connected to isis.RT.UK.EU.ORG.
Escape character is '^]'.
(Continue reading)

Permalink | Reply |
headers
Julien ÉLIE | 1 Jan 2012 18:32
Favicon

Re: Invalid 431 response when server is paused

Hi River,

And Happy New Year 2012 to all the readers of this mailing-list!

> Jan 01 01:36:18 INFO:     feeder: unit0[2001:470:933e::5]:nntp: expected
> msg-id <4efed3fb$0$30390$a8266bb1 <at> newsreader.readnews.com>  but got 431
> Expiring  process 3697
> 
> The problem seems to be the code for handling a paused server state in
> innd/nc.c:
> 
>      } else if (Mode == OMpaused) {
>          cp->Check_deferred++;
>          xasprintf(&buff, "%d %s", NNTP_FAIL_CHECK_DEFER, ModeReason);

That's a pretty good catch.  Thanks for having reported it.
You're right to say there is an issue here.

I have just checked the other responses for innd in streaming mode, and
I believe it is the only one that has a problem.

Besides, getMsgId() in innfeed/misc.c properly finds the message-ID
as first argument, so that's fine to fix innd here.

> isis# ctlinnd pause 'testing'
> Ok
> harmony% telnet isis 433
> Trying 2a01:348:65::3...
> Connected to isis.RT.UK.EU.ORG.
> Escape character is '^]'.
(Continue reading)

Permalink | Reply |
headers
River Tarnell | 1 Jan 2012 21:58

Re: Invalid 431 response when server is paused

In article <4F0098AE.4010206 <at> trigofacile.com>,
Julien ÉLIE  <julien <at> trigofacile.com> wrote:
>> CHECK<a <at> b>
>> 431 testing

>With the patch:
>  http://inn.eyrie.org/trac/changeset/9393

>I now see the right behaviour:

>CHECK <test <at> test>
>431 <test <at> test> testing

Excellent :-) Thanks.

>Incidentally, does the feeder you are speaking about properly handle
>the 400 response code innd gives when paused and TAKETHIS is used?

It will handle it, but only in the same way it handles any unexpected[0]
response: it will disconnect, wait about 60 seconds, then reconnect and 
try again with the same message-id.  At the moment I never send a 
TAKETHIS without a prior CHECK, so if TAKETHIS <a <at> b> replies with 400, 
the next command in the new connection will be CHECK <a <at> b>.

[0] By which I mean "not 239 or 439".

>And, more subtle, does it handle 501 response codes?

No.  Or rather, it does the same thing as 400: it will try to re-send 
the article later.  So if the remote server is more strict about
(Continue reading)

Permalink | Reply |
headers
Julien ÉLIE | 2 Jan 2012 12:31
Favicon

Re: Invalid 431 response when server is paused

Hi River,

>> Incidentally, does the feeder you are speaking about properly handle
>> the 400 response code innd gives when paused and TAKETHIS is used?
>
> It will handle it, but only in the same way it handles any unexpected[0]
> response: it will disconnect, wait about 60 seconds, then reconnect and
> try again with the same message-id.
>
> [0] By which I mean "not 239 or 439".

Even 480 (authentication needed) or 483 (encryption needed)?

Anyway, this behaviour for 400 looks fine.

>> And, more subtle, does it handle 501 response codes?
>
> No.  Or rather, it does the same thing as 400: it will try to re-send
> the article later.  So if the remote server is more strict about
> message-ids than we are, the queue would block trying to send the same
> article forever.

That is unfortunately not the best thing to do with 501.  It can block a 
feeder.

> Of course, since INN doesn't actually send 501 at the moment, we avoid
> the problem here.

Yes, and I do not see well when this behaviour could be changed... 
Especially when reading your other mail in the NNTP working group about
(Continue reading)

Permalink | Reply |
headers
River Tarnell | 3 Jan 2012 01:20

Re: Invalid 431 response when server is paused

Julien ÉLIE:
> >It will handle it, but only in the same way it handles any unexpected[0]
> >response: it will disconnect, wait about 60 seconds, then reconnect and
> >try again with the same message-id.

> >[0] By which I mean "not 239 or 439".

> Even 480 (authentication needed) or 483 (encryption needed)?

Yes -- but again, only because I didn't get around to making it do 
something special with those.  At the moment it doesn't support 
authentication or encryption, so we couldn't do anything useful other 
than log an error anyway.

> >No.  Or rather, it does the same thing as 400: it will try to re-send
> >the article later.  So if the remote server is more strict about
> >message-ids than we are, the queue would block trying to send the same
> >article forever.

> That is unfortunately not the best thing to do with 501.  It can
> block a feeder.

I agree... however, while trying to fix it, I ran into a problem: I 
store separate queues for CHECK and TAKETHIS commands on the same 
connection.  That's fine normally because the response code indicates 
what kind of command it was, but 501 could be either.  

However it shouldn't be too hard to refactor a little to fix that.

> >Of course, since INN doesn't actually send 501 at the moment, we avoid
(Continue reading)

Permalink | Reply |
headers
Julien ÉLIE | 4 Jan 2012 00:18
Favicon

Re: Invalid 431 response when server is paused

Hi River,

> Currently I return 438 for invalid msg-id, but I may try changing it to
> 501 and see what happens.  I think I have peers with most of the
> commonly used server software (INN, Diablo and Cyclone, at least).

As far as INN is concerned, it currently (<= 2.5.3) does not handle 501 
response codes during streaming.
innfeed will try to feed again and again the same message if it receives 
501 in response to IHAVE, CHECK and TAKETHIS.  Anyway, innfeed isn't 
RFC3977-compliant yet.
innd does not send 501 for these three commands because of that.
Only nnrpd properly sends 501 for IHAVE.  (Using IHAVE with nnrpd is 
uncommon, so it should not do any real breaking nowadays.)

>>> For reference, the feeder source is here:
>>> <http://www.rt.uk.eu.org/cvs/rt/nts/feeder.c?view=markup>   (mostly the
>>> fe_running() function).

OK, thanks for the link.
If I understand well the code, I see that the first command you send is 
MODE STREAM, just after the initial greeting response.  Unless you 
receive 203, CAPABILITIES is sent.  I believe it is a workaround for 
Cyclone (probably amongst other servers).
Yet, the availability of IHAVE is still unknown if you do not explicitly 
search for it in fe_read_capabilities().  I would also search for the 
first keyword in each line, instead of matching the whole line.  (Maybe 
a future extension to the IHAVE and STREAMING capabilities could define 
arguments in the capability line for IHAVE and STREAMING; the code would 
then fail to find these facilities.)
(Continue reading)

Permalink | Reply |
headers
River Tarnell | 4 Jan 2012 01:41

Re: Invalid 431 response when server is paused

Julien ÉLIE:
> >Currently I return 438 for invalid msg-id, but I may try changing it to
> >501 and see what happens.  I think I have peers with most of the
> >commonly used server software (INN, Diablo and Cyclone, at least).

> As far as INN is concerned, it currently (<= 2.5.3) does not handle
> 501 response codes during streaming.

Hmm.  In that case maybe I'll leave it.

> If I understand well the code, I see that the first command you send
> is MODE STREAM, just after the initial greeting response.  Unless
> you receive 203, CAPABILITIES is sent.  I believe it is a workaround
> for Cyclone (probably amongst other servers).

Yes.  It seems a bit pointless to check CAPABILITIES if MODE STREAM 
failed, but the code was already there, so...

> Yet, the availability of IHAVE is still unknown if you do not
> explicitly search for it in fe_read_capabilities(). [...]
> I do not see where IHAVE is used.  It is still not implemented in
> RT/NTS, isn't it?

No.  I think when I do implement it, I'll require the admin to actually 
configure a peer to use IHAVE, and otherwise treat lack of streaming as 
an error.  Pretty much everyone supports streaming nowadays, so if it's 
not there, it's probably because we ended up talking to nnrpd for some 
reason; and if we connect to an nnrpd that happens to allow posting from 
us, we'll start injecting articles via nnrpd's IHAVE, which could be 
unfortunate (or at least, be really slow, while appearing to work).
(Continue reading)

Permalink | Reply |
headers
Julien ÉLIE | 4 Jan 2012 21:21
Favicon

Re: Invalid 431 response when server is paused

Hi River,

>> If I understand well the code, I see that the first command you send
>> is MODE STREAM, just after the initial greeting response.  Unless
>> you receive 203, CAPABILITIES is sent.  I believe it is a workaround
>> for Cyclone (probably amongst other servers).
>
> Yes.  It seems a bit pointless to check CAPABILITIES if MODE STREAM
> failed, but the code was already there, so...

If a server supports CAPABILITIES, I believe it will also support MODE 
STREAM.  Sending CAPABILITIES when MODE STREAM fails is therefore 
pointless, as you say, but who knows, maybe in a (far) future, it will 
be useful again owing to the following note in RFC 4644;

2.3.  MODE STREAM Command

    Servers MUST accept the
    MODE STREAM command for backwards compatibility with legacy clients
    that don't use the CAPABILITIES discovery command.

    NOTE: This command may be removed from a future version of this
    specification, therefore clients are urged to transition to the
    CAPABILITIES command wherever possible.

> if we connect to an nnrpd that happens to allow posting from
> us, we'll start injecting articles via nnrpd's IHAVE, which could be
> unfortunate (or at least, be really slow, while appearing to work).

An advantage of nnrpd's IHAVE is that it can benefit from the native
(Continue reading)

Permalink | Reply |
headers
River Tarnell | 4 Jan 2012 21:46

Re: Invalid 431 response when server is paused

Julien ÉLIE:
> An advantage of nnrpd's IHAVE is that it can benefit from the native
> STARTTLS extension nnrpd has (and therefore also compression TLS can
> do), or SASL authentication.

But this is probably outweighed by having to use IHAVE... Diablo 
supports a non-standard compression protocol using "MODE COMPRESS",
which I imagine is more likely to be supported by a transit server ;-)

> Is nnrpd's IHAVE "*really* slow"?  (I have not benchmarked it; I ask
> in case you already compared it to innd's IHAVE and had figures.)

Compared to INN's IHAVE, I don't know.  But it's definitely much slower 
than streaming.  (Perhaps someone should add streaming to nnrpd, and do 
away with innd entirely ;-)

> >(I recall some confusion about whether a line that starts
> >with "." but has further text on it should be escaped or not.)

> A confusion in an existing document?

No, rather someone (on ietf-nntp, I think) who was unclear about what 
exactly should be escaped.  Which given the ad-hoc nature of NNTP, means 
there could be a server out there that gets it wrong.  However I've 
never actually seen that in practice.

Regards,
--

-- 
        -- river.                      | Free Usenet: http://news.rt.uk.eu.org/
Non-Reciprocal Laws of Expectations:   | PGP: 2B9CE6F2
(Continue reading)

Permalink | Reply |
headers
Julien ÉLIE | 6 Jan 2012 20:31
Favicon

Re: Invalid 431 response when server is paused

Hi River,

>> Is nnrpd's IHAVE "*really* slow"?  (I have not benchmarked it; I ask
>> in case you already compared it to innd's IHAVE and had figures.)
>
> Compared to INN's IHAVE, I don't know.  But it's definitely much slower
> than streaming.  (Perhaps someone should add streaming to nnrpd, and do
> away with innd entirely ;-)

It is true that nnrpd only lacks streaming.  Yet, without innd, nnrpd 
does not do many things -- save reading of course.  nnrpd cannot inject 
articles; it just "pushes" articles to innd, which then do all the work.

>>> (I recall some confusion about whether a line that starts
>>> with "." but has further text on it should be escaped or not.)
>
>> A confusion in an existing document?
>
> No, rather someone (on ietf-nntp, I think) who was unclear about what
> exactly should be escaped.  Which given the ad-hoc nature of NNTP, means
> there could be a server out there that gets it wrong.  However I've
> never actually seen that in practice.

Something tricky is to properly count :bytes (for OVER and HDR 
responses).  Another caveat is not to forget to dot-stuff LIST MOTD, 
LIST ACTIVE.TIMES, LIST NEWSGROUPS, LIST DISTRIBUTIONS, HELP, etc. 
responses in case the server reads a file written by a user.

--

-- 
Julien ÉLIE
(Continue reading)

Permalink | Reply |

Gmane