Subject: [NNTP] Re: [ietf-nntp] draft-ietf-nntpext-streaming-01.txt

From: rra at stanford.edu (Russ Allbery)
Date: Mon Oct 4 12:48:54 2004

Ken Murchison <[email-masked]> writes:

> I'm bored so I'm spending time updating this doc to use similar
> formatting as AUTHINFO and STARTTLS.  I decided to take a look back at
> the list traffic and see what the outstanding issues are.

Moving this over to the new list.  Apologies for all the quoting, but I
wanted to make sure that Ken's full message was included.

> Russ Allbery wrote:

>> I go back and forth on this, but I think we should take the stance that
>> there is no technical reason for this command and that the only reason
>> why it's included is that all current streaming implementations send it
>> and the server needs to do something sensible with it.  In other words,
>> include it, state that clients that intend to use streaming commands
>> MUST send either it *or* LIST EXTENSIONS, and state that servers that
>> support streaming commands for those clients MUST NOT require MODE
>> STREAM be sent.

>> I'd at least like to try that and see if it goes over okay with
>> implementors.

>> I don't know of an implementation that would inherently require that
>> this command be sent (unlike with MODE READER).  In practice, it was
>> used instead of LIST EXTENSIONS because sending TAKETHIS without a
>> positive indication that the remote side supports it is a bad idea.
>> Now, clients can just use LIST EXTENSIONS for that purpose.

> Agreed on all of the above.

> Question:

> Should any of the STREAMING commands be allowed after MODE READER?  I'm
> assuming that INN (and possibly other servers) would have a problem
> switching from reader-only mode back to streaming mode.

I think we can handle this via LIST EXTENSIONS, yes?  I expect that most
implementations that do something with MODE READER will not advertise the
STREAMING extension after MODE READER, but there's no particular reason
not to allow them to if they want to.

>> TAKETHIS has no deferral return status.  We could add one as part of
>> this standardization process; it's not a bad idea.

>> If we want to add a deferral, we need to use 432 or some similar code.
>> I don't think we need two types of deferrals; all streaming deferrals
>> should be delayed at least a little while.

> Is this something that we want to add to the document?

I don't see any obvious reason not to add it.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Oct 4 12:56:29 2004

> wanted to make sure that Ken's full message was included.

Oops!  Sorry about that.  I forgot that this thread was old enough to 
have started on the old list.

> not to allow them to if they want to.

Good point.  So we should add text that states that the server MAY 
choose to not advertise STREAMING after MODE READER.

> I don't see any obvious reason not to add it.

So, is 432 the definitive code?

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: rra at stanford.edu (Russ Allbery)
Date: Mon Oct 4 13:01:07 2004

>>>> If we want to add a deferral, we need to use 432 or some similar
>>>> code.  I don't think we need two types of deferrals; all streaming
>>>> deferrals should be delayed at least a little while.

> So, is 432 the definitive code?

432 is, according to my records, previously unused.  (Which is somewhat
suspicious -- I wonder why it was skipped over originally.)

431 is used for a deferral after CHECK, but we shouldn't reuse that code
since a deferral after TAKETHIS is different and the two may be
intermingled.  436 is used as the deferral for IHAVE, both as an initial
response and after the article was sent.  We could potentially use it, but
I'm not sure if that's a good idea or not.

I don't see any obvious Google references to an existing meaning for 432.

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Oct 4 13:05:07 2004

> I'm not sure if that's a good idea or not.

Actually (maybe I'm being thick and/or short-sighted), but does it 
matter whether we're getting deferred via CHECK or TAKETHIS?  In either 
case we're going to have to try again.

---

From: rra at stanford.edu (Russ Allbery)
Date: Mon Oct 4 13:07:28 2004

>> 431 is used for a deferral after CHECK, but we shouldn't reuse that
>> code since a deferral after TAKETHIS is different and the two may be
>> intermingled.  436 is used as the deferral for IHAVE, both as an
>> initial response and after the article was sent.  We could potentially
>> use it, but I'm not sure if that's a good idea or not.

> case we're going to have to try again.

Hm.  That's a good point.  Maybe it doesn't, actually.  Can anyone else
think of a reason why it might matter?  (It would matter for success or
rejection, since the sender of CHECK and TAKETHIS commands is not required
to keep track of order or figure out if a particular response is in reply
to CHECK or to TAKETHIS, but it seems like the behavior would be the same
either way for a deferral.)

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Oct 4 13:13:15 2004

> either way for a deferral.)

The more I think about this, isn't 439 essentially a deferral?  It says 
the transfer failed, but why?  Does mean that we should retry or does 
this mean that we rejected the article (e.g. don't send it again).

I think what we need is a deferral (temp fail) code and a rejection 
code.  We might be able to use 431 and the former and 439 as the latter. 
  Or if 439 already means deferral, then we can probably reuse 438 as a 
rejection code.

My guess is that  439 corresponds to 436 (second stage) and its a temp 
failure.

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Oct 4 13:31:46 2004

> My guess is that  439 corresponds to 436 (second stage) and its a temp 

Just to clarify, here's a table of current codes and meanings as I read 
them, and proposed codes where I have "[NNN]".  "NA" means not applicable.

				IHAVE	CHECK	TAKETHIS
transferred OK:			235	NA	239
send article:			335	238	NA
article not wanted:		435	438	[438]
transfer not possible (defer?):	436	431	[431]
transfer failed (defer?):	436	NA	439
transfer rejected:		437	NA	[438]

---

From: rra at stanford.edu (Russ Allbery)
Date: Mon Oct 4 14:27:24 2004

> this mean that we rejected the article (e.g. don't send it again).

439 is a permanent rejection.  The article should not be sent again.

> code.  We might be able to use 431 and the former and 439 as the
> latter. Or if 439 already means deferral, then we can probably reuse 438
> as a rejection code.

> My guess is that 439 corresponds to 436 (second stage) and its a temp

We definitely need to make it clearer that that's not the case, then.  :)

---

From: rra at stanford.edu (Russ Allbery)
Date: Mon Oct 4 14:29:16 2004

> transfer rejected:                    437	NA	[438]

Article not wanted is 439 for TAKETHIS as previously noted, and similarly
for transfer rejected.  Both of the middle ones are deferral, and as noted
before I'm not sure that having two separate codes there is needed.  This
is what should be 438 (or 432 or 436) for TAKETHIS.

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Oct 4 16:20:48 2004

> is what should be 438 (or 432 or 436) for TAKETHIS.

OK, something like this?

			IHAVE	CHECK	TAKETHIS
xferred OK:		235	NA	239
send article:		335	238	NA
not wanted:		435	438	[438]
xfer not possible:	436	431	[431]
xfer failed:		436	NA	[431]
xfer rejected:		437	NA	439

Simply reusing 431 and 438 for TAKETHIS.  This might invalidate your 
argument that the response codes need to be unique between CHECK and 
TAKETHIS, but I don't see where this breaks anything.   My guess is that 
a client wouldn't be pipelining CHECK and TAKETHIS together.  The client 
will pipeline the CHECKs, and process the results.  Then based on these 
results, it will pipeline TAKETHIS.  Am I missing something?  Hmm,
even if a client *does* interleave CHECK and TAKETHIS this might still 
work (with a side-effect of trying TAKETHIS twice on an article that the 
server doesn't want).

---

From: rra at stanford.edu (Russ Allbery)
Date: Mon Oct 4 16:25:57 2004

> Simply reusing 431 and 438 for TAKETHIS.

If I understand your table correctly, "not wanted" is the decision made
before the article is sent, which isn't possible for TAKETHIS.  So that
column would be NA for TAKETHIS.  I think we only have one new code that
we need to come up with, where you currently have [431].

> This might invalidate your argument that the response codes need to be
> unique between CHECK and TAKETHIS, but I don't see where this breaks
> anything.  My guess is that a client wouldn't be pipelining CHECK and
> TAKETHIS together.  The client will pipeline the CHECKs, and process the
> results.  Then based on these results, it will pipeline TAKETHIS.  Am I
> missing something?  Hmm, even if a client *does* interleave CHECK and
> TAKETHIS this might still work (with a side-effect of trying TAKETHIS
> twice on an article that the server doesn't want).

I think, with the above correction, that this will work okay unless
clients really want to handle TAKETHIS deferrals different than CHECK
deferrals but don't want to do the bookkeeping to keep track of whether
the article was offered with TAKETHIS or with CHECK.  The really important
part is to keep 238, 438, and 439 distinct.

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Oct 4 18:37:44 2004

Russ Allbery wrote:
> column would be NA for TAKETHIS.

I still think that "not wanted" (a.k.a. "already have this msgid") is 
still valid for TAKETHIS.  This could occur because a client tries to 
send an article immediately with TAKETHIS (w/o having done CHECK first) 
or because of the race condition between running CHECK and TAKETHIS 
(e.g. another process accepts the particular message in the interval). 
Having said that, 439 would suffice for this case, but I look at that as 
"not accepting articles for this group" rather than "already have this 
msgid".

I'll defer to the consensus on whether to document 438 for TAKETHIS.

> I think we only have one new code that

[text removed]

> part is to keep 238, 438, and 439 distinct.

I'll go with 431 for now until someone can come up with a case for it 
having to be unique.

Actually the "xfer not possible" and "xfer failed" cases are one in the 
same for TAKETHIS, so I'll just document 431 as "xfer failed".

---

From: rra at stanford.edu (Russ Allbery)
Date: Mon Oct 4 18:44:03 2004

Ken Murchison <[email-masked]> writes:

> (e.g. another process accepts the particular message in the
> interval). Having said that, 439 would suffice for this case, but I look
> at that as "not accepting articles for this group" rather than "already
> have this msgid".

> I'll defer to the consensus on whether to document 438 for TAKETHIS.

Well, I've never seen an implementation that did that; all the
implementations I'm aware of send 439 for that case.  There is no
distinction drawn in the protocol between duplicate messages and messages
rejected for any other reason, just between messages rejected before
they're sent and messages rejected after they're sent.  (They could be
rejected before they're sent for other reasons than being duplicates --
for example, INN can be configured to query an external filter for each
message ID sent via CHECK or IHAVE and allow rejections in advance based
on some external knowledge.)  Reports and statistics are based on that
distinction too (for example, innfeed's statistics report the counts of
439 responses vs. 438 responses, intending them to be used in the above
sense).

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Oct 12 15:17:18 2004

Ken Murchison <[email-masked]> writes:

> After thinking about this some more, should we even bother discussing
> MODE STREAM as a formal part of the STREAMING extension?  Since we all
> agree that MODE STREAM has been historically used as a capability
> discovery mechanism, which is now deprecated by LIST EXTENSIONS, what's
> the point of formalizing it?  I can see possibly documenting it as an
> obsolete command somewhere in the document, but including it as part of
> the STREAMING extension seems wrong.

I'm worried about transition issues.  I believe there are implementations
that will reject CHECK/TAKETHIS unless MODE STREAM has been sent first, so
I'd like to warn implementors that for backward compatibility, they should
implement MODE STREAM on the receiving end even if it's a no-op and send
it before using CHECK/TAKETHIS.  (While strongly discouraging this
implementation method in the future, of course.)

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Oct 12 15:34:42 2004

> implementation method in the future, of course.)

I guess what I'm getting at is is MODE STREAM a formal part of the 
STREAMING extension, or just an obsolete command that some 
implementations may need for backwards compatibility.

It just rubs me the wrong way to formalize a command that we're trying 
toi deprecate.  Of course, I guess we're doing this already aith 
AUTHINFO USER/PASS.

---

From: rra at stanford.edu (Russ Allbery)
Date: Tue Oct 12 15:36:09 2004

> implementations may need for backwards compatibility.

I consider it to be the latter, but I do think that people need to
implement it for the time being.

> toi deprecate.  Of course, I guess we're doing this already aith

Yeah, the case seems somewhat analogous.

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Oct 13 09:49:13 2004

> implementation method in the future, of course.)

Here's the text that I've come up with for MODE STREAM:

2.2. MODE STREAM Command

      This command is deprecated and is only documented for compatibility
      with legacy implementations of the STREAMING extension.  New client
      and server implementations are not required to support this command
      but they are encouraged to do so.

2.2.1. Usage

      This command MUST NOT be pipelined.

      Syntax
         MODE STREAM

      Responses
         203   Streaming permitted

2.2.2. Description

      A client issues the MODE STREAM command to indicate that it wishes
      to use the CHECK and TAKETHIS commands instead of, or in addition
      to, IHAVE.  If the server supports the CHECK and TAKETHIS commands
      a 203 response MUST be returned.

      If a server advertises the STREAMING extension label in the LIST
      EXTENSIONS response, the MODE STREAM command is not required and a
      client SHOULD NOT issue it.  If a legacy server does not support
      the LIST EXTENSIONS command or does not advertise the NNTPv2
      extension label, a client SHOULD issue the MODE STREAM command
      prior to using the CHECK or TAKETHIS commands.

      A server MUST NOT require that the MODE STREAM command be issued
      before accepting the CHECK or TAKETHIS commands.  A server SHOULD
      accept the MODE STREAM command for compatibility with legacy
      clients which don't use the LIST EXTENSIONS discovery mechanism.

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Oct 13 10:33:11 2004

> Here's the text that I've come up with for MODE STREAM:

This looks great to me.

---

From: chl at clerew.man.ac.uk (Charles Lindsey)
Date: Wed Oct 13 19:12:25 2004

>implementations may need for backwards compatibility.

It its situation in any way different from MODE READER? Do we not regard
that as unnecessary, but continue to do it because some implementations
require it? If the situation is similar, then it should be handled in the
same way.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Oct 13 19:14:12 2004

> It its situation in any way different from MODE READER?

Yes, significantly.  We can simply make MODE STREAM go away going forward
(there's no protocol need for it now that we have LIST EXTENSIONS), but
can't do that with MODE READER without outlawing real-world implementation
choices.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Fri Oct 15 03:44:31 2004

[Still catching up, I'm afraid.]

> still valid for TAKETHIS.

I think the table is being looked at wrongly. If you split the two stages
of IHAVE, it becomes more obvious:

                         IHAVE-1   IHAVE-2   CHECK     TAKETHIS
    send article:          335        -       238         -
    not wanted:            435        -       438         -
    xfer not possible:     436        -       431         -
    xferred OK:             -        235       -         239
    xfer failed, retry:     -        436       -        [???]
    xfer rejected:          -        437       -         439

The code in brackets could be 431 (dual-use, just like 436), or it can be a
separate code (on the "one situation one code" principle).

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Oct 15 06:30:28 2004

> separate code (on the "one situation one code" principle).

Yes, your split table illustrates the point better.  The -02 draft uses 
432 to fill the hole, but I'm not convinced that it can't be 431.

Would a client really intermingle CHECK and TAKETHIS within the same 
pipeline of commands?  If not, then why do the response codes need to be 
unique?

Perhaps I just don't understand how CHECK/TAKETHIS is currently used in 
the wild.

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Fri Oct 15 11:20:52 2004

>>>>> "Ken" == Ken Murchison <[email-masked]> writes:

 Ken> Would a client really intermingle CHECK and TAKETHIS within the
 Ken> same pipeline of commands?

Absolutely. It's the normal case, in fact.

-- 
Andrew, Supernews
http://www.supernews.com

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Oct 15 12:34:18 2004

> Absolutely. It's the normal case, in fact.

Can you explain?

I assumed that the client would first pipeline a series of CHECK 
commands to build a list of articles that the server wants.  Then the 
client sends the articles with a pipeline of TAKETHIS commands.

---

From: fantec at proxad.net (Francois Petillon)
Date: Fri Oct 15 15:00:16 2004

> client sends the articles with a pipeline of TAKETHIS commands.

No, this is not optimal on transmission delays. IMHO, after starting a 
new stream by sending a few CHECKs, you should then send a new CHECK
for each answer you get. And if you have to send a TAKETHIS (followed
by the article), then you have to send the CHECK before the TAKETHIS.

   Fran?ois

---

From: andrew at supernews.net (Andrew - Supernews)
Date: Fri Oct 15 15:04:47 2004

>> Absolutely. It's the normal case, in fact.

 Ken> Can you explain?

 Ken> I assumed that the client would first pipeline a series of CHECK
 Ken> commands to build a list of articles that the server wants.
 Ken> Then the client sends the articles with a pipeline of TAKETHIS
 Ken> commands.

The client never actually stops sending CHECKs. Remember that the
objective is to keep the pipe full at all times, unless the supply of
articles to send dries up (which is not usually a problem). The
entire transfer session is just one big pipeline.

-- 
Andrew, Supernews
http://www.supernews.com