Subject: [NNTP] LIST EXTENSIONS and an NNTPv2 capability

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Oct 20 08:01:57 2004

I wrote:
> I should have realized that what I was proposing was too big a chunk for
> people to swallow in one go. Let me try again, explaining what I am
> thinking.

[...]

> I have to stop writing as I need to be elsewhere. Can people please think
> on what I've written so far and comment, and I'll try to continue later
> today.

Okay, to continue ...

(5) If we provide version number information, whether for the main protocol
or for extensions, what form should it take?

There are no doubt many views on this. My personal one is:
* The "major.minor" approach, where changes of "major" are incompatible
  while changes of "minor" are upwards compatible, is well-known and
  probably suffices for our needs.
* A way of providing extra information could be useful, for example when
  a new version is under development and clients want to know if the
  draft is supported.
* We can't restrict non-standardised extensions, but they could also find
  extensioning useful.
But this is a detail once (and if) we've agreed that versioning is worth
doing in the first place.

(6) Is there benefit in providing the information "this extension is
supported by the server but isn't available right now"?

On the one hand servers might want to keep some extensions secret until the
client has authenticated. On the other hand, is there any point in going
through various probing and authenticating steps if the extension is going
to turn out not to be there?

There's also the anomaly that AUTHINFO wants to show what capabilities are
available even when it's going to reject any authentication attempt.

(7) Once we answer all these questions, how do we apply the answers?

Given the answers I've proposed, I think a new command that presents all
this information is the right way to go. But I'm still happy to discuss
everything.

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: ken at oceana.com (Ken Murchison)
Date: Sat Oct 23 08:24:23 2004

>>people to swallow in one go. Let me try again, explaining what I am

I'm curious as to what makes NNTP so different from IMAP/POP/SMTP that 
it requires an extension discovery mechanism that is orders of magnitude 
more complex.  These other protocols have been able to "survive" with a 
mechanism just as simple as the existing LIST EXTENSIONS + NNTPv2 
proposal.  In fact, IMAP is far more complex than NNTP and is able to 
get by with a very simple CAPABILITY command.

My fear is that we are trying to over engineer this to the point of 
solving a non-existent problem.  I'm also afraid that this level of 
complexity will lead to implementation delays or no implementations at all.

This is another case where we need client vendors to determine what is 
necessary.  The only client feedback that I have seen is from Mark 
Crispin, whose c-client library is used as the core of Pine (and other 
packages like PHP) and supports all 4 protocols discussed above.  His 
quote to me (he's not on the mailing list so he may or may not have 
posts sitting in the moderator queue) regarding the STATUS proposal was 
"It is unnecessary and needlessly complex.  It should not happen."

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Oct 26 09:20:13 2004

> get by with a very simple CAPABILITY command.

Firstly, I'm not sure that NNTP is significantly different (but see below).
However, I suspect that it's simply that none of them have run into the
problems I discuss *yet*. And SMTP *has* sort of had the versioning
problem, with the HELO v EHLO commands.

I also don't accept "orders of magnitude".

My concerns, as explained in my previous messages, split into two areas:

(1) The ability to handle variations - version numbers - of extensions.
This *will* happen to lots of protocols, even if it hasn't happened yet.

(2) NNTP's unique (as far as I know) concept of "extensions that aren't
available until you do something separate".

The two can be considered orthogonally. Each, if you accept it is an issue,
introduces a need for something more, though not as much more as you might
think.

> complexity will lead to implementation delays or no implementations at all.

I went through my views in detailed steps. If you disagree with them, I'd
still like to know where; I think that will help move the discussion on.

> "It is unnecessary and needlessly complex.  It should not happen."

I'd like his feedback as well. I'm unconvinced of "unnecessary", but I'd
like to know where we diverge. As for "needlessly complex", all it does is
adds a version number and an "available or not" flag to each extension.
If these are needed, then it's pretty minimal. If not, then let's debate
why not.

---

From: ken at oceana.com (Ken Murchison)
Date: Tue Oct 26 10:44:05 2004

> problem, with the HELO v EHLO commands.

Well *sort* of.  The EHLO command was introduced as the extension 
discovery command (akin to LIST EXTENSIONS).  Prior to EHLO, there was 
no extension discovery mechanism.  So your example is kind of a mixed bag.

I think where we diverge is that the NNTPv2 proposal follows the KISS 
principal and only solves the existing problem(s); by putting a line in 
the sand w.r.t. base functionality and existing extensions which we are 
standardizing.  Your proposal wants to try to solve any problems with 
extensions moving forward.  My response to this is:

1)  Based on past history of IMAP/POP3/SMTP, few, if any, extensions 
will have their functionality changed in any significant way.

2)  If the functionality *is* changed in a drastic, non-backwards 
compatible way, then a new extension should be advertised rather than 
trying to wedge it into the existing extension.

3) If the previous two assumptions turn out to be false, then we can 
always address this issue at a later time.  The existing LIST EXTENSIONS 
syntax already allows for extension labels to take arguments, which 
could be used for versioning and/or enabled/disabled flags.

> I also don't accept "orders of magnitude".

I used this phrase in reference to your original STATUS proposal which 
seemed overwhelmingly complex.  It seems that you have pared down your 
proposal since then.  In that respect I overstated things a bit.

> This *will* happen to lots of protocols, even if it hasn't happened yet.

I agree that functionality *may* change for certain extensions, but my 
guess is that the WGs responsible for the other protocols would just 
create a new extension or separate extension label/capability.

> available until you do something separate".

Well, this is *always* the case with the SASL PLAIN mechanism, which 
requires TLS before being advertised and/or used.  None of the other 
messaging protocols prevent a server from hiding extensions until the 
client jumps through the appropriate hoop(s).  In fact I think Mark's UW 
IMAP server hides most extensions until the client is authenticated.

I don't see this being an issue for clients.  They already know before 
they issue a command (e.g. AUTHINFO, STARTTLS, MODE READER) that it has 
the possibly of changing the output of LIST EXTENSIONS and should be 
prepared to [re-]issue LIST EXTENSIONS.

If we're interested in saving roundtrips, then we could always look at 
Mark's proposal (which I haven't seen get to the list yet) of allowing 
relevant responses to carry extension info.  IMAP already allows this. 
For example:

200 [NNTPv2 AUTHINFO=USER AUTHINFO=SASL:OTP,DIGEST-MD5,CRAM-MD5,NTLM 
STARTTLS STREAMING] toshiba.oceana.com Cyrus NNTP v2.3-alpha server 
ready, posting allowed
STARTTLS
382 Begin TLS negotiation now
LIST EXTENSIONS
202 Extension list follows:
NNTPv2
AUTHINFO USER SASL:LOGIN,OTP,DIGEST-MD5,CRAM-MD5,PLAIN,NTLM
STREAMING
.
GROUP misc.test
480 Authentication required
AUTHINFO SASL CRAM-MD5
383 PDU0NjI1NDYxMC43Nzc0NjE1QHRvc2hpYmEub2NlYW5hLmNvbT4=
bXJjIGRlMDhmZTVjYmQwZDEwNGRlNzY0YWM3NDhlNmFiNTRh
281 [NNTPv2 HDR LISTGROUP OVER STREAMING] Success (tls protection)
GROUP misc.test
211 0 1 0 misc.test

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 3 15:39:05 2004

On Tue, 26 Oct 2004, Ken Murchison wrote:
> principal and only solves the existing problem(s); by putting a line in the 
> sand w.r.t. base functionality and existing extensions which we are 
> standardizing.

I also adhere to this principle.

> 1)  Based on past history of IMAP/POP3/SMTP, few, if any, extensions will 
> have their functionality changed in any significant way.
> 2)  If the functionality *is* changed in a drastic, non-backwards compatible 
> way, then a new extension should be advertised rather than trying to wedge it 
> into the existing extension.
> 3) If the previous two assumptions turn out to be false, then we can always 
> address this issue at a later time.  The existing LIST EXTENSIONS syntax 
> already allows for extension labels to take arguments, which could be used 
> for versioning and/or enabled/disabled flags.

The only thing that I can add to this is a big
 	AMEN!!!!!

>> available until you do something separate".

This is not unique to NNTP.  As Ken pointed out, TLS will change the 
available extensions, and (at least in IMAP) extensions will change after 
authentication.

> If we're interested in saving roundtrips, then we could always look at Mark's 
> proposal (which I haven't seen get to the list yet) of allowing relevant 
> responses to carry extension info.

Please do.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.