Subject: [NNTP] LIST EXTENSIONS and an NNTPv2 capability

From: rra at stanford.edu (Russ Allbery)
Date: Tue Oct 12 15:06:30 2004

Okay, let's see if I can explain this in a way that makes it both fairly
simple and shows why people want it.  I'm going to cc Mark Crispin in the
hope that he'll be willing to clarify any portions of this that I didn't
quite get right.

I'm sorry that this has taken me so long; I have rather too many balls in
the air just at the moment and am scrambling desperately to try to keep
track of all of them.

The way that a proper client of a protocol that supports an extension
mechanism should behave is that immediately after connecting, it inquires
as to the capabilities of the server.  Subsequently, it assumes those
listed capabilities are accurate and does not attempt to use any
capabilities that were not advertised.  I realize that this is not how
NNTP clients have ever worked in the past -- this is something that we're
seeking to change by adding a proper extension mechanism.  This is how
everyone else does it (IMAP, POP, SMTP, etc.), and we're trying to bring
NNTP in line with other Internet protocols.

Given this new approach, and given the way a client should be implemented
when complying with our draft, there are two problems with the LIST
EXTENSIONS mechanism as currently present in our base draft.  After
talking with Mark Crispin about this, I think that this is something that
we can and should correct before the draft is published, since we do have
the opportunity.

The first problem is that, since LIST EXTENSIONS has been around as an
idea for quite some time, there is at least one known badly bogus
implementation.  Netscape Collabra implements LIST EXTENSIONS, but the
information that it returns cannot be relied upon.  In particular, it
advertises OVER but doesn't implement OVER properly.  We'd previously
discussed this problem and had tentatively decided to live with it, but
there's a better solution described below.

The second problem is that, based on our drafts, at least one
implementation (INN) and possibly others have started adding LIST
EXTENSIONS.  However, the extension list was not finalized, and in
particular AUTHINFO was not advertised by the early INN implementation
even though AUTHINFO USER was supported.  A client that uses the extension
mechanism properly would therefore think that those servers didn't support
authentication when they do.

The proposed solution is this:  Specify in our base draft that any server
that fully complies with the new specification advertise the NNTPv2
capability as part of LIST EXTENSIONS.  Clients can then ignore or treat
with dubiousness any server that doesn't advertise that capability, thus
solving the Collabra problem.

We also need to somehow specify that any server that implements AUTHINFO
USER needs to advertise it if it advertises the NNTPv2 capability.  I'm
not completely sure of the best way to do that; one way would be to
specifically mention the three legacy extensions that we're standardizing
(AUTHINFO, STARTTLS, and CHECK/TAKETHIS or STREAMING), and another would
be to require that the server advertise an extension for any commands that
don't begin with X.  A third approach would be to recognize that this
blessing of widely implemented commands that predated the extension
mechanism is a one-time transition.  This probably requires some more
discussion to find the best approach from a transition standpoint.

However, the goal is to make LIST EXTENSIONS reliable, including for
everything that we're blessing under its existing command names, so that
clients can just read the output and then use that information without
having to second-guess it.  Without doing this, LIST EXTENSIONS is both
badly different from the way that other protocols use it and not actually
useful for clients.

I think this approach has a lot of merit, and that it's easy enough to add
the NNTPv2 requirement to the existing draft.  I also think that this is a
minor enough change that it's worth doing this late in the process,
although I'd appreciate it if Scott would check me on this.
(Unfortunately, Mark had missed the IETF-wide last call notification and
this issue didn't come up until afterwards.)  Note that this method of
advertising a protocol version in the extensions output has also been used
successfully with IMAP.

Thoughts?

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Oct 13 08:00:38 2004

Russ Allbery wrote:

[text removed]

I believe you have stated the problem(s) accurately.

> solving the Collabra problem.

I'd submit that LIST EXTENSIONS is no longer an optional command for the 
server.  Any server compliant with with RFC 977bis MUST implement LIST 
EXTENSIONS and MUST advertise at least the NNTPv2 capability, even if no 
extensions are implemented.  This serves to tell the client that 
previously optional/non-standard commands that we have standardized as 
part of the base specification (e.g. DATE) are supported.

> discussion to find the best approach from a transition standpoint.

First, I think that the base doc and/or any extension docs have to make 
it clear that if a NNTPv2 server implements an extension it MUST 
advertise the appropriate extension label via LIST EXTENSIONS.

As far as the legacy extensions which we are standardizing, we need to 
document somewhere that any server which supports one or more of these 
extensions and is upgraded to be compliant with RFC 977bis MUST also be 
upgraded to be compliant with the standardized versions of these 
extensions (including advertising it where appropriate).  This is the 
only way that I can see that a client which sees NNTPv2 advertised will 
know if these extensions are supported and how they will behave.

Unfortunately, I don't know where this should be documented.  I don't 
think we can put it in the base doc since we don't want the base doc 
referencing extension docs.  Putting it in the extensions docs doesn't 
really help because a server author may fail to reference them when 
upgrading to RFC 977bis compliance.  Perhaps the best thing would be to 
have a companion (Informational) document to RFC 977bis discussing 
compatibility/upgrade issues.

-- 
Kenneth Murchison     Oceana Matrix Ltd.
Software Engineer     21 Princeton Place
716-662-8973 x26      Orchard Park, NY 14127
--PGP Public Key--    http://www.oceana.com/~ken/ksm.pgp

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Oct 13 08:11:14 2004

> know if these extensions are supported and how they will behave.

This pretty much requires us to have the base doc and the AUTHINFO, 
STARTTLS, and STREAMING extension docs progress simultaneously.

The AUTHINFO and STARTTLS docs are almost ready for last call, and I'm 
working hard to get a -02 draft for STREAMING out the door which 
hopefully would take no more than one additional revision before last call.

> compatibility/upgrade issues.

Hmm, after thinking about this some more, maybe we can just state in the 
base doc that any server which is upgraded to RFC 977bis SHOULD also 
consult the IANA registry of NNTP extensions and MUST simultaneously 
upgrade any legacy extensions that it already supports.

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Oct 13 10:32:13 2004

>> The proposed solution is this:  Specify in our base draft that any
>> server that fully complies with the new specification advertise the
>> NNTPv2 capability as part of LIST EXTENSIONS.  Clients can then ignore
>> or treat with dubiousness any server that doesn't advertise that
>> capability, thus solving the Collabra problem.

> part of the base specification (e.g. DATE) are supported.

Oh, yeah, I forgot that.  Yes, LIST EXTENSIONS would also become a
mandatory command.

> know if these extensions are supported and how they will behave.

Yup.  That makes sense.

> compatibility/upgrade issues.

This was exactly the same problem that I had, but I really don't want to
create a completely separate document; that will set us back a while.
Referencing the extensions in the base document may be the best option
right now, given the upgrade situation that we're running into.

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Oct 13 10:32:44 2004

> upgrade any legacy extensions that it already supports.

Oh, that's a good idea.  That could also work, yes.

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Oct 13 11:47:25 2004

> solving the Collabra problem.

Hmm, I just noticed that NNTPv2 isn't a valid extension label per the 
latest grammar since extension-label is defined to be 3*12UPPER.  We'll 
either have to come up with a new label or relax the grammar.  Since I 
originally suggested NNTPv2, I'm in favor or relaxing the grammar and 
perhaps making extension-label 3*12A-CHAR.  Is there a reason why this 
wasn't done in the first place?

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Oct 13 12:48:01 2004

> wasn't done in the first place?

I think we were worried about case sensitivity causing problems.

---

From: chl at clerew.man.ac.uk (Charles Lindsey)
Date: Wed Oct 13 19:12:24 2004

>solving the Collabra problem.

I am not particularly enamoured of this idea. Presumably the two
implementations that currently do it wrong will shortly be mended to do it
right. So the alternative of putting up with the situation for a hopefully
shortish transitional period can also be considered.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: rra at stanford.edu (Russ Allbery)
Date: Wed Oct 13 19:15:59 2004

Charles Lindsey <[email-masked]> writes:

> implementations that currently do it wrong will shortly be mended to do
> it right. So the alternative of putting up with the situation for a
> hopefully shortish transitional period can also be considered.

That was raised, but Mark Crispin rightfully pointed out that this is far
too optimistic about the lifespan of existing deployed software, and would
make LIST EXTENSIONS significantly less useful to well-designed clients
for a long time going forward.  After thinking about it for a while, I
agree with him; this is simple and straightforward both to describe and to
implement, matches existing practice with other protocols, has other
merits (since we did tweak some other minor things in the protocol), and
doesn't really have any drawbacks that I can see.

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Oct 13 20:33:04 2004

> shortish transitional period can also be considered.

As Mark pointed out offline (and I agree), simply doing nothing won't 
work because at what point can the client start trusting the output of 
LIST EXTENSIONS?  If there is no definitive way for a client to 
determine that LIST EXTENSIONS follows the updated NNTP spec, the 
command is essentially useless, as clients will *always* have to resort 
to trial and error.

LIST EXTENSIONS, or any other capability discovery mechanism, should not 
be designed or implemented (willingly or otherwise) to be purely 
advisory.  It should be guaranteed that its output is both complete and 
deterministic.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Fri Oct 15 02:04:32 2004

Ken Murchison said:
> EXTENSIONS

Agreed. Or my proposed STATUS command.

> and MUST advertise at least the NNTPv2 capability, even if no 

This is the versioning issue I discuss in my other message.

> advertise the appropriate extension label via LIST EXTENSIONS.

Agreed.

> extensions (including advertising it where appropriate).

Or indicate that they're providing pre-standard versions. Again, I've shown
how to do this in my proposal.

> upgrading to RFC 977bis compliance.

We can't force someone to upgrade their support of an extension. All they
have to do is declare it to be a private extension whose commands happen to
have the same name. Better to recognise this than to pretend it doesn't
happen. There's a similar issue with transition from one version of a
specification to the next one; both of these are addressed in my STATUS
proposal.

[I realize this sounds a bit like "STATUS solves everything" boasting, but
it isn't; the issues in this thread drove a lot of my design.]

-- 
Clive D.W. Feather  | Work:  <[email-masked]>   | Tel:    +44 20 8495 6138
Internet Expert     | Home:  <[email-masked]>  | Fax:    +44 870 051 9937
Demon Internet      | WWW: http://www.davros.org | Mobile: +44 7973 377646
Thus plc            |                            |

---

From: clive at demon.net (Clive D.W. Feather)
Date: Fri Oct 15 02:04:40 2004

> NNTP in line with other Internet protocols.

Okay, agreement so far.

> EXTENSIONS mechanism as currently present in our base draft.
[...]
> implementation.

Right.

> authentication when they do.

This is *always* going to be a problem in the stage between an extension
being invented and the specification being finalised. Agreed it's rather
worse for the stuff that has been around for a while, but it's not unique
to them.

> capability as part of LIST EXTENSIONS.

Right, stop here. I don't like this idea. It's a kludge, and an unstructured
one at that. However, it does identify a genuine problem.

Having read this message and the following thread, I did a lot of thinking.
I also harked back to some of the bitter experiences we had in standardising
the C language.

Firstly, let's recognise that, by doing this, *no* existing server is going
to comply with the new specification without being changed. This is *NOT* a
problem, it's an opportunity. What we are doing at this point is to draw a
line in the sand (or in history, if you like). So let's look at the whole
problem and fix it properly.

======== Line in the sand :-) ========

Rather than saying that LIST EXTENSIONS is unreliable unless the server
sets the "I got it right, honest" flag, I think we're better off having a
new command. There's little downside because server implementers are going
to have to make changes anyway. This can then also fix other problems.

The two big issues I can see that are going to bite us one day (if indeed
they haven't already) are:

(1) There's no good way to indicate conformance to particular versions of
extension protocols. Adding a versioning facility usually turns out to be
a good idea in the long run.

(2) It's better to indicate what capabilities are available altogether,
rather than just what are available right now. It's better for a client to
know "you can do TLS once you've done MODE READER" than for it to have
to guess that doing MODE READER might make STARTTLS magically appear.

So let's fix these now, while we have the chance.

Let me present my idea first as a bit of protocol:

    [C] STATUS
    [S] 101 1.0     List of available extensions follows
    [S] OVER 1.0 + MSGID
    [S] HDR 1.0-RAA +
    [S] LISTGROUP 1.0 -MODE-READER
    [S] STARTTLS 0.0-02 -502
    [S] AUTHINFO 0.0-05-x -502 USER SASL:PLAIN,EXTERNAL
    [S] STREAMING 0.0-perhaps +
    [S] XSECRET -CDWF -483
    [S] XDATA -CDWF +
    [.]

The specification is roughly as follows.

The command cannot fail, apart from the generic 400 and 501 responses. The
argument to the 101 response code is a version-code; see below.

Each line in the extended response describes one extension, as with LIST
EXTENSIONS. However, the line gives more detail; there are always at least
three words:
* The extension name. This becomes case-insensitive just like the rest of
  NNTP, but we continue to require letters only.
* A version-code. This indicates the conformance status of the extension;
  I discuss it further below.
* A status code. This is either "+" to show that the extension is
  *currently* available, or "-" followed by a reason to show that it isn't.
  For reasons, I've used "502", "480", or "483" if use of the extension
  would generate that response, or an extension name (or "MODE-READER")
  if use would generate a 401 response.
Any further words are parameters to the extension, as with LIST EXTENSIONS.

The number of lines in the output does not change during a session; as
extensions become available or unavailable, the status changes instead.
(However, I'd make this a SHOULD rather than a MUST so that servers can
still have secret extensions if they want.) This allows a client to
determine what *can* be available as well as what is available right now.
I think there's significant benefit in this.

Note that we can also indicate "some commands need MODE READER" by a line
in the capabilities, or even by defining a second argument to the 101 code.

The main benefit of this proposal is the version codes. Here's how I see
them working:

* The format of a version code is:

    version-code = standard-version / private-version
    standard-version = major-version "." minor-version [version-detail]
    private-version = version-detail
    major-version = 1*DIGIT
    minor-version = 1*DIGIT
    version-detail = "-" arbitrary-string

* For a standard extension, the version is of the form "1.3". Each
published specification states the version number for that specification.
Changes in the minor version number are upwards compatible; that is, a
server that advertises 1.3 is *by definition* conformant with the 1.0, 1.1,
and 1.2 specifications as well [1.10 comes after 1.9.] Changes in the
major version are generally assumed to be incompatible.

* To allow for further development work on a standard extension, a
version-detail can be added to show conformance to drafts or other
experiments. The minor version remains that of the last real specification
that the server conforms to. The contents of the version-detail are under
the control of the new specification author.

So if work is going on to replace version 1.2, the eventual spec will be
1.3 and, in the meantime, servers can advertise "1.2-d05" to indicate they
conform to draft 05 of the new specification. Clients would normally ignore
what's after the dash.

* Version 0.0 is used for an extension that's being developed but hasn't
yet been standardised. Again, the version-detail is available.

* For a private extension, the version begins with a dash; anything beyond
that is entirely up to the implementer.

* Version 0.1 could be used to indicate conformance to RFCs that predate
our specification. But a better approach might be to treat them as private
with a code of "-RFC2980" (or whatever number is involved). Or how do
people feel about 0.2980?

The ability to flag a private extension also means that we can say that the
server *MUST* list all commands it supports instead of the present
weasel-wording.

Okay, repeating the above example so I can explain it further:

    [C] STATUS
    [S] 101 1.0     List of available extensions follows
    [S] OVER 1.0 + MSGID
    [S] HDR 1.0-RAA +
    [S] LISTGROUP 1.0 -MODE-READER
    [S] STARTTLS 0.0-02 -502
    [S] AUTHINFO 0.0-05-x -502 USER SASL:PLAIN,EXTERNAL
    [S] STREAMING 0.0-perhaps +
    [S] SECRET -CDWF -483
    [S] XDATA -CDWF +
    [.]

the meaning is:
* This server conforms to specification 1.0.
* The server supports 8 extensions.
* OVER and LISTGROUP conform to specification 1.0. So does HDR, but there's
  some development work going on with it (tagged RAA). STARTTLS, AUTHINFO,
  and STREAMING conform to draft specifications. SECRET and XDATA are
  private.
* Right now, OVER MSGID, HDR, STREAMING, and XDATA are all available while
  the other four aren't.
* LISTGROUP will work once the client has done MODE READER.
* STARTTLS and AUTHINFO aren't available for the rest of this session.
* SECRET is available if appropriate privacy is negotiated.

Note: 601 would be a better response code; we've said it should be used
for future multi-line responses. But I don't want this to divert the main
discussion.

Further thought: I'd like to find a way to provide further granularity in
the status information, showing that some bits are available and some
aren't. For example, a parameter ":" could mean "here's another status
code":

    [S] AUTHINFO 0.6-d02 + SASL:CRAM-MD5 : -483 USER SASL:PLAIN,EXTERNAL

meaning "I'll do CRAM-MD5 now, but if you want USER, PLAIN, or EXTERNAL
you'll need to sort out privacy". Or this may be getting too complex;
perhaps it should be left to individual extensions to sort out.

But I'd certainly like to encourage extensions to indicate both what is
available and what is currently-unavailable, rather than just the former.
And this is something that I'd like to see even if we don't adopt this
STATUS proposal.

> discussion to find the best approach from a transition standpoint.

Or we can bite the bullet and say that servers must advertise every command
that they implement.

However, I'm not 100% convinced. If a server does AUTHINFO SASL to spec,
and AUTHINFO GENERIC, what should it advertise?

Agreed.

> although I'd appreciate it if Scott would check me on this.

I think that my above proposal is worth doing, even if we have to repeat
Last Call.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Fri Oct 15 02:07:11 2004

> upgrade any legacy extensions that it already supports.

As I just said, I don't see how you enforce this.

If we have a versioning mechanism, we automatically encourage people to
check the IANA registry and resulting specifications so that they know what
value to put under "version supported". It also eliminates the temptation
to lie ("I almost do AUTHINFO USER correctly, so I'll advertise it").

One of the bugbears of C standardisation was all the compliers who define
__STDC__ to 1 on the grounds that "I do prototypes, so I look more like
Standard C than K&R C". Even worse are those who define it to 0 or 2 for
some strange reason. The lesson to learn is that you need to provide a way
to say "I've got something similar but not quite right" as a transitional
feature.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Fri Oct 15 02:15:40 2004

> wasn't done in the first place?

Nobody seems to know why we made it upper-only rather than case
insensitive, and it's a strange anomaly. I'd be very happy for us to fix
it.

However, we do have an implicit assumption elsewhere (see the 401 response
code) that dashes don't occur in extension labels. This can be worked
around, though.

At present a command name is not restricted in any way (other than no
spaces). I wonder if it would be a good idea to limit both extension labels
and command names to some smaller character set (say alphanumeric plus dot
and dash)? Do we want to forbid commands like "401" or "!@%)$&:#". Or, even
worse, names with quotes? Whatever we decide, I think that we should use
the same rule for extension labels and command names.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Oct 15 06:14:41 2004

> worse for the stuff that has been around for a while, but it's not unique

I don't see that.  Either early implementations advertise the new 
extension label as soon as they support the new command(s) (risking 
incompatibility if the functionality is a moving target), or (ideally) 
new extensions are implemented as X- commands until finalized.

> one at that.

I disagree that its a kludge, it simply states that the server is 
compliant with RFC 977bis.  Any server which does not advertise it, is 
at best compliant with RFC 977 and LIST EXTENSIONS may or may not be 
accurate (a.k.a. the currently deployed mess).  The beauty is in the 
simplicity.  As stated earlier, this approach has worked well for IMAP.

> to have to make changes anyway. This can then also fix other problems.

I *could* be convinced that a new command is worthwhile, but I'd have to 
see a compelling argument.

> a good idea in the long run.

I don't think this is necessary.  If the functionality of an existing 
command changes, then a new extension label should be created.

> to guess that doing MODE READER might make STARTTLS magically appear.

I also don't think this is necessary at all.  IMAP, POP and SMTP clients 
don't seem to have any problem.

>     [S] STREAMING 0.0-perhaps +

Yikes!  This seems like complete overkill, not to mention incredibly 
complex.

> And this is something that I'd like to see even if we don't adopt this

I don't see why this is necessary.  We really need client authors to 
weigh in here.  We, as server authors shouldn't be deciding what's good 
for clients in a vacuum.

> Or we can bite the bullet and say that servers must advertise every command

*Every* command?  Why?  The NNTPv2 capability states all of the base 
commands that the server supports (other than the optional LIST xxx 
stuff -- which is still a klugde IMHO).  Any other commands MUST be 
advertised by an extension label.

> and AUTHINFO GENERIC, what should it advertise?

AUTHINFO GENERIC is not standardized relative to RFC 977bis guidelines 
and doesn't have any extension label.  Since AUTHINFO SASL *is* 
standarized and provides equal, if not superior, functionality, that's 
what a client should use.  A client is still free to probe for AUTHINFO 
GENERIC if it so chooses.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Oct 15 06:18:30 2004

> specification to the next one; both of these are addressed in my STATUS

I agree that we can't force people to upgrade their software, but if 
they choose to advertise NNTPv2 without also fixing any legacy 
extensions to also be compliant, market pressure *will* force them to do 
so based on user complaints that their software is broken.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Oct 15 06:19:08 2004

> As I just said, I don't see how you enforce this.

Answered in another post.

---

From: ken at oceana.com (Ken Murchison)
Date: Fri Oct 15 06:20:42 2004

> the same rule for extension labels and command names.

Yes, I could see allowing alphanumeric plus a few other special chars, 
and probably force the first char to be alpha.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Mon Oct 18 02:48:04 2004

Pulling this one bit out.

Mark Crispin said:
> As far as MODE READER goes, the very existance of that command is a 
> protocol design wart to work around an implementation design bug.  NNTPv2 
> should make it MANDATORY that MODE READER is a no-op in any NNTPv2 
> compliant server; in particular it is NOT required of a client, and does 
> NOT change any server protocol modes.  The sole purpose of MODE READER 
> should be to try to communicate with pre-NNTPv2 servers.
> 
> You can't claim to draw a line in the sand between old cruft and new 
> goodness if you leave behind these warts.

As far as MODE READER is concerned, I personally agree with you. However,
server implementers will tell you otherwise. If you want to argue for this
change, please do so and let's see what happens.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Mon Oct 18 03:13:20 2004

Splitting this out as a sub-thread:

> and probably force the first char to be alpha.

Okay, here's the proposal. Any comments or objections?

In 3.1:

- Keywords MUST consist of printable US-ASCII characters.
+ Keywords MUST contain only US-ASCII letters, digits, dot ("." or %x2E),
+ and dash ("-" or %x2D) and MUST begin with a letter.

[The bit about case-insensitive is further down.]

In 8:

- An extension is identified by a unique extension-label, which is a string
- of 1 to 12 uppercase US-ASCII letters.
+ An extension is identified by a unique extension-label, which is a
+ keyword; it MUST be between 3 and 12 characters, MUST contain only
+ US-ASCII letters, digits, dot ("." or %x2E), and dash ("-" or %x2D),
+ and MUST begin with a letter. Extension-labels are case insensitive.

In 9.1:

- x-command-name = 3*12A-CHAR
+ x-command-name = keyword

In 9.4:

- extension-label = 1*12UPPER
+ extension-label = keyword

In 9.5:

+ keyword = ALPHA 2*11(ALPHA / DIGIT / "." / "-")

---

From: puntomaupunto at tin.it (maurizio codogno)
Date: Mon Oct 18 03:19:34 2004

>>>I wonder if it would be a good idea to limit both extension labels
>>> and command names to some smaller character set (say alphanumeric 
>>>plus dot and dash)?

>Okay, here's the proposal. Any comments or objections?

fine for me. 
Underscore could have been fine, but I see no mandatory use for it,
so it's better to stick to alphanum plus dot and dash.

ciao, .mau.

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Oct 18 07:10:06 2004

> + keyword = ALPHA 2*11(ALPHA / DIGIT / "." / "-")

Looks reasonable to me, provided there aren't any other special chars 
that might be desirable.  What is the rationale for restricting a 
keyword to something less that all printable chars?

---

From: chl at clerew.man.ac.uk (Charles Lindsey)
Date: Mon Oct 18 13:13:28 2004

>simplicity.  As stated earlier, this approach has worked well for IMAP.

FWIW, I am with Clive on this one.

Not that I would necessarily go with his full proposal (perhaps a cut down
version). It might have been a good thing to consider earlier on, but not
at this late stage.

---

From: ken at oceana.com (Ken Murchison)
Date: Mon Oct 18 15:26:18 2004

> version). It might have been a good thing to consider earlier on, but not

Just out of curiosity, are you and Clive saying that a simple NNTPv2
capability does not solve the problem at hand, or that it doesn't take
things far enough?

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Oct 19 00:16:52 2004

>> version). It might have been a good thing to consider earlier on, but not

The whole matter never struck me until Russ's post the other day.

> capability does not solve the problem at hand, or that it doesn't take

The latter, I think.

I should have realized that what I was proposing was too big a chunk for
people to swallow in one go. Let me try again, explaining what I am
thinking.

(1) Do we need a way to determine that a server conforms to the new core
specification?

I think we all agree that the answer to this one is "yes". Let's not worry
about how for the moment.

(2) Do we need a way to determine that a server conforms to the new
specification for extension X, where we are changing that specification?

I think we do.

(3) Is it sufficient to say that servers conforming to the new core
specification must support the new specification for extension X?

No, it isn't.

There is a specific set of extensions - AUTHINFO, STARTTLS, STREAMING,
LISTGROUPS, HDR, OVER - where the new specification is being published
at the same time as the new core specification. It is therefore very
tempting to say that these must be kept in sync.

However, this doesn't suffice. Firstly because any update to any other
extension is not going to be published at the right time. And secondly
because extension definitions may themselves change.

Are you with me so far?

(4) Why not just have a new extension-label ever time an extension changes?

Well, we could do that, but let's consider the sitation a little way down
the line with, say, two minor changes having been issued to the definition
of OVER. Do we really want the LIST EXTENSIONS output to have to say:

    OVER
    OVERv1.1
    OVERv1.2

just to cope with the clients that haven't yet been updated? I don't think
so.

So this implies that we need some kind of versioning arrangement for
extensions, just as for the core specification.

Yes?

====

I have to stop writing as I need to be elsewhere. Can people please think
on what I've written so far and comment, and I'll try to continue later
today.

---

From: chl at clerew.man.ac.uk (Charles Lindsey)
Date: Tue Oct 19 19:12:28 2004

>capability does not solve the problem at hand, or that it doesn't take

A bit of both. Either you omit the NNTPv2 proposal, and then a few
currently half-compliant clients and/or servers will behave oddly during
the tansitional period (and their administrators will work around as best
they can).

Or else we provide something along the lines Clive suggested (but it does
seem a bit complex to me).

But anything in between these is a kludge.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Tue Oct 26 09:56:34 2004

Mark Crispin said:
> From the point of view of a messaging client implementor who implements 
> NNTP as one of many messaging protocols, what is needed is:
>  1) A binary token that indicates if the server complies with a particular
>     specification.

A particular specification, or a particular *version* of a specification?

>  2) For each extension to that specification, a binary token that
>     indicates if the server offers that specification.

What about "the server will offer that specification, but only if you do
this bit of magic first"? Because that's what NNTP does right now. More
precisely, it says "does not offer that specification" until you do the
magic, at which point it suddenly say "does offer that specification".

> I certainly do not want or need anything like that STATUS command.

*All* that STATUS command adds is version numbers and the actual "the
server offers this specification" token (though not binary). The rest of
what is there was there already.

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Oct 27 00:59:18 2004

> keyword to something less that all printable chars?

It makes things more readable. Do we really want unbalanced parentheses and
quotation marks in command names?

It gives us room for expansion and for special cases, by using the
characters other than those we permit for commands.

For example, if we found a need for some kind of scoping of commands
(e.g. simultaneous use of more than one authentication ID) we can do this
with a non-permitted character:
    [user=clive] GROUP local.public
...
    [user=root] GROUP local.secret
(I blithely skip over the issues of managing multiple IDs).

For example, assume we decide that my STATUS proposal isn't worth doing
but, a year down the line, discover that there's a real need to say "this
protocol is supported but not available right now". We can allow the LIST
EXTENSIONS response line to begin with (say) "!" to indicate this.

For example (and this is really blue-sky stuff; please don't criticise it)
we could allow a prefix of "&" to mean "do this asynchronously and I'll ask
for the result later". This would be a more generalised form of streaming.
Or we could allow commands to be inserted in a posting body by prefixing
them with a dot, but this relies on command names themselves not beginning
with a dot (so that they look like dot-escaped lines).

I don't know what we will want to do with it, but I'd rather not rule the
possibility out at this stage.

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Oct 27 04:37:42 2004

> quotation marks in command names?

Absolutely not.  I guess the only additional char (beyond what you've 
outlined) which sticks out as potentially usable for keywords is '_'

---

From: clive at demon.net (Clive D.W. Feather)
Date: Wed Oct 27 04:41:21 2004

Ken Murchison said:
> Absolutely not.

Phew :-)

> I guess the only additional char (beyond what you've 

I think we only want one of '-' and '_'. I don't greatly care which, though
I think the former is slightly preferable.

---

From: ken at oceana.com (Ken Murchison)
Date: Wed Oct 27 06:24:10 2004

> I think the former is slightly preferable.

I don't care enough to make a big issue out of it.  I'll let the rest of 
the WG (you and Russ) decide which one(s) will be used.

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 3 15:39:02 2004

> Right, stop here. I don't like this idea. It's a kludge, and an unstructured

It has worked fine in IMAP for 10 years.  Nor has anything with the gothic 
complexity of your proposed STATUS command been necessary.

It is long past time that NNTP interoperate with other messaging 
protocols, and leverage the lessons of these other protocols.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Wed Nov 3 15:39:04 2004

> change, please do so and let's see what happens.

It certainly can be argued that this is a server internal detail which 
should not burden the protocol.  It also adds an additional RTT (6 by my 
count).  Consider the normal startup situation:

1) Connect, get a 200 or 201 response.

2) Send LIST EXTENSIONS.  If STARTTLS listed as an extension:
    a) send STARTTLS
    b) send LIST EXTENSIONS

3) If client is configured always-authenticate, or if client wants to post
    and greeting was 201, negotiate authentication.

4) Send MODE READER.  If get 480, negotiate authentication, then repeat
    MODE READER.

5) Send LIST EXTENSIONS (since MODE READER may have changed it)

And yes, we really want to do MODE READER *after* STARTTLS, since we want 
any authentication to be after TLS.

That's a lot of RTTs:
 	S: 200 NNTP server
 	C: LIST EXTENSIONS
 	S: 202 Must negotiate TLS now:
 	S: STARTTLS
 	S: .
 	C: STARTTLS
 	S: 382 TLS starting
 	C: LIST EXTENSIONS
 	S: 202 Must authentication now using:
 	S: AUTHINFO SASL:PLAIN
 	S: .
 	C: AUTHINFO SASL PLAIN AHRlc3QAMTIzNA==
 	S: 281 Logged in
 	C: MODE READER
 	S: 200 NNTP server
 	C: LIST EXTENSIONS
 	S: 202 Here's what you can do now:
 	S: LISTGROUP
 	S: OVER
 	S: .
 	C: GROUP comp.mail.misc

Better would be something like the following, which uses an IMAP-like 
mechanism to carry capabilities in responses, and flushes MODE READER:

 	S: 200 [STARTTLS] NNTP server, negotiate TLS now
 	C: STARTTLS
 	S: 382 TLS starting
 	C: LIST EXTENSIONS
 	S: 202 Must authentication now using:
 	S: AUTHINFO SASL:PLAIN
 	S: .
 	C: AUTHINFO SASL PLAIN AHRlc3QAMTIzNA==
 	S: 281 [LISTGROUP OVER] Now you can do stuff
 	C: GROUP comp.mail.misc

-- Mark --