Subject: #320: add advice on defining auth scheme parameters

From: Julian Reschke <[email-masked]>
Date: Thu, 27 Oct 2011 22:03:13 +0200

(copied from new ticket, triggered from current discussion over in the 
oauth WG:)

When new schemes define new auth parameters, they of course need to 
stick to the base syntax.

In theory they *can* profile the allowable syntax, but doing so will not 
work well with consumers that use auth-scheme-agnostic parsers. It's 
thus best to define auth params based on what a parser would return 
*after* processing quoted-strings.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 28 Oct 2011 19:15:29 +0200

> *after* processing quoted-strings.

Proposed change: 
<http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/320/320.diff>

This adds in "Considerations for New Authentication Schemes":

    o  The parsing of challenges and credentials is defined by this
       specification, and cannot be modified by new authentication
       schemes.  When the auth-param syntax is used, all parameters ought
       to support both token and quoted-string syntax, and syntactical
       constraints ought to be defined on the field value after parsing
       (i.e., quoted-string processing).  This is necessary so that
       recipients can use a generic parser that applies to all
       authentication schemes.

       Note: the fact that "realm" only allows quoted-string syntax was a
       bad design choice not to be repeated in new schemes.

and also adds an example for WWW-Authenticate with multiple challenges:

    For instance:

      WWW-Authenticate: Newauth realm="apps", type=1,
                        title="Login to \"apps\"", Basic realm="simple"

    This header field contains two challenges; one for the "Newauth"
    scheme with a realm value of "apps", and two additional parameters
    "type" and "title", and another one for the "Basic" scheme with a
    realm value of "simple".

Feedback appreciated,

Julian

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 29 Oct 2011 22:23:06 +1100

I'm +1 on this. Normally we shouldn't have to go to this level of detail, but the syntax here is very brittle, so it's worth doing.

Any objections?

Regards,

On 29/10/2011, at 4:15 AM, Julian Reschke wrote:

> Proposed change: <http://trac.tools.ietf.org/wg/httpbis/trac/attachment/ticket/320/320.diff>
> 

Mark Nottingham
http://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 29 Oct 2011 14:23:18 +0200

Hi Mark, Julian,

> I'm +1 on this. Normally we shouldn't have to go to this level of detail, but the syntax here is very brittle, so it's worth doing.

[disclaimer: I've not read the initial issue]

In the following header from Julian's example :

I'm seeing 4 independant comma-delimited values, which could also be
expressed and individually processed like this with the same meaning :

     WWW-Authenticate: Newauth realm="apps"
     WWW-Authenticate: type=1
     WWW-Authenticate: title="Login to \"apps\""
     WWW-Authenticate: Basic realm="simple"

I'm not sure this is precisely what's desired since Julian suggested
there were two challenges in this example (Newauth and Basic). Thus I
would have written them that way with semi-colons to split the attributes :

     WWW-Authenticate: Newauth realm="apps"; type=1;
                       title="Login to \"apps\"", Basic realm="simple"

which would then be equivalent to :

     WWW-Authenticate: Newauth realm="apps"; type=1; title="Login to \"apps\""
     WWW-Authenticate: Basic realm="simple"

It would be nice to clarify this point since it can be confusing.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Sat, 29 Oct 2011 14:38:14 +0200

>       WWW-Authenticate: Basic realm="simple"

...except a recipient would allow to ignore the 2nd and the 3rd because, 
in isolation, they do not conform to the header ABNF.

> It would be nice to clarify this point since it can be confusing.

Willy,

indeed. WWW-Authenticate is a big mess, because it uses the same 
delimiter *between* challenges and *inside* challenges. We can't change 
that, but we can warn.

The spec already says (and has been in 2617):

"User agents are advised to take special care in parsing the 
WWW-Authenticate field value as it might contain more than one 
challenge, or if more than one WWW-Authenticate header field is 
provided, the contents of a challenge itself can contain a 
comma-separated list of authentication parameters."

Best regards, Julian

---

From: Yutaka OIWA <[email-masked]>
Date: Sat, 29 Oct 2011 13:54:21 +0000

mostly +1 for requiring generic-parser-compatible syntaxes.

Could you explain what

means?
I don't think it is a bad idea, as long as ["realm=" token] pattern is
not valid.  (because it is equivalent to use a generic parser first and
then to require quoted-string as a value.)

---

From: Julian Reschke <[email-masked]>
Date: Sat, 29 Oct 2011 16:04:51 +0200

On 2011-10-29 15:54, Yutaka OIWA wrote:
> ...

What I mean is that the ABNF requires the realm value to be a token, but 
in practice all UAs accept both forms.

See test case: <http://greenbytes.de/tech/tc/httpauth/#simplebasictok>

Best regards, Julian

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 29 Oct 2011 16:29:26 +0200

> in isolation, they do not conform to the header ABNF.

Good point.

(...)
> delimiter *between* challenges and *inside* challenges. We can't change 

Couldn't we suggest that new schemes must use semi-colon instead ?
After all, existing parsers will have to be extended to support new
schemes anyway, so making them support a more reliable syntax makes
sense.

> comma-separated list of authentication parameters."

I did not notice this warning, thanks for point it it to me.

Regards,
Willy

---

From: Julian Reschke <[email-masked]>
Date: Sun, 30 Oct 2011 09:18:16 +0100

>       bad design choice not to be repeated in new schemes.

Rephrasing to...:

       Note: the fact that the value syntax for the "realm" parameter is
       restricted to quoted-string was a bad design choice not to be
       repeated for new parameters.

> then to require quoted-string as a value.)

The issue is that - depending on the API - after you have run a generic 
parser you simply might not have that kind of information.

It's comparable to require an XML attribute to be quoted with single 
quotes, and to disallow double quotes. It can be made to work by 
customizing the parser, but why would you want to?

Best regards, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 31 Oct 2011 14:05:19 +1100

> schemes anyway, so making them support a more reliable syntax makes

Not necessarily; an implementation can (and should be able to) use a generic parser no matter what new schemes are introduced.

Also, if a non-upgraded parser gets a message with multiple challenges in it, it could get confused by this.

To be clear -- I'm no fan of the current syntax, but we need to be *extremely* careful to make sure we don't make the situation worse.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 31 Oct 2011 10:00:36 +0100

> repeated for new parameters.

-> Applied with <http://trac.tools.ietf.org/wg/httpbis/trac/changeset/1465>.

Best regards, Julian

---

From: Yutaka OIWA <[email-masked]>
Date: Tue, 1 Nov 2011 05:25:42 +0900

> parser you simply might not have that kind of information.

Hmm, I don't agree with this idea, actually.
Tokens (meanings usually defined for each token except general integers etc.)
has a distinct semantics than strings in general.
If the spec meant to allow special characters (e.g. commas)
in some parameters, that should be limited to quoted-strings
on the sender side.

For receivers side, I agree that some parsers takes drops the distinctions
between tokens and strings, and because of that I agree that new
specifications should not overload tokens (1, 2, 3 etc.) and
_equivalent_ strings ("1", "2", "3") for having a different meaning,
supporting "the RFC1123 way" of parsing.
Can't it be phrased in this way?

If we meant that quote-string is just a way to "quote tokens with commas etc",
we should have have clear definitions for equivalence between tokens
and strings, and ask every implementors to implement in that way.
I am not favor of this idea, because it also breaks existing implementations.

> It's comparable to require an XML attribute to be quoted with single quotes,
> and to disallow double quotes. It can be made to work by customizing the
> parser, but why would you want to?

No, in XML cases, single quotes and double quotes has been clearly
defined to be two representations of the same thing.
There is a clear semantic definitions, a definition how to compare these twos,
and a definition that anyone should not distinguish between these.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 31 Oct 2011 22:05:03 +0100

On 2011-10-31 21:25, Yutaka OIWA wrote:
> 2011/10/30 Julian Reschke<[email-masked]>:

I disagree. It's just a syntactical difference.

> Can't it be phrased in this way?

In P2, we say:

"Many header fields use a format including (case-insensitively) named 
parameters (for instance, Content-Type, defined in Section 6.8 of 
[Part3]). Allowing both unquoted (token) and quoted (quoted-string) 
syntax for the parameter value enables recipients to use existing parser 
components. When allowing both forms, the meaning of a parameter value 
ought to be independent of the syntax used for it (for an example, see 
the notes on parameter handling for media types in Section 2.3 of [Part3])."

> and strings, and ask every implementors to implement in that way.

See above.

> I am not favor of this idea, because it also breaks existing implementations.

For instance?

> and a definition that anyone should not distinguish between these.

And I believe that we need to make this statement for parameters in HTTP 
header fields as well.

Do you have an example where this interpretation breaks current 
implementations?

Best regards, Julian

---

From: Yutaka OIWA <[email-masked]>
Date: Tue, 01 Nov 2011 08:25:40 +0900

Dear Julian,
thank you for detailed and prompt response.
Please allow me to answer in a random order.

> And I believe that we need to make this statement for parameters in HTTP

Hmm, this makes sense.  Current "equivalence" rule seems too implicit for me.

> parameters (for instance, Content-Type, defined in Section 6.8 of [Part3]).
> Allowing both unquoted (token) and quoted (quoted-string) syntax for the
> parameter value enables recipients to use existing parser components. When
> allowing both forms, the meaning of a parameter value ought to be
> independent of the syntax used for it (for an example, see the notes on
> parameter handling for media types in Section 2.3 of [Part3])."

How about rephrasing this to something about:
  Many header fields use a format including (case-insensitively) named
  parameters (for instance, Content-Type, defined in Section 6.8 of [Part3]).
  It should be aware that many existing parser components
  do not distinguish unquoted (token) and quoted (quoted-string) syntax
  for parameter values. Therefore,
  whenever defining a new parameter, the meaning of a parameter value
  SHOULD NOT be dependent of the syntax used for it
  (for an example, see the notes on parameter handling for media types
  in Section 2.3 of [Part3]).
  << Receivers are RECOMMENDED to tolerate both forms of parameters
  interchangeably. >> (this << >> may or may not be included)

Technically speaking,
To enable "recipients to use existing parser components",
the most important thing is the third sentence of the above paragraph.
If we used 1 and "1" in different and defined meanings,
it will break such a parser.
I agree this direction is correct, under the "Postel's principle".

# I assume that a token in realm is currently an "undefined behavior",
# allowing receivers to treat it as if it were a string.

Oppositely, Sending side does not need to send out both forms randomly,
and specifications do not need to strongly certify both.
Allowing sender-side to sending string unquoted is a bad idea,
as it always gets complex and is bug-prone when special characters are found
to be sent.  If any implementations have a feature correctly quoting any
non-token strings, they can just send a quoted string anytime.  So,
My opinion is still to require "normative forms" for the sender's side,
also as par the Postel's principle.
# So, if we had the above << >> sentence, the "realm "=" quoted-string"
# rule is not a bad thing, I think (now read as the sender's principle).

> I disagree. It's just a syntactical difference.

Personally still disagreeing, but it is just the matter of definition.
If we explicitly make it clear for HTTP/1.1 BIS and future,
I will follow it for future (including current drafts).

But one thing to be considered: case insensitivity.
Tokens in RHS are often (not always?) case insensitive, and
strings are mostly (always?) case sensitive.
If we say, for example, "strings to be accepted whenever tokens
are requested", it is OK for me.  But I am a bit still hesitate
about saying "tokens are just special cases of unquoted strings."

e.g. in RFC2617 Digest, the parameter "stale" is explicitly case insensitive.
"qop" and others are implicitly insensitive (by a text in RFC2616, Sec 2.1),
but the naming of LHEX (lower hex?) rule confuses me for nc-value.

> Do you have an example where this interpretation breaks current

Personally YES, but because mine is not in the wild, I can change it now :-)
I briefly checked sources of a few Digest implementations,
and all of them works with this change.
(But we may need more interop check for Digest implementations...)

-- 
Yutaka OIWA, Ph.D.                                       Research Scientist
                            Research Center for Information Security (RCIS)
    National Institute of Advanced Industrial Science and Technology (AIST)
                      Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5]