Subject: Re: #320: add advice on defining auth scheme parameters

From: Julian Reschke <[email-masked]>
Date: Tue, 01 Nov 2011 20:59:33 +0100

On 2011-11-01 00:25, Yutaka OIWA wrote:
> Dear Julian,
> thank you for detailed and prompt response.
> Please allow me to answer in a random order.
>
> 2011/11/1 Julian Reschke<[email-masked]>:
>
>>> No, in XML cases, single quotes and double quotes has been clearly
>>> defined to be two representations of the same thing.
>>> There is a clear semantic definitions, a definition how to compare these
>>> twos,
>>> and a definition that anyone should not distinguish between these.
>>
>> And I believe that we need to make this statement for parameters in HTTP
>> header fields as well.
>
> Hmm, this makes sense.  Current "equivalence" rule seems too implicit for me.
>
>> In P2, we say:
>>
>> "Many header fields use a format including (case-insensitively) named
>> parameters (for instance, Content-Type, defined in Section 6.8 of [Part3]).
>> Allowing both unquoted (token) and quoted (quoted-string) syntax for the
>> parameter value enables recipients to use existing parser components. When
>> allowing both forms, the meaning of a parameter value ought to be
>> independent of the syntax used for it (for an example, see the notes on
>> parameter handling for media types in Section 2.3 of [Part3])."
>
> How about rephrasing this to something about:
>    Many header fields use a format including (case-insensitively) named
>    parameters (for instance, Content-Type, defined in Section 6.8 of [Part3]).
>    It should be aware that many existing parser components
>    do not distinguish unquoted (token) and quoted (quoted-string) syntax
>    for parameter values. Therefore,
>    whenever defining a new parameter, the meaning of a parameter value
>    SHOULD NOT be dependent of the syntax used for it
>    (for an example, see the notes on parameter handling for media types
>    in Section 2.3 of [Part3]).
>    <<  Receivers are RECOMMENDED to tolerate both forms of parameters
>    interchangeably.>>  (this<<  >>  may or may not be included)

Could you elaborate why you think it needs rephrasing?

> Technically speaking,
> To enable "recipients to use existing parser components",
> the most important thing is the third sentence of the above paragraph.
> If we used 1 and "1" in different and defined meanings,
> it will break such a parser.
> I agree this direction is correct, under the "Postel's principle".

Indeed.

> # I assume that a token in realm is currently an "undefined behavior",
> # allowing receivers to treat it as if it were a string.

Undefined means undefined, so recipients essentially can do whatever 
they want.

> Oppositely, Sending side does not need to send out both forms randomly,
> and specifications do not need to strongly certify both.
> Allowing sender-side to sending string unquoted is a bad idea,
> as it always gets complex and is bug-prone when special characters are found
> to be sent.  If any implementations have a feature correctly quoting any
> non-token strings, they can just send a quoted string anytime.  So,
> My opinion is still to require "normative forms" for the sender's side,
> also as par the Postel's principle.
> # So, if we had the above<<  >>  sentence, the "realm "=" quoted-string"
> # rule is not a bad thing, I think (now read as the sender's principle).

If I understand you correctly, you are arguing that we should leave the 
ABNF alone and just put additional requirements on the recipient. That's 
one way to do it, but *if* we require recipients to accept tokens as 
well it seems to be pointless not to say that in the ABNF.

>>> Hmm, I don't agree with this idea, actually.
>>> Tokens (meanings usually defined for each token except general integers
>>> etc.)
>>> has a distinct semantics than strings in general.
>>
>> I disagree. It's just a syntactical difference.
>
> Personally still disagreeing, but it is just the matter of definition.
> If we explicitly make it clear for HTTP/1.1 BIS and future,
> I will follow it for future (including current drafts).
>
> But one thing to be considered: case insensitivity.
> Tokens in RHS are often (not always?) case insensitive, and
> strings are mostly (always?) case sensitive.
> If we say, for example, "strings to be accepted whenever tokens
> are requested", it is OK for me.  But I am a bit still hesitate
> about saying "tokens are just special cases of unquoted strings."

No, tokens aren't always case-insensitive. One example are method names.

> e.g. in RFC2617 Digest, the parameter "stale" is explicitly case insensitive.
> "qop" and others are implicitly insensitive (by a text in RFC2616, Sec 2.1),
> but the naming of LHEX (lower hex?) rule confuses me for nc-value.
>
>> Do you have an example where this interpretation breaks current
>> implementations?
>
> Personally YES, but because mine is not in the wild, I can change it now :-)
> I briefly checked sources of a few Digest implementations,
> and all of them works with this change.
> (But we may need more interop check for Digest implementations...)
>

Best regards, Julian

---

From: Yutaka OIWA <[email-masked]>
Date: Fri, 04 Nov 2011 08:23:48 +0900

Dear Julian,

I think you're asserting that
> Undefined means undefined, so recipients essentially can do whatever
and that recipients are NOT REQUIRED to reject such undefined inputs.
I am right on that, but is it really true as blessed by the specification?

Assuming that, I generally agreeing with the general direction that
new schemes should not break existing generic parsers (but not only the one
you're mentioning).  Especially, it is really bad that <<someparam="1">> and
<<someparam=1>> have different meanings.

The summary of my points are:

1) Allowing, not mandating receivers to accept tokens as
   strings is enough for "generic parsers" you have mentioned.
   The original claim "This is necessary so that
      recipients can use a generic parser that applies to all
      authentication schemes"
   seems theoretically unsound (too strong) for me.
   At least I want to replace this claim with more modest one.

2) It is really meaningful for ABNF to specify
   what kinds of values are expected.
   One example is the stale parameter definition in RFC2617, saying

      stale  = "stale" "=" ( "true" | "false" )

   It is clearly saying that either true or false are accepted,
   and that they are case insensitive.

# Don't take that I am specifically saying about the stale parameter.
# It is just a example of the set-limited-tokens parameters.

3) Then, the rule << realm = quoted-string >> makes sense
   that it says that arbitrary list of ASCII characters are accepted.
   At least it is not as "bad design choice" as you said,
   as receivers are currently not required to reject any token realms.
   I also want to remove this declaration or replace with more modest one.

I don't want the stale rule to be any of the below:

   "stale" "=" word  # it should be either true or false, case insensitive

   "stale" "=" ( "true" | "false"
               | DQUOTE "true" DQUOTE | DQUOTE "false" DQUOTE
               | ...<too much more rules for deprecated excess backslashes> )

   "stale" "=" ( <either token or quoted-string representing
                  case-insensitive name either true or false> )

The current definition is the most clear and informative for implementors,
given that RFC2617 currently does not define generic framework for
the way of parsing you have specified as "ought to be".
What you are saying specifically for the realm parameter
seems to be the same class to the above thing, lacking symmetricity.

You're saying that <<"realm" "=" quoted-string>> is really bad and
should have been defined as <<"realm" "=" (quoted-string | token)>> in the past,
but it also means the stale should have been define as above.
Is it really a choice?

Also, is it too hard "for senders" to require realms to be quoted?
Do we really need senders to be allowed for randomly quoting any parameters?
You really want my parsers (generic in another way) to be rewritten for just
allowing senders for this capability?

# Alternatively, I would be happy if we really define the new generic semantics
# framework for parsing and interpreting the auth-param syntax at this moment, #
and define the realm parameter *using that* (with a caution that senders
# SHOULD send realm in quoted-string for backward compatibility, other Digest
# params SHOULD be cared too), without just saying the previous designers were
# bad.  This is really consistent, and give a clear straight-forward way for
# future designers (including me and OAuth-MAC people) that "syntactical
# constraints ought to be defined on the field value after parsing".

-- 
Yutaka OIWA, Ph.D.                                       Research Scientist
                            Research Center for Information Security (RCIS)
    National Institute of Advanced Industrial Science and Technology (AIST)
                      Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5]

---

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Nov 2011 09:43:44 +0100

> I am right on that, but is it really true as blessed by the specification?

Until recently, the spec didn't say anything about it at all. Fact is 
that existing implementations treat invalid header fields in an ad-hoc 
fashion, and there's no interop here, nor is the majority "draconian".

With -17, the spec says:

"Unless noted otherwise, Recipients MAY take steps to recover a usable 
protocol element from an invalid construct. However, HTTP does not 
define specific error handling mechanisms, except in cases where it has 
direct impact on security. This is because different uses of the 
protocol require different error handling strategies; for example, a Web 
browser may wish to transparently recover from a response where the 
Location header field doesn't parse according to the ABNF, whereby in a 
systems control protocol using HTTP, this type of error recovery could 
lead to dangerous consequences."

(as per <http://trac.tools.ietf.org/wg/httpbis/trac/ticket/186>).

> you're mentioning).  Especially, it is really bad that<<someparam="1">>  and

+1. It would be a terrible idea.

>     At least I want to replace this claim with more modest one.

What's the benefit of only allowing to do this?

>     and that they are case insensitive.

Yes, I'm not against using ABNF, but it should be restricted to 
constrain the parameter *value* (after potential q-s handling).

So you would just say:

stale-value = "true" / "false"

> 3) Then, the rule<<  realm = quoted-string>>  makes sense

All browsers except the token form. To me this indicates that making it 
invalid is pointless.

>     "stale" "=" (<either token or quoted-string representing

Yes, see above - just state the ABNF for the value after parameter parsing.

> You're saying that<<"realm" "=" quoted-string>>  is really bad and
> should have been defined as<<"realm" "=" (quoted-string | token)>>  in the past,

No, it's not. The problem here is that not all programmers read specs, 
and thus do what "works". If it works "everywhere", and it's *harmless*, 
then maybe it should have been valid in the first place.

> allowing senders for this capability?

I would want to recommend that for all *new* header fields that take 
param syntax, and for all *new* authentication schemes.

> # constraints ought to be defined on the field value after parsing".

That's my goal :-)

Best regards, Julian