Subject: Backwards definition of authentication header

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 4 Dec 2009 00:53:53 -0700

(I'm in the process of writing a new HTTP authentication scheme per the OAuth WG charter. I will be sending questions and issues regarding RFC 2617 and draft-ietf-httpbis-p7-auth in the hopes that someone will both be able to clarify them as well as find this feedback useful.)

Trying to follow the definition of the WWW-Authenticate and Authorization headers is like chasing one's own tail. The headers are defined in draft-ietf-httpbis-p7-auth but their content is defined using an imported definition from 2617, which in turn relies on the header definition in 2616. There isn't a single place where an implementer can read about the complete structure of these headers.

Why not move the entire section 1 from 2617 into draft-ietf-httpbis-p7-auth?

It is also confusing that Basic auth does not comply with the syntax defined for 'credentials' (no key="value" pair, only base64 username and password). I am not suggesting changing Basic, but I am suggesting changing the definition of 'credentials' to better reflect reality or at lease note why basic is different (I assume for historical reasons).

What is not clear to me is if the definition provided for 'credentials' is binding for new schemes, or if these schemes can go and redefine it as they see if.

EHL

---

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Dec 2009 16:59:01 +0100

> (I'm in the process of writing a new HTTP authentication scheme per the OAuth WG charter. I will be sending questions and issues regarding RFC 2617 and draft-ietf-httpbis-p7-auth in the hopes that…

Feedback from somebody trying to define a new HTTP auth scheme it 
*totally* useful. Thanks for that in advance.

> Trying to follow the definition of the WWW-Authenticate and Authorization headers is like chasing one's own tail. The headers are defined in draft-ietf-httpbis-p7-auth but their content is defined…

When you say "which in turn relies on the header definition in 2616", do 
you refer to more than "token" and "quoted-string"?

> Why not move the entire section 1 from 2617 into draft-ietf-httpbis-p7-auth?

I think I came to the conclusion a long time ago and thought that I 
opened a ticket for that; apparently I didn't. Yes, we need to make sure 
that Part 7 contains all but the actual authentication scheme definitions.

> It is also confusing that Basic auth does not comply with the syntax defined for 'credentials' (no key="value" pair, only base64 username and password). I am not suggesting changing Basic, but I am…

I assume the reasons are historical.

It appears the ABNF was broken when RFC2068/9 was revised as RFC2616/7, 
see <http://tools.ietf.org/html/rfc2068#section-11> which has:

           credentials    = basic-credentials
                          | auth-scheme #auth-param

We probably should record an erratum for RFC 2617 for now.

> What is not clear to me is if the definition provided for 'credentials' is binding for new schemes, or if these schemes can go and redefine it as they see if.

Is there anything *except* for the broken ABNF with respect to Basic 
that makes you think the definition isn't binding?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Dec 2009 17:09:59 +0100

Julian Reschke wrote:
> ...
> ...

I just checked the history of RFC 2617, and the change happened between 
draft 01 and draft 02 
(<http://tools.ietf.org/rfcdiff?url2=draft-ietf-http-authentication-02.txt>), 
when

-- draft 01 --
A user agent that wishes to authenticate itself with an origin server--
usually, but not necessarily, after receiving a 401 (Unauthorized)--MAY
do so by including an Authorization header field with the request. A
client that wishes to authenticate itself with a proxy--usually, but not
necessarily, after receiving a 407 (Proxy Authentication Required)--MAY
do so by including a Proxy-Authorization header field with the request.
Both the Authorization field value and the Proxy-Authorization field
value consist of credentials containing the authentication information
of the client for the realm of the resource being requested.

     credentials = basic-credentials | auth-scheme #auth-param
-- draft 01 --

was replaced by

-- draft 02 --
A user agent that wishes to authenticate itself with an origin server--

usually, but not necessarily, after receiving a 401 (Unauthorized)--MAY
do so by including an Authorization header field with the request. A
client that wishes to authenticate itself with a proxy--usually, but not
necessarily, after receiving a 407 (Proxy Authentication Required)--MAY
do so by including a Proxy-Authorization header field with the request.
Both the Authorization field value and the Proxy-Authorization field
value consist of credentials containing the authentication information
of the client for the realm of the resource being requested. The user
agent MUST choose to use one of the challenges with the strongest auth-
scheme it understands and request credentials from the user based upon
that challenge.

credentials = auth-scheme #auth-param

      Note that many browsers will only recognize Basic and will require
      that it be the first auth-scheme presented. Servers should only
      include Basic if it is minimally acceptable.
-- draft 02 --

So the intention may have been to replace the special case in the ABNF 
by prose, but, as far as I can tell, that was the wrong thing to do here.

Best regards, Julian

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 4 Dec 2009 09:54:09 -0700

> -----Original Message-----
> From: Julian Reschke [mailto:[email-masked]]
> Sent: Friday, December 04, 2009 7:59 AM
> To: Eran Hammer-Lahav
> Cc: HTTP Working Group ([email-masked])
> Subject: Re: Backwards definition of authentication header
> 
> Eran Hammer-Lahav wrote:
> > (I'm in the process of writing a new HTTP authentication scheme per
> > the OAuth WG charter. I will be sending questions and issues regarding
> > RFC 2617 and draft-ietf-httpbis-p7-auth in the hopes that someone will
> > both be able to clarify them as well as find this feedback useful.)
> 
> > Trying to follow the definition of the WWW-Authenticate and Authorization
> headers is like chasing one's own tail. The headers are defined in draft-ietf-
> httpbis-p7-auth but their content is defined using an imported definition
> from 2617, which in turn relies on the header definition in 2616. There isn't a
> single place where an implementer can read about the complete structure of
> these headers.
> 
> When you say "which in turn relies on the header definition in 2616", do you
> refer to more than "token" and "quoted-string"?

Yes, the actual header definition. Because draft-ietf-httpbis-p7-auth only updates 2617, but obsoletes 2616, it gets confusing as to where should someone reading 2617 go when trying to get the full authenticating picture. Moving the entire section 1 from 2617 to draft-ietf-httpbis-p7-auth will solve this.

> I think I came to the conclusion a long time ago and thought that I opened a
> ticket for that; apparently I didn't. Yes, we need to make sure that Part 7
> contains all but the actual authentication scheme definitions.

This will be a significant improvement.

> > It is also confusing that Basic auth does not comply with the syntax defined
> for 'credentials' (no key="value" pair, only base64 username and password). I
> am not suggesting changing Basic, but I am suggesting changing the definition
> of 'credentials' to better reflect reality or at lease note why basic is different
> (I assume for historical reasons).
> 
> > What is not clear to me is if the definition provided for 'credentials' is
> binding for new schemes, or if these schemes can go and redefine it as they
> see if.
> 
> Is there anything *except* for the broken ABNF with respect to Basic that
> makes you think the definition isn't binding?

No. But since Basic is 50% of 2617, it is a pretty big exception... :-)

EHL

---

From: Thomas Maslen <[email-masked]>
Date: Fri, 4 Dec 2009 09:04:13 -0800

[...]
> No. But since Basic is 50% of 2617, it is a pretty big exception... :-)

For what it's worth, the "Negotiate" and :"NTLM" auth schemes are like Basic inasmuch as they just have the scheme name followed by a Base64 blob.

(Perhaps schemes such as Digest that actually satisfy the ABNF are in the minority?)

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 4 Dec 2009 10:22:19 -0700

Is there a list somewhere of all existing HTTP auth schemes and their specifications?

EHL

> From: Thomas Maslen [mailto:[email-masked]]
> Sent: Friday, December 04, 2009 9:04 AM
> To: Eran Hammer-Lahav; Julian Reschke
> Subject: RE: Backwards definition of authentication header
> 
> [...]
> For what it's worth, the "Negotiate" and :"NTLM" auth schemes are like Basic
> inasmuch as they just have the scheme name followed by a Base64 blob.
>

---

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Dec 2009 18:26:00 +0100

> Is there a list somewhere of all existing HTTP auth schemes and their specifications?

Nothing official, as there's no registry: 
<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/141>.

Best regards, Julian

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Fri, 4 Dec 2009 11:24:48 -0700

This is a useful resource:

http://code.google.com/p/browsersec/wiki/Part3#HTTP_authentication

EHL

> From: Eran Hammer-Lahav
> Sent: Friday, December 04, 2009 9:22 AM
> To: 'Thomas Maslen'; Julian Reschke
> > >> Is there anything *except* for the broken ABNF with respect to
> > >> Basic that makes you think the definition isn't binding?
> > >
> > For what it's worth, the "Negotiate" and :"NTLM" auth schemes are like
> > Basic inasmuch as they just have the scheme name followed by a Base64
> blob.
> >
> > (Perhaps schemes such as Digest that actually satisfy the ABNF are in
> > the
> > minority?)

---

From: Julian Reschke <[email-masked]>
Date: Tue, 08 Dec 2009 15:46:20 +0100

OK,

so let's report an erratum against RFC 2617 to get this on the record:

-- snip --
Section 1.2, paragraph 4:
OLD:

        challenge   = auth-scheme 1*SP 1#auth-param

NEW:

     Note: for historic reasons, the "Basic" authentication scheme (see
     Section 2) uses a different format, thus the special case in the
     ABNF.

-- snip --

Best regards, Julian

Julian Reschke wrote:
>> It appears the ABNF was broken when RFC2068/9 was revised as 
>> RFC2616/7, see <http://tools.ietf.org/html/rfc2068#section-11> which has:
>>

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Wed, 9 Dec 2009 10:08:23 -0700

> Sent: Tuesday, December 08, 2009 6:46 AM
> To: Eran Hammer-Lahav
> Subject: Proposed RFC 2617 erratum, Re: Backwards definition of
> authentication header
> 
> OK,
> 

Don't you need the 1*SP in there?

EHL

> > I just checked the history of RFC 2617, and the change happened
> > between draft 01 and draft 02
> > A user agent that wishes to authenticate itself with an origin
> > server-- usually, but not necessarily, after receiving a 401
> > (Unauthorized)--MAY do so by including an Authorization header field
> > with the request. A client that wishes to authenticate itself with a
> > proxy--usually, but not necessarily, after receiving a 407 (Proxy
> > Authentication Required)--MAY do so by including a Proxy-Authorization
> header field with the request.
> > usually, but not necessarily, after receiving a 401
> > (Unauthorized)--MAY do so by including an Authorization header field
> > with the request. A client that wishes to authenticate itself with a
> > proxy--usually, but not necessarily, after receiving a 407 (Proxy
> > Authentication Required)--MAY do so by including a Proxy-Authorization
> header field with the request.
> > agent MUST choose to use one of the challenges with the strongest
> > auth- scheme it understands and request credentials from the user
> > based upon that challenge.
> >

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Dec 2009 18:15:17 +0100

> Don't you need the 1*SP in there?

Not really, the "#" construct already allows leading linear white space 
(otherwise RFC 2068 would have been incorrect as well :-).

BR, Julian

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Wed, 9 Dec 2009 10:43:37 -0700

> Sent: Wednesday, December 09, 2009 9:15 AM
> To: Eran Hammer-Lahav
> Subject: Re: Proposed RFC 2617 erratum, Re: Backwards definition of
> authentication header
> 
> Eran Hammer-Lahav wrote:
> >> OK,
> >>

Allows or requires it?

EHL

---

From: Julian Reschke <[email-masked]>
Date: Wed, 09 Dec 2009 18:49:19 +0100

>> (otherwise RFC 2068 would have been incorrect as well :-).

Allows. You win.

So, updated proposal:

-- snip --

NEW:

        credentials = basic-credentials | auth-scheme 1*SP #auth-param

-- snip --

Thanks, Eran.

Best regards, Julian

---

From: Eran Hammer-Lahav <[email-masked]>
Date: Wed, 9 Dec 2009 10:58:33 -0700

Looks good.

Thanks.

EHL

> Sent: Wednesday, December 09, 2009 9:49 AM
> To: Eran Hammer-Lahav
> >> Not really, the "#" construct already allows leading linear white
> >> space (otherwise RFC 2068 would have been incorrect as well :-).
> >
> > Allows or requires it?
> 
> Allows. You win.
> 
> So, updated proposal:
> 
> -- snip --

---

From: Julian Reschke <[email-masked]>
Date: Thu, 10 Dec 2009 14:17:03 +0100

> EHL

Reported as <http://www.rfc-editor.org/errata_search.php?eid=1959> (with 
1*SP replaced by SP, as pointed out to me off-list).

Best regards, Julian

---

From: "Manger, James H" <[email-masked]>
Date: Sat, 12 Dec 2009 12:04:57 +1100

> Reported as <http://www.rfc-editor.org/errata_search.php?eid=1959>
>
> credentials = basic-credentials | auth-scheme SP #auth-param

This looks wrong.
Basic includes the scheme.
The example in the spec is:

  Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==

Perhaps it should be:

  credentials = auth-scheme SP { basic-credentials | #auth-param }

[note: I am not proficient with ABNF]

NTLM and Negotiate also use a scheme followed by a base64-encoded blob, just like Basic.
The following example is from RFC 4559 "SPNEGO-based Kerberos and NTLM HTTP Auth in MS Windows" (which annoying looks like lower-case hex, though the text says it is base64):

  Authorization: Negotiate a87421000492aa874209af8bc028

The ABNF may as well support the Basic/NTLM/Negotiate form regardless of scheme, instead of a special case for just Basic (either as an RFC 2617 errata or an httpbis item?).

I am not sure how to write the ABNF. Here is a wild guess:

  credentials = auth-scheme SP { token | #auth-param }

-- 
James Manger

---

From: Paul Leach <[email-masked]>
Date: Sat, 12 Dec 2009 04:49:46 +0000

The ABNF may not be optimal, but it is correct.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Manger, James H
Sent: Friday, December 11, 2009 5:05 PM
To: Julian Reschke; Eran Hammer-Lahav
Cc: HTTP Working Group ([email-masked])
Subject: RE: Proposed RFC 2617 erratum, Re: Backwards definition of authentication header

This looks wrong.

Perhaps it should be:

-- 
James Manger

---

From: Paul Leach <[email-masked]>
Date: Sat, 12 Dec 2009 05:02:41 +0000

I do not understand the proposed erratum (eid=1959). Can someone please explain what the issue is? 

Prima-facie, the proposed fix looks wrong: how can the definition of "challenge" be replaced by one for "credentials"?

---

From: Julian Reschke <[email-masked]>
Date: Sat, 12 Dec 2009 12:06:06 +0100

>   Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==

You are right. I copied over a bug that is also present in RFC 2068.

Also, I tried to replace the rule for credentials, although the one for 
challenge is broken (thanks, Paul).

> [note: I am not proficient with ABNF]

So, let's restart. What's broken in RFC 2617 is:

because that ABNF does not allow basic credentials.

This one used to be in RFC 2068:

which special cases "Basic", but does so incorrectly (because 
basic-credentials doesn't contain the scheme name).

A fix for that (and *only* for that) would be:

   credentials = "Basic" basic-credentials
               | auth-scheme #auth-param

>   credentials = auth-scheme SP { token | #auth-param }

That's an orthogonal issue for which we should open an httpbis tracker 
issue. For now let's concentrate on fixing the outright bug in RFC 2617 :-).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 15 Dec 2009 13:31:56 +0100

>               | auth-scheme #auth-param

So, last call: should I report this erratum? (I don't think I can update 
the bad one I already sent...)

Best regards, Julian

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 21 Dec 2009 22:36:13 +0000

>Prima-facie, the proposed fix looks wrong: how can the definition of "challenge" be replaced by one for "credentials"?
You are right, it should be something like this instead:

OLD:

NEW:
       credentials = "Basic" basic-credentials | auth-scheme #auth-param

The issue with the original ABNF is that Basic wouldn't conform to the 
specified BNF, as auth-param is defined:

      auth-param     = token "=" ( token | quoted-string )

And Basic is defined:

      credentials = "Basic" basic-credentials
      basic-credentials = base64-user-pass
      base64-user-pass  = <base64 [4] encoding of user-pass,
                       except not limited to 76 char/line>

So basic-credentials doesn't match auth-param.