Subject: RFC 2617: Which character should be used?

From: Yngve Nysaeter Pettersen <[email-masked]>
Date: Tue, 15 Apr 2003 23:26:45 +0200

Hi,

My name is Yngve N. Pettersen, I am a developer at Opera Software ASA, the 
company producing the Opera browser. One of my areas of responsibility is 
our HTTP protocol support.

Some time ago, while implementing Opera's support for international 
character sets I discovered that RFC 2617 did not specify the character set 
to be used when encoding the username and password arguments for Basic and 
Digest authentication.

Given that BCP 18/RFC 2277 strongly encouraged UTF-8 support in protocols, 
and that it may be impossible to determine the server's preferred 
characterset, among other reasons, I decided to use UTF-8 as the 
characterset when encoding the username and password before generating the 
authentication strings.

Recently we received a report concerning problems with this way of 
generating authentication strings (apparantly other clients does not 
convert national characters in Western European languages, at least, I 
don't know how they treat Asian languages), and while researching the 
current state of the protocol, I noticed that the current errata does not 
address this point.

I would therefore like to suggest that an item specifying which character 
set should be used when generating Basic and Digest authentication strings 
is added to the errata.

My suggestion is that UTF-8 is selected as the character set used to encode 
the username and password values when creating the "user-pass" string (sec. 
2) and the "username-value" and "passwd" strings in sec. 3.2.2. It might 
also be an idea to specify the same for other text attributes as well.

As mentioned above BCP 18 indicates UTF-8 is the preferred charset for 
protocols.

Additionally, I believe it would be very difficult to create a foolproof 
guessing method that would decide the charset based on such things as the 
charset of the authentication challenge response body, toplevel domain of 
the server, or the same from the referrer (if any), or the character set 
used on the client's computer (which may not match what is used on the 
server). As an example, the challenge may use a default message in English, 
while passwords and documents are encoded in a Japanese character set.

I think the best way of avoiding (any further) ambiguities is to specify a 
single character set that MUST be used, and UTF-8 is the character set 
recommended by BCP 18.

-- 
Sincerely,
Yngve N. Pettersen

********************************************************************
Senior Developer		             Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 51              Fax:    +47 24 16 40 01
********************************************************************

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 15 Apr 2003 18:11:45 -0600

Hi,

> single character set that MUST be used, and UTF-8 is the character set

Although I am not big expert on this, but here is some information for you.

You are right, RFC 2617 doesn't specify any character set. RFC 2617 is a
revision of RFC 2069 which predates RFC 2277. So I suspect that when the RFC
2069 was revised, nobody noticed this issue.

RFC 2831 (Using Digest Authentication as a SASL Mechanism) which is based on
RFC 2617 had to deal with this as well. RFC 2831 has a phrase "The directive is
needed for backwards compatibility with HTTP Digest, which only supports ISO
8859-1." which suggests that ISO 8859-1 is the default for HTTP. RFC 2831 had
to add a new "charset" directive and a complex rule to convert UTF-8
usernames/passwords [that can be fully expressed as ISO 8859-1] to ISO 8859-1.
This is a mess :-(.

So, although I tend to agree with your choice to use UTF-8, however it seems
that the reality is a bit more complicated than that.

Regards,
Alexey Melnikov
__________________________________________
R & D, ACI Worldwide/MessagingDirect
Watford, UK

Work Phone: +44 1923 81 2877
Home Page: http://orthanc.ab.ca/mel
IETF standard
related pages: http://orthanc.ab.ca/mel/devel/Links.html

I speak for myself only, not for my employer.
__________________________________________

---

From: Scott Lawrence <[email-masked]>
Date: 16 Apr 2003 08:20:15 -0400

Yngve Nysaeter Pettersen <[email-masked]> writes:

> the username and password values when creating the "user-pass" string
> (sec. 2) and the "username-value" and "passwd" strings in sec. 3.2.2. It
> might also be an idea to specify the same for other text attributes as well.

I just took a look at the spec to try to come up with specific
language for this.  

Section 3.2.2.2 A1 add:

   The passwd value used should be encoded using UTF-8.
 
I don't think it's an issue for the user-pass string or
username-value, since these are just literals that are passed in the
clear to the server anyway.  Can't the server just use them as is?

-- 
Scott Lawrence        
  Actively seeking work 
  http://skrb.org/scott/
  [ <[email-masked]> is deprecated ]

---

From: "Joris Dobbelsteen" <[email-masked]>
Date: Wed, 16 Apr 2003 20:04:01 +0200

>-----Original Message-----
>From: [email-masked]
>[mailto:[email-masked]]On
>Behalf Of Scott Lawrence
>Sent: Wednesday, 16 April 2003 14:20
>To: [email-masked]
>Cc: [email-masked]
>Subject: Re: RFC 2617: Which character should be used?
>
>
>

I believe this might be a problem as it might differ from existing
implementations.
Making passwords UTF-8 before MD5 yields a complete different result from using
ASCII and then MD5 for Digest. This is also true for Basic (using Base64).
I would expect implementations to currently use the ASCII character-set.

This does indeed not solve the issues regarding languages using another
character set. I don't have any details how current implementations do this.

HTTP (including HTTP/1.1) is much older than BCP 18 (RFC 2277), so I don't
believe its recommendation is used.

- Joris

---

From: Scott Lawrence <[email-masked]>
Date: 16 Apr 2003 14:30:11 -0400

> I would expect implementations to currently use the ASCII character-set.

To quote RFC 2279 (UTF-8):

   UTF-8, the object of this memo, uses all bits of an octet, but has
   the quality of preserving the full US-ASCII range: US-ASCII
   characters are encoded in one octet having the normal US- ASCII
   value, and any octet with such a value can only stand for an
   US-ASCII character, and nothing else.

So passwords containing only ASCII would not change value, right?  For
passwords that can't be expressed in ASCII, the current spec doesn't
define the right behaviour, and this thread started with a report that
it's already produced incompatible implementations.

> believe its recommendation is used.

The purpose of the errata is to help current implementors to agree on
resolutions of issues discovered since the spec was issued.  When/if
the spec is updated, it is expected that these resolutions will be
incorporated; at that point, it would be good to have solutions that
are in harmony with other standards issued during that time.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 16 Apr 2003 14:49:00 -0400

At 14:30 03/04/16 -0400, Scott Lawrence wrote:

>So passwords containing only ASCII would not change value, right?

Yes, correct. Also, anything else than ASCII characters, when
encoded in UTF-8, have all most significant bits set in all
octets, so that it is impossible to produce something that
looks like ASCII but isn't.

Regards,    Martin.

>For

---

From: "Larry Masinter" <[email-masked]>
Date: Wed, 16 Apr 2003 21:56:13 -0700

The topic was discussed in the HTTP working group in September 1998.
http://lists.w3.org/Archives/Public/ietf-http-wg-old/1998SepDec/0040.htm
l
and the end of the dialog was:

   How about we restrict user-ids and typed-in passwords to US-ASCII for
now,
   declare encoding of non-ASCII characters in those fields undefined
but
   explicitly forbid use of localized charsets (e.g., ISO-8859-1)?  Then
we
   can amend it to use UTF-8 later with a spec that progresses
separately
   on the standards track.

However, this apparently didn't make it into the spec. I think that
as far as 'errata', what's missing is a pointer to the discussion and
this particular 'resolution'.

I admit that it's not a great resolution of the issue (we basically just
avoided it), but that's at least a baseline: HTTP isn't as functional
as one would like in this area.

I'm less sanguine about mandating new behavior at this point, for
the same reasons given in 1998. So I'm not sure there is a "fix",
unfortunately.  Clients can do things that servers won't understand,
and servers can expect clients to do things that they can't.
Non-ASCII user names will be unreliable until new behavior
is proposed and accepted by the community.

The notion (in 1998) is that someone would write a spec that would
progress independently on standards track (or at least thru Proposed
and Draft) so that we can update HTTP to full standard some day.

Larry
-- 
http://larry.masinter.net

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
On Behalf Of Alexey Melnikov
Sent: Tuesday, April 15, 2003 5:12 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: RFC 2617: Which character should be used?

Hi,

> the username and password values when creating the "user-pass" string
(sec.
> single character set that MUST be used, and UTF-8 is the character set

Although I am not big expert on this, but here is some information for
you.

You are right, RFC 2617 doesn't specify any character set. RFC 2617 is a
revision of RFC 2069 which predates RFC 2277. So I suspect that when the
RFC
2069 was revised, nobody noticed this issue.

RFC 2831 (Using Digest Authentication as a SASL Mechanism) which is
based on
RFC 2617 had to deal with this as well. RFC 2831 has a phrase "The
directive is
needed for backwards compatibility with HTTP Digest, which only supports
ISO
8859-1." which suggests that ISO 8859-1 is the default for HTTP. RFC
2831 had
to add a new "charset" directive and a complex rule to convert UTF-8
usernames/passwords [that can be fully expressed as ISO 8859-1] to ISO
8859-1.
This is a mess :-(.

So, although I tend to agree with your choice to use UTF-8, however it
seems
that the reality is a bit more complicated than that.

Regards,
Alexey Melnikov

---

From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Date: Mon, 21 Apr 2003 05:05:51 +0200

Hi Scott,

[note: I corrected the subject]

> clear to the server anyway.  Can't the server just use them as is?

I'm afraid not.

Remember, the server must not just be able to perform calculations using 
the password, it must also be able to look up the appropriate username 
entry in its database. If the client and the server are using different 
character sets in any phase of creating, updating and referencing this 
database there will be no match.

However, I've just noticed that RFC 2616 actually do comment on this in 
section 2.2, and requires RFC 2047 encoding for any TEXT not using iso- 
8859-1 encoding.

The question then becomes: Should the errata of RFC 2617 override that 
requirement and mandate UTF-8, or should an extension to the current header 
methods be formulated, or should completely new authentication methods be 
formulated that will handle UTF-8 usernames/passwords?

One way of overriding that section would be to change the defintions of 
usernames and passwords to using OCTET (minus control characters and 
special characters) instead of TEXT. Something similar was proposed in the 
thread referenced by Larry Masinter.

Personally I'd prefer to override RFC 2616 sec 2.2 for RFC 2617 
credentials, as I think BCP 18 should be the guideline. If that is not 
possible I'd like to avoid RFC 2047 syntax (E.g: Which charset should be 
used for the password in digest authentication, and how do we tell the 
server?).

I can think of several alternatives if UTF-8 cannot be made mandatory:

Alternative 1: Specify that if all characters in both the username and 
password is in the iso-8859-1 charset, then iso-8859-1 can be used, in all 
other cases utf-8 is used. This will probably lead to some 
username/password collisions; I do not know how serious this will be.

Alternative 2: Extend RFC 2617 with two new methods (e.g.) "Basic8" and 
"Digest8", with mostly the same syntax as the present "Basic" and "Digest" 
methods, but with the requirement that username and password is encoded in 
UTF-8. However, this will require the server to send one extra header for 
each method it supports, and would probably need to be specified separately 
as an RFC.

Alternative 3: Extend the syntax of "Basic" and "Digest" authentication 
headers with a "utf-8" parameter which, when included in the server's 
challenge, indicate that the server understands UTF-8 encoded usernames and 
passwords. When a UTF-8 enable client sees this parameter it can then 
encode the username and password in UTF-8 and add a utf-8 parameter to the 
authorization header it sends to the server to indicate that the 
authorization is in UTF-8.

Examples:

  WWW-Authenticate: Basic realm="realm", utf-8
  WWW-Authenticate: Digest realm="realm", <digest parameters>, utf-8

  Authorization: Basic <basic-credentials>, utf-8
  Authorization: Digest <digest-response>, utf-8

By using a utf-8 parameter instead of a charset parameter, it's possible to 
limit the charset permutatitons the client and the server have to be able 
to handle.

However, given that the A1 value for Digest authentication may be 
calculated in advance and by a thirdparty server, that means that two A1 
values must be prepared, and distributed, when non-US-ASCII 
usernames/passwords are used (Come to think of it, this will also be the 
case if utf-8 is mandated, at least in a transition phase).

Personally, as mentioned, I prefer making utf-8 mandatory, but if 
alternative specifications are needed I think alternatives 1 and 3 are the 
most acceptable of the alternatives above.

-- 
Sincerely,
Yngve N. Pettersen