Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8

From: Lisa Dusseault <[email-masked]>
Date: Mon, 25 Sep 2006 11:31:13 -0700

I agree it would be good to update RFC2617.  Is anybody going to take  
a stab at it?

Lisa

On Sep 22, 2006, at 7:55 PM, Martin Duerst wrote:

> Many thanks to Bjoern for the detailed checking and report.
>
> My summary of the situation would be as follows:
> There is currently widely varying practice. Current implementations
> are anyways broken and non-interoperable. The main reason for this
> is most probably that there is no clear specification. This means
> that an update of RFC 2617 is desirable. The new specification
> should probably go for UTF-8, while noting that there is still
> some varying practice.
>
> Regards,   Martin.
>
>
> At 01:41 06/09/23, Bjoern Hoehrmann wrote:
>> * Alexey Melnikov wrote:
>>> Does anybody know if updating RFC 2617 to say that username/ 
>>> passwords
>>> are UTF-8 would break any major implementation? For example, does
>>> anybody know if a major HTTP client/server implementation assume  
>>> ISO 8859-1?
>>
>> It appears that for Basic authentication the german version of  
>> Internet
>> Explorer 6 running on the german version of Windows 2003 as well  
>> as the
>> latest english Internet Explorer 7 release candidate running on the
>> german version of Windows XP will use something like ISO-8859-1  
>> for both
>> manual as well as XMLHttpRequest requests. Trying to use U+20AC as  
>> user
>> name and password they got encoded as 0x80 (Windows-1252) for  
>> manual re-
>> quests, and to '?' for XHR. Characters not included in  
>> Windows-1252 come
>> out as '?' regardless of the method used. For XHR my test cases  
>> include
>> documents encoded as ISO-8859-1 and UTF-8; there did not appear to be
>> any difference.
>>
>> The latest en-us version of Firefox uses UTF-8 for XHR and the lower
>> byte of the character when encoded using UTF-16BE (so for U+20AC you
>> get 0xAC) when using manual input. For manually entered http:// 
>> u:p@...
>> URLs Firefox uses Windows-1252 if possible, UTF-8 otherwise. When XHR
>> is used with such a URL, it uses UTF-8. The latest en-us version of
>> Opera9 always uses UTF-8, as far as I can tell based on my limited
>> testing. Results might well be different on with different default  
>> code
>> pages, language settings, and so on. Note that the illegal http:// 
>> u:p@..
>> addressing scheme allows to use arbitrary octet sequences using %hh
>> escape sequences, with some browser-specific limitations.
>> -- 
>> Bj�n H�rmann キ mailto:[email-masked] キ http:// 
>> bjoern.hoehrmann.de
>> Weinh. Str. 22 キ Telefon: +49(0)621/4309674 キ http:// 
>> www.bjoernsworld.de
>> 68309 Mannheim キ PGP Pub. KeyID: 0xA4357E78 キ http:// 
>> www.websitedev.de/
>> _______________________________________________
>> Ietf-http-auth mailing list
>> [email-masked]
>> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http- 
>> auth
>
>
> #-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
> #-#-#  http://www.sw.it.aoyama.ac.jp        
> mailto:[email-masked]
>
> _______________________________________________
> Ietf-http-auth mailing list
> [email-masked]
> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

---

From: Chris Drake <[email-masked]>
Date: Tue, 26 Sep 2006 07:47:12 +1000

Hi,

Do not the HTTP headers from the browser already say what character
encoding has been used, and do not the HTTP headers and HTML from the
server beforehand dictate this stuff too?  Except for the case where
someone is trying to type foreign characters on a PC which can't
understand them (eg: holidaymakers at net cafes where the net cafe has
not installed the language they want to use) - everything should
already be working just fine, yes?

Kind Regards,
Chris Drake

Tuesday, September 26, 2006, 4:31:13 AM, you wrote:

LD> I agree it would be good to update RFC2617.  Is anybody going to take
LD> a stab at it?

LD> Lisa

LD> On Sep 22, 2006, at 7:55 PM, Martin Duerst wrote:

>>> Bj?n H?rmann ? mailto:[email-masked] ? http:// 
>>> bjoern.hoehrmann.de
>>> Weinh. Str. 22 ? Telefon: +49(0)621/4309674 ? http:// 
>>> www.bjoernsworld.de
>>> 68309 Mannheim ? PGP Pub. KeyID: 0xA4357E78 ? http:// 
>>> www.websitedev.de/

LD> _______________________________________________
LD> Ietf-http-auth mailing list
LD> [email-masked]
LD> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

---

From: Julian Reschke <[email-masked]>
Date: Mon, 25 Sep 2006 23:56:08 +0200

> encoding has been used, and do not the HTTP headers and HTML from the

That's for the request body, not the headers.

> already be working just fine, yes?

I don't think so.

Speaking of which, where does HTML come into play here? We're talking 
about HTTP authentication à la RFC2617, not HTML forms based login.

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 26 Sep 2006 00:08:11 +0200

>about HTTP authentication à la RFC2617, not HTML forms based login.

For HTML form submissions, unless the author indicated something else,
web browsers tend to use the character encoding of the document that in-
cludes the form to encode the characters; they could apply the same
logic to the submission of credentials when there is such a document
(e.g., the user clicked a link to a HTTP Auth protected page, the page
with the link could then be used to determine some encoding). Based on
my limited testing, I found this to be not the case.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Date: Tue, 26 Sep 2006 01:21:03 +0200

> my limited testing, I found this to be not the case.

In the HTML case the information is directly available, as part of the  
form's own environment, which it would not be when the authentication  
credentials are being processed.

Which characterset/encoding should the application choose when that  
information is not available?

E.g. The information would not be available if the user goes directly to  
the site, by entering the URL directly. Similar situation could arise when  
the URL is loaded in another tab than the originating document, and  
obtaining the information would also not be entirely straight forward even  
if it is opened in the same tab. Or what about proxy authentication?

Also: What if the original page is using a different characterset/encoding  
than used by the server requesting authentication? E.g. What if a Russian  
language page directs you to an authenticated Japanese site? And AFAIK  
several languages actually have multiple encodings.

Also, characterset/encoding information may not be available in the HTTP  
header either.

It is not possible to define a heuristic that will fit all scenarios. The  
best approach is to define a common characterset/encoding that will be  
used by all compliant servers.

As RFC 2617 was not able to assist, the only guidance I had when I chose  
the I18N policy for Opera's RFC 2617 support, was RFC 2277/BCP 18, which I  
interprete to say that protocols should use UTF-8 unless they specify  
otherwise either in the specification (i.e. the RFC) or in a specific  
field of the protocol (in this case, that would mean an attribute in the  
WWW-Authenticate header). The problem is that work on the RFC 2617  
protocol probably started before RFC 2217 was finished.

Given that the current system is broken anyway, since client and server  
have to agree out-of-band on which characterset/encoding to use, it is in  
my opinion best to define a proper solution, which IMO means UTF-8,  
instead of trying to patch up the broken system . (And remember: Even a  
patch of the current system would have to be deployed in new clients and  
servers).

-- 
Sincerely,
Yngve N. Pettersen
 
********************************************************************
Senior Developer                     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
********************************************************************

---

From: Jim Luther <[email-masked]>
Date: Mon, 25 Sep 2006 17:34:35 -0700

While we're on this subject... In rfc2617 secction 3.2.1, it says:

> realm
>      A string to be displayed to users so they know which username and
>      password to use.

It would be also nice to define the encoding of the realm string so  
that clients that display the realm to users can display it correctly.  
We've seen realms from servers encoded UTF-8, ISO-8859-1, and with  
various Windows encodings. There's no good way to guess which encoding  
to use and so whatever is used is currently wrong on some servers.

All the best,

Jim

On Sep 25, 2006, at 11:31 AM, Lisa Dusseault wrote:

>
> I agree it would be good to update RFC2617.  Is anybody going to  
> take a stab at it?
>
> Lisa
>
>>> Bj�n H�rmann キ mailto:[email-masked] キ http://bjoern.hoeh 
>>> rmann.de
>>> Weinh. Str. 22 キ Telefon: +49(0)621/4309674 キ http://www.bjoernsworl 
>>> d.de
>>> 68309 Mannheim キ PGP Pub. KeyID: 0xA4357E78 キ http://www.websitedev 
>>> .de/
>> #-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]
>>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Sep 2006 02:42:39 +0200

Jim Luther schrieb:
> It would be also nice to define the encoding of the realm string so that 
> clients that display the realm to users can display it correctly. We've 
> seen realms from servers encoded UTF-8, ISO-8859-1, and with various 
> Windows encodings. There's no good way to guess which encoding to use 
> and so whatever is used is currently wrong on some servers.

Hm.

I was thinking "should be UTF-8, of course". But doesn't really RFC2045 
apply here at least in theory?

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 26 Sep 2006 02:48:17 +0200

>apply here at least in theory?

The realm-value is a quoted-string, and quoted-string is defined as

       quoted-string  = ( <"> *(qdtext | quoted-pair ) <"> )
       qdtext         = <any TEXT except <">>

and TEXT is

   The TEXT rule is only used for descriptive field contents and values
   that are not intended to be interpreted by the message parser. Words
   of *TEXT MAY contain characters from character sets other than ISO-
   8859-1 [22] only when encoded according to the rules of RFC 2047
   [14].

       TEXT           = <any OCTET except CTLs,
                        but including LWS>

So you could use realm="=?utf-8?b?..." or its variants. As you say, in
theory; I am unaware of any implementation that supports encoded words
in HTTP headers..
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Martin Duerst <[email-masked]>
Date: Tue, 26 Sep 2006 10:56:24 +0900

I agree with Julian and Bjoern here: IN THEORY, RFC 2047
(http://www.ietf.org/rfc/rfc2047.txt) (except for iso-8859-1)
would apply, but:
- it's not a very good theory: outdated because based on a view
  that the Web is basically iso-8859-1, and difficult to implement
- practice is different
- IETF policy is different

Regards,    Martin.

At 09:48 06/09/26, Bjoern Hoehrmann wrote:
>Bj$B‹S(Bn H$B‹I(Brmann $B%-(B mailto:[email-masked] $B%-(B http://bjoern.hoehrmann.de
>Weinh. Str. 22 $B%-(B Telefon: +49(0)621/4309674 $B%-(B http://www.bjoernsworld.de
>68309 Mannheim $B%-(B PGP Pub. KeyID: 0xA4357E78 $B%-(B http://www.websitedev.de/

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 26 Sep 2006 09:39:07 +0200

> - IETF policy is different

For interoperability, the encoding of usernames and passwords seems  
to be most interesting. If the encoding of the realm is misunderstood  
by the client (in a deterministic way) then interop could still be  
achieved, right?

//Stefan

---

From: Stefan Eissing <[email-masked]>
Date: Tue, 26 Sep 2006 09:42:08 +0200

Am 26.09.2006 um 09:39 schrieb Stefan Eissing:

> to be most interesting. If the encoding of the realm is  
> misunderstood by the client (in a deterministic way) then interop  
> could still be achieved, right?

To answer myself: not in digest authentication, stupid.

//Stefan

---

From: Martin Duerst <[email-masked]>
Date: Tue, 26 Sep 2006 16:42:39 +0900

I can help with the pieces related to i18n/utf-8, but I'm
not at all an expert in HTTP auth, so I won't try, sorry.

Regards,     Martin.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 27 Sep 2006 20:52:30 +0100

>not at all an expert in HTTP auth, so I won't try, sorry.
I can do 2617 update, but I am not an HTTP expert either.
So I would need a co-editor, who is.