Subject: [Ietf-http-auth] Re: Updating RFC 2617 (HTTP Digest) to use UTF-8
From: Frank Ellermann <[email-masked]>
To: [email-masked]
Date: Thu, 28 Sep 2006 01:10:02 +0200
List-Id: ietf-http-auth.osafoundation.org

Martin Duerst wrote:
 
> That's the THEORY bit

Yes, any spec. is theory until somebody implements it and gets
it right.  I've no clue what existing servers and clients do,
something like a reference test server would be nice.

In theory (again :-) I could patch my dummy server and check
what some older clients actually do.  In practice I'm lazy and
predict that they behave as for random Web forms wrt Latin-1.
(probably claiming that windows-1252 is the same as Latin-1).

A more interesting question, does any client support RFC 2617
or 2069 at all (minus AUTH BASIC) ?  I vaguely recall that it
was an optional apache module, but maybe I confuse it.

 {2831]
>| The directive is needed for backwards compatibility with
>| HTTP Digest, which only supports ISO 8859-1.

One of the RFC 2831 authors is also an 2617 author, we have to
assume that this is no nonsense.  The RFC 2617 <username-value>
is a <quoted-string> without backslash-canonicalization, as in
RFC 2831.  Just copied from RFC 2616, which got it from 2068,
and 2068 got it from 1945.

Apparently 2068 introduced <quoted-pair> (the backslash stuff).

2068 also introduced 1522 non-ASCII characters in header fields
in addition to the Latin-1 default of 1945, and that's now 2047
wrt 2616.  Maybe this issue is limited to HTTP/1.1, but I fear
that doesn't help.

> RFC 2617 doesn't mention ISO 8859-1 once, so it's also not
> supported.

Its defined in RFC 2616:

| Words of *TEXT MAY contain characters from character sets
| other than ISO-8859-1 [22] only when encoded according to
| the rules of RFC 2047 [14].

RFC 2047 offers:  "An 'encoded-word' MUST NOT appear within a
'quoted-string'."  The text in quoted strings is supposed to
be US-ASCII, so far it's simple (modulo backslashes).

The general pattern is as in MIME, and RFC 2616 3.6 says:
       parameter               = attribute "=" value
       attribute               = token
       value                   = token | quoted-string

But RFC 2617 doesn't allow the <token> case for among others
<realm> and <username-value>.  For 2617bis we'd have to fix:

-        username-value   = quoted-string
+        username-value   = quoted-string | token

<password> and <userid> are *TEXT, and covered by the overall
2616 rule, that's actually for AUTH BASIC.  For AUTH DIGEST...

      passwd   = < user's password >

...is apparently the same as <password>, some *TEXT, default
Latin-1 as always for HTTP.

Frank

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Date: Wed, 27 Sep 2006 23:23:05 +0200
Organization: <URL:http://purl.net/xyzzy>
Lines: 15
Sender: news <[email-masked]>
Subject: [Ietf-http-auth] Re: Updating RFC 2617 (HTTP Digest) to use  UTF-8

Alexey Melnikov wrote:

> I can do 2617 update, but I am not an HTTP expert either.
> So I would need a co-editor, who is.

Maybe ask Phil.  One detail that's IMO not "obvious" is an
exact definition when a 2069-fallback is required for a
general purpose DIGEST-AUTH implementation.  I tried to
get away with "certainly not for a given qop or md5-sess".

But that's guesswork, I could as well take "not if CNONCE
isn't empty", or similar combinations.

Frank

Date: Wed, 27 Sep 2006 20:52:30 +0100
From: Alexey Melnikov <[email-masked]>
To: Lisa Dusseault <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use  UTF-8
Cc: HTTP authentication list <[email-masked]>, HTTP Working Group <[email-masked]>

Martin Duerst wrote:

>I can help with the pieces related to i18n/utf-8, but I'm
>not at all an expert in HTTP auth, so I won't try, sorry.
>  
>
I can do 2617 update, but I am not an HTTP expert either.
So I would need a co-editor, who is.

>Regards,     Martin.
>
>At 03:31 06/09/26, Lisa Dusseault wrote:
>  
>
>>I agree it would be good to update RFC2617.  Is anybody going to take  
>>a stab at it?
>>
>>Lisa
>>    
>>

Date: Wed, 27 Sep 2006 16:30:51 +0100
From: Alexey Melnikov <[email-masked]>
To: Martin Duerst <[email-masked]>
Subject: Re: [Ietf-http-auth] Re: Updating RFC 2617 (HTTP Digest) to use UTF-8
Cc: Frank Ellermann <[email-masked]>, [email-masked]

Martin Duerst wrote:

>At 22:40 06/09/26, Frank Ellermann wrote:
>  
>
>>Lisa Dusseault wrote:
>>    
>>
>>>Is anybody going to take a stab at it?
>>>      
>>>
>>Please merge this effort with 2831bis as far as possible.
>>
>>An important part would be the 2069 fallback.  Dropping
>>Latin-1 as default is tricky, after all it's the http 
>>default.
>>    
>>
>
>That's the THEORY bit. If you are in Europe, it may sometimes
>look like theory and practice match, but in Asia, it's really
>just theory.
>  
>
>>The last 2831bis draft
>>    
>>
>What's the full name of this thing?
>  
>
Currently draft-ietf-sasl-rfc2831bis-09.txt

>RFC 2931 is differing from HTTP as such in that it allows UTF-8
>directly (rather than RFC 2047-encoded) when properly labeled,
>and it is moving from iso-8859-1 to utf-8, which is the right
>thing to do on a higher level.
>
>But then it says:
>
>   charset
>      This directive, if present, specifies that the client has used
>      UTF-8 encoding for the username and password. If not present, the
>      username and password must be encoded in ISO 8859-1 (of which
>      US-ASCII is a subset). The client should send this directive only
>      if the server has indicated it supports UTF-8. The directive is
>      needed for backwards compatibility with HTTP Digest, which only
>      supports ISO 8859-1.
>
>According to Bjoern's research, the claim that HTTP Digest only
>supports ISO 8859-1 is wrong. And RFC 2617 doesn't mention
>ISO 8859-1 once, so it's also not supported. Whoever wrote
>RFC 2617 (it has a lot of authors) didn't think about
>internationalization, and what we are seeing now is the result
>of it.
>  
>
I agree.
It sounds like RFC 2617bis should just go for UTF-8.

Date: Wed, 27 Sep 2006 17:18:42 +0900
To: Frank Ellermann <[email-masked]>, [email-masked]
From: Martin Duerst <[email-masked]>
Subject: Re: [Ietf-http-auth] Re: Updating RFC 2617 (HTTP Digest) to use UTF-8

At 22:40 06/09/26, Frank Ellermann wrote:
>Lisa Dusseault wrote:
> 
>> Is anybody going to take a stab at it?
>
>Please merge this effort with 2831bis as far as possible.
>
>An important part would be the 2069 fallback.  Dropping
>Latin-1 as default is tricky, after all it's the http 
>default.

That's the THEORY bit. If you are in Europe, it may sometimes
look like theory and practice match, but in Asia, it's really
just theory.

>The last 2831bis draft

What's the full name of this thing?

RFC 2931 is differing from HTTP as such in that it allows UTF-8
directly (rather than RFC 2047-encoded) when properly labeled,
and it is moving from iso-8859-1 to utf-8, which is the right
thing to do on a higher level.

But then it says:

   charset
      This directive, if present, specifies that the client has used
      UTF-8 encoding for the username and password. If not present, the
      username and password must be encoded in ISO 8859-1 (of which
      US-ASCII is a subset). The client should send this directive only
      if the server has indicated it supports UTF-8. The directive is
      needed for backwards compatibility with HTTP Digest, which only
      supports ISO 8859-1.

According to Bjoern's research, the claim that HTTP Digest only
supports ISO 8859-1 is wrong. And RFC 2617 doesn't mention
ISO 8859-1 once, so it's also not supported. Whoever wrote
RFC 2617 (it has a lot of authors) didn't think about
internationalization, and what we are seeing now is the result
of it.

Regards,     Martin.

>with it's "backslash
>canonicalization" needs a test case, and I've no idea
>why this incompatibility with RFC 2831 is necessary.
>
>Frank
>
>
>_______________________________________________
>Ietf-http-auth mailing list
>[email-masked]
>http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]     

To: [email-masked]
From: Frank Ellermann <[email-masked]>
Date: Tue, 26 Sep 2006 15:40:47 +0200
Organization: <URL:http://purl.net/xyzzy>
Lines: 14
Sender: news <[email-masked]>
Subject: [Ietf-http-auth] Re: Updating RFC 2617 (HTTP Digest) to use UTF-8

Lisa Dusseault wrote:
 
> Is anybody going to take a stab at it?

Please merge this effort with 2831bis as far as possible.

An important part would be the 2069 fallback.  Dropping
Latin-1 as default is tricky, after all it's the http 
default.  The last 2831bis draft with it's "backslash
canonicalization" needs a test case, and I've no idea
why this incompatibility with RFC 2831 is necessary.

Frank

Date: Tue, 26 Sep 2006 16:42:39 +0900
To: Lisa Dusseault <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Cc: HTTP authentication list <[email-masked]>, HTTP Working Group <[email-masked]>

I can help with the pieces related to i18n/utf-8, but I'm
not at all an expert in HTTP auth, so I won't try, sorry.

Regards,     Martin.

At 03:31 06/09/26, Lisa Dusseault wrote:
>I agree it would be good to update RFC2617.  Is anybody going to take  
>a stab at it?
>
>Lisa

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]     

From: Stefan Eissing <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use   UTF-8
Date: Tue, 26 Sep 2006 09:42:08 +0200
To: Stefan Eissing <[email-masked]>
Cc: [email-masked], [email-masked]

Am 26.09.2006 um 09:39 schrieb Stefan Eissing:

>
> Am 26.09.2006 um 03:56 schrieb Martin Duerst:
>
>>
>> I agree with Julian and Bjoern here: IN THEORY, RFC 2047
>> (http://www.ietf.org/rfc/rfc2047.txt) (except for iso-8859-1)
>> would apply, but:
>> - it's not a very good theory: outdated because based on a view
>>   that the Web is basically iso-8859-1, and difficult to implement
>> - practice is different
>> - IETF policy is different
>
> For interoperability, the encoding of usernames and passwords seems =20=

> to be most interesting. If the encoding of the realm is =20
> misunderstood by the client (in a deterministic way) then interop =20
> could still be achieved, right?

To answer myself: not in digest authentication, stupid.

//Stefan

> //Stefan
>
>> Regards,    Martin.
>>
>> At 09:48 06/09/26, Bjoern Hoehrmann wrote:
>>> * Julian Reschke wrote:
>>>> Jim Luther schrieb:
>>>>> While we're on this subject... In rfc2617 secction 3.2.1, it says:
>>>>>
>>>>>> realm
>>>>>>      A string to be displayed to users so they know which =20
>>>>>> username and
>>>>>>      password to use.
>>>>>
>>>>> It would be also nice to define the encoding of the realm =20
>>>>> string so that
>>>>> clients that display the realm to users can display it =20
>>>>> correctly. We've
>>>>> seen realms from servers encoded UTF-8, ISO-8859-1, and with =20
>>>>> various
>>>>> Windows encodings. There's no good way to guess which encoding =20
>>>>> to use
>>>>> and so whatever is used is currently wrong on some servers.
>>>
>>>> I was thinking "should be UTF-8, of course". But doesn't really =20
>>>> RFC2045
>>>> apply here at least in theory?
>>>
>>> The realm-value is a quoted-string, and quoted-string is defined as
>>>
>>>       quoted-string  =3D ( <"> *(qdtext | quoted-pair ) <"> )
>>>       qdtext         =3D <any TEXT except <">>
>>>
>>> and TEXT is
>>>
>>>   The TEXT rule is only used for descriptive field contents and =20
>>> values
>>>   that are not intended to be interpreted by the message parser. =20
>>> Words
>>>   of *TEXT MAY contain characters from character sets other than =20
>>> ISO-
>>>   8859-1 [22] only when encoded according to the rules of RFC 2047
>>>   [14].
>>>
>>>       TEXT           =3D <any OCTET except CTLs,
>>>                        but including LWS>
>>>
>>> So you could use realm=3D"=3D?utf-8?b?..." or its variants. As you =20=

>>> say, in
>>> theory; I am unaware of any implementation that supports encoded =20
>>> words
>>> in HTTP headers..
>>> --=20
>>> Bj=EF=BF=BDn H=EF=BF=BDrmann =E3=82=AD mailto:[email-masked] =
=E3=82=AD http://=20
>>> bjoern.hoehrmann.de
>>> Weinh. Str. 22 =E3=82=AD Telefon: +49(0)621/4309674 =E3=82=AD =
http://=20
>>> www.bjoernsworld.de
>>> 68309 Mannheim =E3=82=AD PGP Pub. KeyID: 0xA4357E78 =E3=82=AD =
http://=20
>>> www.websitedev.de/
>>> _______________________________________________
>>> Ietf-http-auth mailing list
>>> [email-masked]
>>> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-=20=

>>> auth
>>
>>
>> #-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
>> #-#-#  http://www.sw.it.aoyama.ac.jp       =20
>> mailto:[email-masked]
>>
>>
>
> _______________________________________________
> Ietf-http-auth mailing list
> [email-masked]
> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

From: Stefan Eissing <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use   UTF-8
Date: Tue, 26 Sep 2006 09:39:07 +0200
To: Martin Duerst <[email-masked]>
Cc: [email-masked], [email-masked]

Am 26.09.2006 um 03:56 schrieb Martin Duerst:

>
> I agree with Julian and Bjoern here: IN THEORY, RFC 2047
> (http://www.ietf.org/rfc/rfc2047.txt) (except for iso-8859-1)
> would apply, but:
> - it's not a very good theory: outdated because based on a view
>   that the Web is basically iso-8859-1, and difficult to implement
> - practice is different
> - IETF policy is different

For interoperability, the encoding of usernames and passwords seems =20
to be most interesting. If the encoding of the realm is misunderstood =20=

by the client (in a deterministic way) then interop could still be =20
achieved, right?

//Stefan

> Regards,    Martin.
>
> At 09:48 06/09/26, Bjoern Hoehrmann wrote:
>> * Julian Reschke wrote:
>>> Jim Luther schrieb:
>>>> While we're on this subject... In rfc2617 secction 3.2.1, it says:
>>>>
>>>>> realm
>>>>>      A string to be displayed to users so they know which =20
>>>>> username and
>>>>>      password to use.
>>>>
>>>> It would be also nice to define the encoding of the realm string =20=

>>>> so that
>>>> clients that display the realm to users can display it =20
>>>> correctly. We've
>>>> seen realms from servers encoded UTF-8, ISO-8859-1, and with =20
>>>> various
>>>> Windows encodings. There's no good way to guess which encoding =20
>>>> to use
>>>> and so whatever is used is currently wrong on some servers.
>>
>>> I was thinking "should be UTF-8, of course". But doesn't really =20
>>> RFC2045
>>> apply here at least in theory?
>>
>> The realm-value is a quoted-string, and quoted-string is defined as
>>
>>       quoted-string  =3D ( <"> *(qdtext | quoted-pair ) <"> )
>>       qdtext         =3D <any TEXT except <">>
>>
>> and TEXT is
>>
>>   The TEXT rule is only used for descriptive field contents and =20
>> values
>>   that are not intended to be interpreted by the message parser. =20
>> Words
>>   of *TEXT MAY contain characters from character sets other than ISO-
>>   8859-1 [22] only when encoded according to the rules of RFC 2047
>>   [14].
>>
>>       TEXT           =3D <any OCTET except CTLs,
>>                        but including LWS>
>>
>> So you could use realm=3D"=3D?utf-8?b?..." or its variants. As you =20=

>> say, in
>> theory; I am unaware of any implementation that supports encoded =20
>> words
>> in HTTP headers..
>> --=20
>> Bj=EF=BF=BDn H=EF=BF=BDrmann =E3=82=AD mailto:[email-masked] =E3=82=
=AD http://=20
>> bjoern.hoehrmann.de
>> Weinh. Str. 22 =E3=82=AD Telefon: +49(0)621/4309674 =E3=82=AD http://=20=

>> www.bjoernsworld.de
>> 68309 Mannheim =E3=82=AD PGP Pub. KeyID: 0xA4357E78 =E3=82=AD http://=20=

>> www.websitedev.de/
>> _______________________________________________
>> Ietf-http-auth mailing list
>> [email-masked]
>> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-=20
>> auth
>
>
> #-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
> #-#-#  http://www.sw.it.aoyama.ac.jp       =20
> mailto:[email-masked]
>
>

Date: Tue, 26 Sep 2006 10:56:24 +0900
To: Bjoern Hoehrmann <[email-masked]>, Julian Reschke <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Cc: [email-masked], [email-masked]

I agree with Julian and Bjoern here: IN THEORY, RFC 2047
(http://www.ietf.org/rfc/rfc2047.txt) (except for iso-8859-1)
would apply, but:
- it's not a very good theory: outdated because based on a view
  that the Web is basically iso-8859-1, and difficult to implement
- practice is different
- IETF policy is different

Regards,    Martin.

At 09:48 06/09/26, Bjoern Hoehrmann wrote:
>* Julian Reschke wrote:
>>Jim Luther schrieb:
>>> While we're on this subject... In rfc2617 secction 3.2.1, it says:
>>> 
>>>> realm
>>>>      A string to be displayed to users so they know which username and
>>>>      password to use.
>>> 
>>> It would be also nice to define the encoding of the realm string so that 
>>> clients that display the realm to users can display it correctly. We've 
>>> seen realms from servers encoded UTF-8, ISO-8859-1, and with various 
>>> Windows encodings. There's no good way to guess which encoding to use 
>>> and so whatever is used is currently wrong on some servers.
>
>>I was thinking "should be UTF-8, of course". But doesn't really RFC2045 
>>apply here at least in theory?
>
>The realm-value is a quoted-string, and quoted-string is defined as
>
>       quoted-string  = ( <"> *(qdtext | quoted-pair ) <"> )
>       qdtext         = <any TEXT except <">>
>
>and TEXT is
>
>   The TEXT rule is only used for descriptive field contents and values
>   that are not intended to be interpreted by the message parser. Words
>   of *TEXT MAY contain characters from character sets other than ISO-
>   8859-1 [22] only when encoded according to the rules of RFC 2047
>   [14].
>
>       TEXT           = <any OCTET except CTLs,
>                        but including LWS>
>
>So you could use realm="=?utf-8?b?..." or its variants. As you say, in
>theory; I am unaware of any implementation that supports encoded words
>in HTTP headers..
>-- 
>Bj$B‹S(Bn H$B‹I(Brmann $B%-(B mailto:[email-masked] $B%-(B http://bjoern.hoehrmann.de
>Weinh. Str. 22 $B%-(B Telefon: +49(0)621/4309674 $B%-(B http://www.bjoernsworld.de
>68309 Mannheim $B%-(B PGP Pub. KeyID: 0xA4357E78 $B%-(B http://www.websitedev.de/ 
>_______________________________________________
>Ietf-http-auth mailing list
>[email-masked]
>http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]     

From: Bjoern Hoehrmann <[email-masked]>
To: Julian Reschke <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Date: Tue, 26 Sep 2006 02:48:17 +0200
Cc: [email-masked], [email-masked]

* Julian Reschke wrote:
>Jim Luther schrieb:
>> While we're on this subject... In rfc2617 secction 3.2.1, it says:
>> 
>>> realm
>>>      A string to be displayed to users so they know which username and
>>>      password to use.
>> 
>> It would be also nice to define the encoding of the realm string so that 
>> clients that display the realm to users can display it correctly. We've 
>> seen realms from servers encoded UTF-8, ISO-8859-1, and with various 
>> Windows encodings. There's no good way to guess which encoding to use 
>> and so whatever is used is currently wrong on some servers.

>I was thinking "should be UTF-8, of course". But doesn't really RFC2045 
>apply here at least in theory?

The realm-value is a quoted-string, and quoted-string is defined as

       quoted-string  = ( <"> *(qdtext | quoted-pair ) <"> )
       qdtext         = <any TEXT except <">>

and TEXT is

   The TEXT rule is only used for descriptive field contents and values
   that are not intended to be interpreted by the message parser. Words
   of *TEXT MAY contain characters from character sets other than ISO-
   8859-1 [22] only when encoded according to the rules of RFC 2047
   [14].

       TEXT           = <any OCTET except CTLs,
                        but including LWS>

So you could use realm="=?utf-8?b?..." or its variants. As you say, in
theory; I am unaware of any implementation that supports encoded words
in HTTP headers..
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 

Date: Tue, 26 Sep 2006 02:42:39 +0200
From: Julian Reschke <[email-masked]>
To: HTTP Working Group <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Cc: HTTP authentication list <[email-masked]>

Jim Luther schrieb:
> 
> While we're on this subject... In rfc2617 secction 3.2.1, it says:
> 
>> realm
>>      A string to be displayed to users so they know which username and
>>      password to use.
> 
> It would be also nice to define the encoding of the realm string so that 
> clients that display the realm to users can display it correctly. We've 
> seen realms from servers encoded UTF-8, ISO-8859-1, and with various 
> Windows encodings. There's no good way to guess which encoding to use 
> and so whatever is used is currently wrong on some servers.

Hm.

I was thinking "should be UTF-8, of course". But doesn't really RFC2045 
apply here at least in theory?

Best regards, Julian

Date: Tue, 26 Sep 2006 01:21:03 +0200
To: "Bjoern Hoehrmann" <[email-masked]>, "Julian Reschke" <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
From: "Yngve N. Pettersen (Developer Opera Software ASA)" <[email-masked]>
Organization: Opera Software AS
Cc: HTTP authentication list <[email-masked]>, HTTP Working Group <[email-masked]>

On Tue, 26 Sep 2006 00:08:11 +0200, Bjoern Hoehrmann <[email-masked]>  
wrote:

> * Julian Reschke wrote:
>> Speaking of which, where does HTML come into play here? We're talking
>> about HTTP authentication à la RFC2617, not HTML forms based login.
>
> For HTML form submissions, unless the author indicated something else,
> web browsers tend to use the character encoding of the document that in-
> cludes the form to encode the characters; they could apply the same
> logic to the submission of credentials when there is such a document
> (e.g., the user clicked a link to a HTTP Auth protected page, the page
> with the link could then be used to determine some encoding). Based on
> my limited testing, I found this to be not the case.

In the HTML case the information is directly available, as part of the  
form's own environment, which it would not be when the authentication  
credentials are being processed.

Which characterset/encoding should the application choose when that  
information is not available?

E.g. The information would not be available if the user goes directly to  
the site, by entering the URL directly. Similar situation could arise when  
the URL is loaded in another tab than the originating document, and  
obtaining the information would also not be entirely straight forward even  
if it is opened in the same tab. Or what about proxy authentication?

Also: What if the original page is using a different characterset/encoding  
than used by the server requesting authentication? E.g. What if a Russian  
language page directs you to an authenticated Japanese site? And AFAIK  
several languages actually have multiple encodings.

Also, characterset/encoding information may not be available in the HTTP  
header either.

It is not possible to define a heuristic that will fit all scenarios. The  
best approach is to define a common characterset/encoding that will be  
used by all compliant servers.

As RFC 2617 was not able to assist, the only guidance I had when I chose  
the I18N policy for Opera's RFC 2617 support, was RFC 2277/BCP 18, which I  
interprete to say that protocols should use UTF-8 unless they specify  
otherwise either in the specification (i.e. the RFC) or in a specific  
field of the protocol (in this case, that would mean an attribute in the  
WWW-Authenticate header). The problem is that work on the RFC 2617  
protocol probably started before RFC 2217 was finished.

Given that the current system is broken anyway, since client and server  
have to agree out-of-band on which characterset/encoding to use, it is in  
my opinion best to define a proper solution, which IMO means UTF-8,  
instead of trying to patch up the broken system . (And remember: Even a  
patch of the current system would have to be deployed in new clients and  
servers).

-- 
Sincerely,
Yngve N. Pettersen
 
********************************************************************
Senior Developer                     Email: [email-masked]
Opera Software ASA                   http://www.opera.com/
Phone:  +47 24 16 42 60              Fax:    +47 24 16 40 01
********************************************************************

From: Bjoern Hoehrmann <[email-masked]>
To: Julian Reschke <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Date: Tue, 26 Sep 2006 00:08:11 +0200
Cc: HTTP authentication list <[email-masked]>, HTTP Working Group <[email-masked]>

* Julian Reschke wrote:
>Speaking of which, where does HTML come into play here? We're talking 
>about HTTP authentication à la RFC2617, not HTML forms based login.

For HTML form submissions, unless the author indicated something else,
web browsers tend to use the character encoding of the document that in-
cludes the form to encode the characters; they could apply the same
logic to the submission of credentials when there is such a document
(e.g., the user clicked a link to a HTTP Auth protected page, the page
with the link could then be used to determine some encoding). Based on
my limited testing, I found this to be not the case.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/ 

Date: Mon, 25 Sep 2006 23:56:08 +0200
From: Julian Reschke <[email-masked]>
To: Chris Drake <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Cc: HTTP authentication list <[email-masked]>, HTTP Working Group <[email-masked]>

Chris Drake schrieb:
> Hi,
> 
> Do not the HTTP headers from the browser already say what character
> encoding has been used, and do not the HTTP headers and HTML from the

That's for the request body, not the headers.

> server beforehand dictate this stuff too?  Except for the case where
> someone is trying to type foreign characters on a PC which can't
> understand them (eg: holidaymakers at net cafes where the net cafe has
> not installed the language they want to use) - everything should
> already be working just fine, yes?

I don't think so.

Speaking of which, where does HTML come into play here? We're talking 
about HTTP authentication à la RFC2617, not HTML forms based login.

Best regards, Julian

Date: Tue, 26 Sep 2006 07:47:12 +1000
From: Chris Drake <[email-masked]>
To: Lisa Dusseault <[email-masked]>
Subject: Re[2]: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Cc: HTTP authentication list <[email-masked]>, HTTP Working Group <[email-masked]>

Hi,

Do not the HTTP headers from the browser already say what character
encoding has been used, and do not the HTTP headers and HTML from the
server beforehand dictate this stuff too?  Except for the case where
someone is trying to type foreign characters on a PC which can't
understand them (eg: holidaymakers at net cafes where the net cafe has
not installed the language they want to use) - everything should
already be working just fine, yes?

Kind Regards,
Chris Drake

Tuesday, September 26, 2006, 4:31:13 AM, you wrote:

LD> I agree it would be good to update RFC2617.  Is anybody going to take
LD> a stab at it?

LD> Lisa

LD> On Sep 22, 2006, at 7:55 PM, Martin Duerst wrote:

>> Many thanks to Bjoern for the detailed checking and report.
>>
>> My summary of the situation would be as follows:
>> There is currently widely varying practice. Current implementations
>> are anyways broken and non-interoperable. The main reason for this
>> is most probably that there is no clear specification. This means
>> that an update of RFC 2617 is desirable. The new specification
>> should probably go for UTF-8, while noting that there is still
>> some varying practice.
>>
>> Regards,   Martin.
>>
>>
>> At 01:41 06/09/23, Bjoern Hoehrmann wrote:
>>> * Alexey Melnikov wrote:
>>>> Does anybody know if updating RFC 2617 to say that username/ 
>>>> passwords
>>>> are UTF-8 would break any major implementation? For example, does
>>>> anybody know if a major HTTP client/server implementation assume
>>>> ISO 8859-1?
>>>
>>> It appears that for Basic authentication the german version of  
>>> Internet
>>> Explorer 6 running on the german version of Windows 2003 as well  
>>> as the
>>> latest english Internet Explorer 7 release candidate running on the
>>> german version of Windows XP will use something like ISO-8859-1  
>>> for both
>>> manual as well as XMLHttpRequest requests. Trying to use U+20AC as
>>> user
>>> name and password they got encoded as 0x80 (Windows-1252) for  
>>> manual re-
>>> quests, and to '?' for XHR. Characters not included in  
>>> Windows-1252 come
>>> out as '?' regardless of the method used. For XHR my test cases  
>>> include
>>> documents encoded as ISO-8859-1 and UTF-8; there did not appear to be
>>> any difference.
>>>
>>> The latest en-us version of Firefox uses UTF-8 for XHR and the lower
>>> byte of the character when encoded using UTF-16BE (so for U+20AC you
>>> get 0xAC) when using manual input. For manually entered http:// 
>>> u:p@...
>>> URLs Firefox uses Windows-1252 if possible, UTF-8 otherwise. When XHR
>>> is used with such a URL, it uses UTF-8. The latest en-us version of
>>> Opera9 always uses UTF-8, as far as I can tell based on my limited
>>> testing. Results might well be different on with different default
>>> code
>>> pages, language settings, and so on. Note that the illegal http://
>>> u:p@..
>>> addressing scheme allows to use arbitrary octet sequences using %hh
>>> escape sequences, with some browser-specific limitations.
>>> -- 
>>> Bj?n H?rmann ? mailto:[email-masked] ? http:// 
>>> bjoern.hoehrmann.de
>>> Weinh. Str. 22 ? Telefon: +49(0)621/4309674 ? http:// 
>>> www.bjoernsworld.de
>>> 68309 Mannheim ? PGP Pub. KeyID: 0xA4357E78 ? http:// 
>>> www.websitedev.de/
>>> _______________________________________________
>>> Ietf-http-auth mailing list
>>> [email-masked]
>>> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-
>>> auth
>>
>>
>> #-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
>> #-#-#  http://www.sw.it.aoyama.ac.jp        
>> mailto:[email-masked]
>>
>> _______________________________________________
>> Ietf-http-auth mailing list
>> [email-masked]
>> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

LD> _______________________________________________
LD> Ietf-http-auth mailing list
LD> [email-masked]
LD> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

From: Lisa Dusseault <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Date: Mon, 25 Sep 2006 11:31:13 -0700
To: Martin Duerst <[email-masked]>
Cc: HTTP authentication list <[email-masked]>, HTTP Working Group <[email-masked]>

I agree it would be good to update RFC2617.  Is anybody going to take =20=

a stab at it?

Lisa

On Sep 22, 2006, at 7:55 PM, Martin Duerst wrote:

> Many thanks to Bjoern for the detailed checking and report.
>
> My summary of the situation would be as follows:
> There is currently widely varying practice. Current implementations
> are anyways broken and non-interoperable. The main reason for this
> is most probably that there is no clear specification. This means
> that an update of RFC 2617 is desirable. The new specification
> should probably go for UTF-8, while noting that there is still
> some varying practice.
>
> Regards,   Martin.
>
>
> At 01:41 06/09/23, Bjoern Hoehrmann wrote:
>> * Alexey Melnikov wrote:
>>> Does anybody know if updating RFC 2617 to say that username/=20
>>> passwords
>>> are UTF-8 would break any major implementation? For example, does
>>> anybody know if a major HTTP client/server implementation assume =20
>>> ISO 8859-1?
>>
>> It appears that for Basic authentication the german version of =20
>> Internet
>> Explorer 6 running on the german version of Windows 2003 as well =20
>> as the
>> latest english Internet Explorer 7 release candidate running on the
>> german version of Windows XP will use something like ISO-8859-1 =20
>> for both
>> manual as well as XMLHttpRequest requests. Trying to use U+20AC as =20=

>> user
>> name and password they got encoded as 0x80 (Windows-1252) for =20
>> manual re-
>> quests, and to '?' for XHR. Characters not included in =20
>> Windows-1252 come
>> out as '?' regardless of the method used. For XHR my test cases =20
>> include
>> documents encoded as ISO-8859-1 and UTF-8; there did not appear to be
>> any difference.
>>
>> The latest en-us version of Firefox uses UTF-8 for XHR and the lower
>> byte of the character when encoded using UTF-16BE (so for U+20AC you
>> get 0xAC) when using manual input. For manually entered http://=20
>> u:p@...
>> URLs Firefox uses Windows-1252 if possible, UTF-8 otherwise. When XHR
>> is used with such a URL, it uses UTF-8. The latest en-us version of
>> Opera9 always uses UTF-8, as far as I can tell based on my limited
>> testing. Results might well be different on with different default =20=

>> code
>> pages, language settings, and so on. Note that the illegal http://=20
>> u:p@..
>> addressing scheme allows to use arbitrary octet sequences using %hh
>> escape sequences, with some browser-specific limitations.
>> --=20
>> Bj=EF=BF=BDn H=EF=BF=BDrmann =E3=82=AD mailto:[email-masked] =E3=82=
=AD http://=20
>> bjoern.hoehrmann.de
>> Weinh. Str. 22 =E3=82=AD Telefon: +49(0)621/4309674 =E3=82=AD http://=20=

>> www.bjoernsworld.de
>> 68309 Mannheim =E3=82=AD PGP Pub. KeyID: 0xA4357E78 =E3=82=AD http://=20=

>> www.websitedev.de/
>> _______________________________________________
>> Ietf-http-auth mailing list
>> [email-masked]
>> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-=20
>> auth
>
>
> #-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
> #-#-#  http://www.sw.it.aoyama.ac.jp       =20
> mailto:[email-masked]
>
> _______________________________________________
> Ietf-http-auth mailing list
> [email-masked]
> http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

Date: Sat, 23 Sep 2006 11:55:42 +0900
To: Bjoern Hoehrmann <[email-masked]>, Alexey Melnikov <[email-masked]>
From: Martin Duerst <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Cc: [email-masked]

Many thanks to Bjoern for the detailed checking and report.

My summary of the situation would be as follows:
There is currently widely varying practice. Current implementations
are anyways broken and non-interoperable. The main reason for this
is most probably that there is no clear specification. This means
that an update of RFC 2617 is desirable. The new specification
should probably go for UTF-8, while noting that there is still
some varying practice.

Regards,   Martin.

At 01:41 06/09/23, Bjoern Hoehrmann wrote:
>* Alexey Melnikov wrote:
>>Does anybody know if updating RFC 2617 to say that username/passwords 
>>are UTF-8 would break any major implementation? For example, does 
>>anybody know if a major HTTP client/server implementation assume ISO 8859-1?
>
>It appears that for Basic authentication the german version of Internet
>Explorer 6 running on the german version of Windows 2003 as well as the
>latest english Internet Explorer 7 release candidate running on the
>german version of Windows XP will use something like ISO-8859-1 for both
>manual as well as XMLHttpRequest requests. Trying to use U+20AC as user
>name and password they got encoded as 0x80 (Windows-1252) for manual re-
>quests, and to '?' for XHR. Characters not included in Windows-1252 come
>out as '?' regardless of the method used. For XHR my test cases include
>documents encoded as ISO-8859-1 and UTF-8; there did not appear to be
>any difference.
>
>The latest en-us version of Firefox uses UTF-8 for XHR and the lower
>byte of the character when encoded using UTF-16BE (so for U+20AC you
>get 0xAC) when using manual input. For manually entered http://u:p@...
>URLs Firefox uses Windows-1252 if possible, UTF-8 otherwise. When XHR
>is used with such a URL, it uses UTF-8. The latest en-us version of
>Opera9 always uses UTF-8, as far as I can tell based on my limited
>testing. Results might well be different on with different default code
>pages, language settings, and so on. Note that the illegal http://u:p@..
>addressing scheme allows to use arbitrary octet sequences using %hh
>escape sequences, with some browser-specific limitations.
>-- 
>Bj$B‹S(Bn H$B‹I(Brmann $B%-(B mailto:[email-masked] $B%-(B http://bjoern.hoehrmann.de
>Weinh. Str. 22 $B%-(B Telefon: +49(0)621/4309674 $B%-(B http://www.bjoernsworld.de
>68309 Mannheim $B%-(B PGP Pub. KeyID: 0xA4357E78 $B%-(B http://www.websitedev.de/ 
>_______________________________________________
>Ietf-http-auth mailing list
>[email-masked]
>http://lists.osafoundation.org/cgi-bin/mailman/listinfo/ietf-http-auth

#-#-#  Martin J. Du"rst, Assoc. Professor, Aoyama Gakuin University
#-#-#  http://www.sw.it.aoyama.ac.jp       mailto:[email-masked]     

From: Bjoern Hoehrmann <[email-masked]>
To: Alexey Melnikov <[email-masked]>
Subject: Re: [Ietf-http-auth] Updating RFC 2617 (HTTP Digest) to use UTF-8
Date: Fri, 22 Sep 2006 18:41:36 +0200
Cc: [email-masked]

* Alexey Melnikov wrote:
>Does anybody know if updating RFC 2617 to say that username/passwords 
>are UTF-8 would break any major implementation? For example, does 
>anybody know if a major HTTP client/server implementation assume ISO 8859-1?

It appears that for Basic authentication the german version of Internet
Explorer 6 running on the german version of Windows 2003 as well as the
latest english Internet Explorer 7 release candidate running on the
german version of Windows XP will use something like ISO-8859-1 for both
manual as well as XMLHttpRequest requests. Trying to use U+20AC as user
name and password they got encoded as 0x80 (Windows-1252) for manual re-
quests, and to '?' for XHR. Characters not included in Windows-1252 come
out as '?' regardless of the method used. For XHR my test cases include
documents encoded as ISO-8859-1 and UTF-8; there did not appear to be
any difference.

The latest en-us version of Firefox uses UTF-8 for XHR and the lower
byte of the character when encoded using UTF-16BE (so for U+20AC you
get 0xAC) when using manual input. For manually entered http://u:p@...
URLs Firefox uses Windows-1252 if possible, UTF-8 otherwise. When XHR
is used with such a URL, it uses UTF-8. The latest en-us version of
Opera9 always uses UTF-8, as far as I can tell based on my limited
testing. Results might well be different on with different default code
pages, language settings, and so on. Note that the illegal http://u:p@..
addressing scheme allows to use arbitrary octet sequences using %hh
escape sequences, with some browser-specific limitations.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Weinh. Str. 22 · Telefon: +49(0)621/4309674 · http://www.bjoernsworld.de
68309 Mannheim · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/