Subject: Re: [http-auth] Unicode normalization, was: Draft Minutes Posted for IETF 87 HTTP-AUTH Session

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Feb 2014 17:55:59 +0100

On 2013-10-05 11:01, Julian Reschke wrote:
> On 2013-09-12 12:35, Julian Reschke wrote:
>> On 2013-08-21 21:22, Matthew Lepinski wrote:
>>> Draft minutes for the HTTP-AUTH session have been posted.
>>>
>>> They can be found at:
>>> http://www.ietf.org/proceedings/87/minutes/minutes-87-httpauth
>>>
>>> If you notice any omissions or other errors in the minutes, please let
>>> us know.
>>> ...
>>
>> OK, the minutes mention:
>>
>> "Unicode Normalization : Getting from what is typed in to Unicode code
>> points will require discussion"
>>
>> So how do we proceed from here? Any concrete proposals for what to say?
>
> It seems we don't know what to say then, right?
>
> How about: "Beware that differing Unicode normalization forms can cause
> interoperability problems. See [http://unicode.org/reports/tr15/]."?
>
>
> Best regards, Julian

So, does anybody have a good plan how to approach the normalization problem?

Otherwise we'll just have to state that there are dragons out there, and 
that we don't know the solution...

Best regards, Julian

---

From: Michael Sweet <[email-masked]>
Date: Mon, 03 Feb 2014 12:06:42 -0500

Julian,

I thought (perhaps just hoped) we were going to reference RFC 5198 (network unicode) and provide some words about normalization forms so that clients and servers don't blindly send and receive UTF-8 regardless of the normalization form.

On Feb 3, 2014, at 11:55 AM, Julian Reschke <[email-masked]> wrote:

> Otherwise we'll just have to state that there are dragons out there, and that we don't know the solution...
> 
> Best regards, Julian
> 
> _______________________________________________
> http-auth mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/http-auth

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Feb 2014 19:10:55 +0100

> I thought (perhaps just hoped) we were going to reference RFC 5198 (network unicode) and provide some words about normalization forms so that clients and servers don't blindly send and receive UTF-…

Well, what *exactly* should we say, and is that deployable in practice?

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Mon, 3 Feb 2014 12:47:53 -0600

> So, does anybody have a good plan how to approach the normalization problem?

Strictly in the context of authentication, or more generally in the
context of apps like WebDAV?

In general, IMO, the best thing to do is to RECOMMEND
normalization-insensitive string comparison semantics, applied at the
last possible moment, and only if necessary.

So, for example, in the case of Basic auth any normalization can be
left to the server.  But since so many existing servers (all of them
using Basic) fail to do this, it may help to RECOMMEND that the client
normalize to NFC.

In the case of Digest the password needs to be in a canonical form on
both sides (I know, the server stores a verifier; that verifier needs
to be computed from the canonical password) before proceeding.  The
same "but" applies to the username and domain as to the username in
Basic.

In the case of something like a filesystem, file names should be
matched in a normalization-insensitive (but preferably
form-preserving) manner.  But this can only be reasonably implemented
in the filesystem itself, not a WebDAV server (or NFSv4, or...), so if
the filesystem doesn't do this then the only alternatives are: "tough"
and "normalize on CREATE and LOOKUP on the server".  Note that
normalizing on CREATE and LOOKUP on the client precludes any migration
to "the filesystem makes it right" model.

All of this applies any time a protocol exposes a "search" or "match"
function on Unicode strings.  So if a server exposes a "search all
documents for input string" function, then it'd be best if the search
(and any indexing) could be done in normalization-insensitive,
form-preserving manner.

All of this applies to Internet protocols in general, not just to HTTP auth.

> that we don't know the solution...

That's fine for HTTP auth.  For Latin scripts it really helps that
input modes almost universally (if not universally) output
pre-composed codepoints.  For other scripts the situation may be
messier.

Nico
--

---

From: Michael Sweet <[email-masked]>
Date: Mon, 03 Feb 2014 14:25:42 -0500

> Well, what *exactly* should we say, and is that deployable in practice?

How about the following (using the Digest draft as a basis, since the Basic draft currently does not include the charset parameter):

   4 Internationalization

   In challenges, servers SHOULD use the "charset" authentication
   parameter (case-insensitive) to express the character encoding they
   expect the user agent to use.

   The only allowed value is "UTF-8", to be matched case-insensitively,
   indicating that the server expects the UTF-8 character encoding to be
   used ([RFC3629][RFC5198]).

   If the user agent supports the encoding indicated by the server, it
   MUST use normalization form D as required for the Unicode Format for
   Network Interchange [RFC5198] and SHOULD add the "charset" parameter,
   with the value it received from the server, to the Proxy-Authenticate
   or WWW-Authenticate header fields it sends back to the server.

   If the user agent does not support the encoding indicated by the
   server, it MUST fail the request.

   Servers and user agents that support the "UTF-8" character encoding
   MUST convert any UTF-8 username or password to NFD prior to applying
   the hash function.

(The latter paragraph can be omitted for the Basic draft...)

We should also reference the RFC 3629, section 10 and RFC 5198, section 6 security considerations sections from the basic and digest security considerations sections since they talk about common UTF-8 issues for normalization, string comparison, and so forth.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Feb 2014 22:01:17 +0100

> context of apps like WebDAV?

Right now just for authentication.

> using Basic) fail to do this, it may help to RECOMMEND that the client

We'll need to make sure that if I type "Ü" in different browsers / on 
different platforms the right thing happens. Optimally, we'd test what 
those browsers that do use UTF-8 already do, but I didn't yet have time 
for this.

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Mon, 3 Feb 2014 15:40:22 -0600

> Right now just for authentication.

OK.  BTW, this has come up recently in the context of SASL and PRECIS.
 The point: it's not just about HTTP; similar issues come up in other
contexts.

Also: it's not just about authentication; similar issues come up in
other contexts.

Generic advice would be nice to give in an Informational/BCP RFC.  I'd
bolunteer to write one (since I have a fair portion of one written).

> different platforms the right thing happens. Optimally, we'd test what those
> browsers that do use UTF-8 already do, but I didn't yet have time for this.

I suspect that browsers depend on native input methods exclusively,
but maybe nowadays browsers have enough functionality not found in
native input methods that they must implement their own?

When I last looked into input methods (some years ago) I found that
Windows, OS X, Solaris, and Linux all output pre-computed codepoints
for Latin script characters I knew how to input (e.g., using Compose
key sequences or US-International keyboard layouts).

I very much doubt that any input modes produce NFC, or NFD.  I suspect
that they produce whatever is most convenient to produce given the
Compose / international keyboard layout input sequence, and internal
details (mapping tables).

The thing I found fascinating (and obnoxious) was that OS X's input
modes output pre-composed sequences but HFS+ would normalize to NFD on
CREATE.  A recipe for disaster (with a lower-case 'd' anyways) IMO.
This was one of the things that led to ZFS implementing a
form-insensitive/form-preserving scheme.

Oh, and another thing I found fascinating was less than universal
support for rendering decomposed sequences; I forget which OSes could
and which couldn't.  IIRC Solaris and Linux couldn't.

That testing was too long ago -and too focused on Latin- to be of any
value now.  Fairly comprehensive tests need to be done now.

Nico
--

---

From: Michael Sweet <[email-masked]>
Date: Mon, 03 Feb 2014 17:10:11 -0500

Err, make that *NFC*, not NFD. That'll teach me to send an email on my way out the door...

Sent from my iPad

> On Feb 3, 2014, at 2:25 PM, Michael Sweet <[email-masked]> wrote:
>

---

From: Nico Williams <[email-masked]>
Date: Mon, 3 Feb 2014 16:40:55 -0600

> Err, make that *NFC*, not NFD. That'll teach me to send an email on my way out the door...

I'd make it a SHOULD, not a MUST, explaining the likely result if one
does not normalize to NFC and what users can do about any non-interop
resulting from clients failing to normalize.  This is to document what
happens now and how users can deal with it (e.g., mostly by either
avoiding characters with multiple forms, or by avoiding systems whose
input modes do not produce expected results (NFC), which is, after
all, what basically happens every day).

Nico
--

---

From: Yutaka OIWA <[email-masked]>
Date: Tue, 4 Feb 2014 09:18:14 +0900

Dear Julian and Peter (added),

how about the things ongoing about handling of
HTTP-AUTH normalization in context of PRECIS?

I proposed general-purpose HTTP-AUTH normalization
profile to PRECIS WG (just because I need it :-),
and they considered merging it with new SASLPREPbis.
My current draft is
http://tools.ietf.org/html/draft-oiwa-precis-httpauthprep-00 .
SASLPREPbis is in WG pool as
http://tools.ietf.org/html/draft-ietf-precis-saslprepbis-06 .

I am awaiting actions for whether the merging
will actually happen or not.
In my understanding, removing of SASL-dependent
natures (e.g. that in Username grammer) from current
saslprepbis is not going forward yet, and current
SASLPREPbis is, at least personally, not applicable
for any HTTP auth schemes except SASL-backed ones.
For clarify, SASLPREPbis is really good, and the differences
are not large but critical.

I think there is several possible directions for us to go:

1) Go merging: push forward to make saslprepbis a
    general-purpose precis profile by separating
    still-remaining SASL-only features.
    IMO, in this case we may need two separate
    application notes documents for SASL and HTTP-AUTH.

2) Go separate: discuss HTTPAUTH in context of
    PRECIS separately from SASLPREP.
    I believe that my draft will give us a good starting point,
    as my best effort.

3) for Julian, one possible best current cheating, if you
    can't wait PRECIS WG, might be just specify NFC as a
    canonical form.  Both SASLPREP and HTTPAUTHprep
    (and many other PRECIS profiles) are NFC based,
    so it will not likely harm future development of proper
    PRECIS-based "preparation" (including normalization).

Also, I would be happy if Julian (as talked in Vancouver)
and other people in HTTPAUTH WG and PRECIS WG
could give us a feedback on my proposal from the
both WG's points of view.

-- 
Yutaka OIWA, Ph.D.                 Leader, System Life-cycle Research Group
                               Research Institute for Secure Systems (RISEC)
     National Institute of Advanced Industrial Science and Technology (AIST)
                       Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5]

---

From: Nico Williams <[email-masked]>
Date: Mon, 3 Feb 2014 18:32:11 -0600

+1 to using/referencing PRECIS, but first make sure that it doesn't
continue to say anything inappropriate about case folding.

Nico
--

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 03 Feb 2014 19:31:54 -0700

> continue to say anything inappropriate about case folding.

Nico, as you might recall, with your help we revised the saslprepbis 
document to remove the inappropriate text about case folding. :-)

Also, after talking with Yutaka and Julian at the last IETF meeting, 
personally I concluded that we do indeed need a separate PRECIS profile 
for HTTPAUTH (not merge it into saslprepbis).

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Mon, 03 Feb 2014 19:33:48 -0700

Yes, I think it is best to define a separate profile for HTTPAUTH (based 
on various conversations at the last IETF meeting). I will try to review 
your document again very soon.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 04 Feb 2014 16:33:15 +0100

> your document again very soon.

Peter: that would be awesome.

That being said I fear that this is going to take some time.

Alternatives for "Basic":

1) Keep the separation of the base spec 
(draft-ietf-httpauth-basicauth-update) from the one addressing I18N 
(draft-ietf-httpauth-basicauth-enc), and try to get the former out of 
the door as soon as possible.

2) As previously agreed upon, merge the two specs but only do some 
handwaving with respect to normalization for now.

Best regards, Julian

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Wed, 05 Feb 2014 02:26:16 +0100

>handwaving with respect to normalization for now.

I like 2).
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Yutaka OIWA <[email-masked]>
Date: Wed, 5 Feb 2014 18:14:51 +0900

Thank you very much for making our next steps very clear.

I'll do my best to improve our proposal.

2014-02-04 Peter Saint-Andre <[email-masked]>:
> Yes, I think it is best to define a separate profile for HTTPAUTH (based on
> various conversations at the last IETF meeting). I will try to review your
> document again very soon.
>
> Peter
>
>

---

From: Yutaka OIWA <[email-masked]>
Date: Wed, 5 Feb 2014 18:21:02 +0900

>>handwaving with respect to normalization for now.

Given taking 2), is it technically possible to make an "update" to
the merged Basic-auth spec by a PRECIS profile document defined
later (but sooner)?

# I think, if any, the update may be something to "RECOMMEND"
# or to say "SHOULD" do, or possibly "MAY" do, but not to say "MUST",
# considering very pervasive and pragmatic nature of the Basic authentication.

---

From: Yoav Nir <[email-masked]>
Date: Wed, 5 Feb 2014 09:30:49 +0000

[With http-auth chair hat on]

It is possible for one draft document to normatively reference another draft document.  It can go through WGLC, IETF-LC, and IESG review and approval with this reference. 

In such cases, the document does not get published until the reference also becomes an RFC, at which point the RFC editor converts the reference to a reference to the new RFC.

There are two downsides to this process:
 - It delays the original document indefinitely. Not a problem in our case because nobody (other than the authors is waiting for our document)
 - You get into a messy situation if the precis document ends up stalled.

If we have a direction, and we have authors volunteering to write this in precis, I'm not opposed to taking this route. 

Please note, that precise encoding issues apply to both passwords and usernames, as the new Digest algorithm allows hiding of usernames ([1])

Yoav
[1] http://tools.ietf.org/html/draft-ietf-httpauth-digest-04#section-3.4.4

-----Original Message-----
From: http-auth [mailto:[email-masked]] On Behalf Of Yutaka OIWA
Sent: Wednesday, February 05, 2014 11:21 AM
To: Bjoern Hoehrmann
Cc: Julian Reschke; [email-masked]; [email-masked]
Subject: Re: [http-auth] [precis] Unicode normalization, was: Draft Minutes Posted for IETF 87 HTTP-AUTH Session

>>handwaving with respect to normalization for now.

Given taking 2), is it technically possible to make an "update" to the merged Basic-auth spec by a PRECIS profile document defined later (but sooner)?

# I think, if any, the update may be something to "RECOMMEND"
# or to say "SHOULD" do, or possibly "MAY" do, but not to say "MUST", # considering very pervasive and pragmatic nature of the Basic authentication.

2014-02-05 Bjoern Hoehrmann <[email-masked]>:
> Björn Höhrmann · mailto:[email-masked] · 
> http://bjoern.hoehrmann.de Am Badedeich 7 · Telefon: +49(0)160/4415681 
> · http://www.bjoernsworld.de

OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5] _______________________________________________

Email secured by Check Point

---

From: Julian Reschke <[email-masked]>
Date: Wed, 05 Feb 2014 10:41:08 +0100

> # considering very pervasive and pragmatic nature of the Basic authentication.

We're going for Proposed Standard here. We can always revise the 
document again, change this, and go to Full Standard later on.

The key issue aren't the specs, but what implementations do. So we need 
to agree on what they should do and try to get *that* implemented, 
Otherwise the whole exercise is completely academic.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 05 Feb 2014 10:42:57 +0100

>   - It delays the original document indefinitely. Not a problem in our case because nobody (other than the authors is waiting for our document)

Understood, but no, I believe we should get this done. Either by 
ignoring I18N for now (a), or by documenting what happens in practice (b).

(a) Status Quo
(b) Requires research.

Best regards, Julian

---

From: Yutaka OIWA <[email-masked]>
Date: Wed, 5 Feb 2014 19:45:40 +0900

Dear Julian,

> The key issue aren't the specs, but what implementations do. So we need to agree on what they should do and try to get *that* implemented, Otherwise the whole exercise is completely academic.

Your point is really understood and agreed.
We need a thorough discussion for requirement strength
of string preparation (including Unicode normalization)
for Basic and Digest schemes.
There is a tough trade-off: if the requirement is too weak,
it decreases interoperability and the users will experience
a frustrating failed authentications; if too strong, implementors
will not just follow it.  We need to find a good mid-point.

My current position is not actually insisting PRECIS-based
handling for a MUST requirement of UTF-8 Basic.
My current preference is, assuming that the majority of people
think MUST-required PRECIS handling too strong, the following:

  "to require NFC Unicode normalization a SHOULD, and some PRECIS
   handling a MAY with stating it's a good practice to do."

The reason is: Unicode normalization is hard to be
worked-around by end users; other PRECIS preparations
are generally workaround-able by users by avoiding
some characters in their input.

(Oppositely, if the majority thought that UTF-8 Basic should do
 a correct PRESIS-based preparation as a MUST for the
 best interoperability assurance, I would happily follow it.)

 *

Regarding general issue on string preparation for HTTP authentication,
I have two strong opinions around here:

  1) "single solution":
      Within HTTPAUTH technology area, There should not be
      randomly many ways of preparations with minor differences
      with no justifiable reasons.
      There should be a single default way of preparation which will be
      used for almost all HTTP-related authentications *if they do preparation*,
      unless there are some other dependencies
      (like SASLprep for SASL-backed schemes.)

  2) "new good things for new technologies":
      For *new* auth schemes, for which we can expect they will
      have a new implementation efforts anyway,
      we have a motivation for MUST-or-SHOULD
      requirement of correct I18N handling.
      (It is something like an exercise anyway,
       so we may try to do the "correct" thing.)

However, for retrofitting schemes like Basic and Digest,
I suspect that the normative requirements for correct I18N may be
too strong.  I guess we may need to allow implementors
to "ignore" such complex handling, and still saying
their implementations are compliant to the new UTF-8 Basic.
That's why, in my HTTPAUTHprep draft, I said it will serve
as a dual-purpose document, both as a normative reference
for new things and a best practice for other existing schemes.
(See Section 1.2 of my draft, understanding the phrase is not yet polished).

2014-02-05 Julian Reschke <[email-masked]>:
> We're going for Proposed Standard here. We can always revise the document
> again, change this, and go to Full Standard later on.
>
> The key issue aren't the specs, but what implementations do. So we need to
> agree on what they should do and try to get *that* implemented, Otherwise
> the whole exercise is completely academic.
>
> Best regards, Julian
>