Subject: Re: [http-auth] Normalization forms in draft-ietf-httpauth-basicauth-enc

From: Paul Hoffman <[email-masked]>
Date: Mon, 1 Jul 2013 07:16:26 -0700

On Jun 30, 2013, at 10:59 PM, Julian Reschke <[email-masked]> wrote:

> So what if we mandated normalization to happen on the client?

And they implemented it correctly every time? That would work great.

Does the history of IDNA2003 and various protocols that mandate various kinds of StringPrep give you the feeling that such a mandate will lead to consistent results?

--Paul Hoffman

---

From: Yutaka OIWA <[email-masked]>
Date: Tue, 2 Jul 2013 11:52:31 +0900

2013/6/30 Julian Reschke <[email-masked]>:
> On 2013-06-30 16:28, [email-masked] wrote:

> For now, there's one open issue left: we currently do not say anything about
> Unicode normalization forms, but we probably should.

+2 for considering normalization: it is almost a MUST for interactive
(e.g. browser) use-cases.
Just as an instance, default Unicode encoding of
Japanese language differs between existing major implementations
(composed (Win) or decomposed (Mac) for hiragana/katakana).
This already has made a huge cumbersome on exchanged filenames.
Without normalization, Japanese users will not be able to reliably use
internationalized usernames in authentication.

At the same time, as Paul suggested, it has a too much overhead for
servers and non-interactive clients.  It is very unlikely and unwise
to be implemented on wget and curl, for example.
Also, server-side normalization is not compatible with any
other schemes which apply hashes and alike to usernames/passwords.

I want to find a good phrase, which "effectively" means that
normalization is a "to do" only for modern browsers but
"ignorable" for all other clients, and "not to do" for all servers.

One current solution is make it a SHOULD "to send normalized", without
specifying how.  Non-interactive clients may require configurations
to prepare strings beforehand, or just require an ASCII subset,
both of which is to be considered to meet the SHOULD requirement.
However, I still want more elegant solutions.

At the same time, PRECIS WG had a small mention on
string preparation (which includes Unicode normalizations and more)
on HTTP in Orlando,  and I am currently preparing a straw-man.

--
Yutaka OIWA, Ph.D.                 Leader, System Life-cycle Research Group
                               Research Institute for Secure Systems (RISEC)
     National Institute of Advanced Industrial Science and Technology (AIST)
                       Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5]

---

From: Yoav Nir <[email-masked]>
Date: Tue, 2 Jul 2013 07:59:14 +0000

Hi

For those of us not so well versed in I18N issues, what do you mean by normalization? 

Is it just consolidating look-alike code points, like the multiple hyphens that exist in Unicode? 
Does it also involve removing Arabic and Hebrew points? (I think Paul raised this one) 
Does it involve removing diacritics?
Does it involve splitting combined characters (like U+00E6 into 'a' and 'e')?
Is there a standard we can point to and say "do this before comparing or hashing"?

Thanks

Yoav

---

From: Julian Reschke <[email-masked]>
Date: Tue, 02 Jul 2013 10:37:50 +0200

> Is there a standard we can point to and say "do this before comparing or hashing"?

Dunno what the others think, but I was focusing on NFC vs NFD 
(<http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms>), with 
a preference to recommend NFC.

That being said, it might be interesting to test what those UAs that 
currently *do* use UTF-8 do.

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Tue, 02 Jul 2013 18:03:34 +0900

> Is there a standard we can point to and say "do this before comparing or hashing"?

In the narrow sense, it involves one of the four forms (NFC, NFKC, NFD, 
NFKD) defined in UAX #15 (http://www.unicode.org/reports/tr15/). NFC/D 
deal with precomposed/decomposed issues for accents, although not with 
cases such as U+00E6, where the distinction between "ae" and "æ" should 
be clearly visible. A Hebrew example would be U+FB1D, HEBREW LETTER YOD 
WITH HIRIQ, which as an exception is decomposed both in NFC and NFD to 
U+05D9 U+05B4 (usually, NFC is precomposed).

The K variants also include other variant changes, sometimes some very 
visible ones. For example, they take １ (full-width) and ① to 1. Here, 
most hyphen variants are not included, although some are (e.g. U+207B, 
subscript hyphen).

Case folding/equivalence is separate from both of the above cases.

Removing diacritics is also separate, because in many cases, that would 
be besides the point.

Paul has already mentioned stringprep, which is the base for nameprep 
and saslprep (and maybe others that I'm not aware of). That work is 
currently followed up by the work of the IETF precis WG.

Regards,   Martin.

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 02 Jul 2013 11:58:20 +0200

>So what if we mandated normalization to happen on the client?

I see no benefit in the server receiving normalize("$user:$pass") over
receiving "$user:$pass" given that the server can and probably has to do
`normalize` on its own, e.g. during account creation or password change,
especially considering that `normalize` may vary over time. I also note
efforts like <http://tools.ietf.org/html/draft-ietf-precis-nickname-05>.
-- 
Björn Höhrmann · mailto:[email-masked] · http://bjoern.hoehrmann.de
Am Badedeich 7 · Telefon: +49(0)160/4415681 · http://www.bjoernsworld.de
25899 Dagebüll · PGP Pub. KeyID: 0xA4357E78 · http://www.websitedev.de/

---

From: Yoav Nir <[email-masked]>
Date: Tue, 2 Jul 2013 11:44:52 +0000

> efforts like <http://tools.ietf.org/html/draft-ietf-precis-nickname-05>.

I realize that this thread is specifically about basicauth. But this group is also considering other authentication schemes (digest, mutual-auth, scram). In all of those, you won't get the password in the clear, so you can't expect the server to perform the manipulations. 

The group can decide to mandate one behavior (entirely server-side) for basicauth, and something else for others (like maybe negotiate the kind of normalization to perform). But I'd rather we use the same kind of processing for all methods unless there's a good reason to do so. 

Yoav

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 02 Jul 2013 06:31:55 -0600

On 7/2/13 3:58 AM, Bjoern Hoehrmann wrote:
> I see no benefit in the server receiving normalize("$user:$pass")
> over receiving "$user:$pass" given that the server can and probably
> has to do `normalize` on its own, e.g. during account creation or
> password change, especially considering that `normalize` may vary
> over time. I also note efforts like
> <http://tools.ietf.org/html/draft-ietf-precis-nickname-05>.

See also the in-process revisions to SASLprep, which might be more
appropriate for this group:

http://datatracker.ietf.org/doc/draft-ietf-precis-saslprepbis/

In the XMPP community, we originally used stringprep and are now
transitioning to PRECIS. Also, we were not clear on exactly who is
responsible for normalization and string preparation. We're cleaning
that up in 6122bis (see Section 3):

https://datatracker.ietf.org/doc/draft-ietf-xmpp-6122bis/

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Michael Sweet <[email-masked]>
Date: Tue, 02 Jul 2013 08:54:14 -0400

RFC 5198?

On 2013-07-02, at 3:59 AM, Yoav Nir <[email-masked]> wrote:

> _______________________________________________
> http-auth mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/http-auth

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 02 Jul 2013 07:18:15 -0600

> For those of us not so well versed in I18N issues, what do you mean

Martin described it already.

You might find it useful to look at the slides I put together for a
two-hour tutorial on internationalization at an IETF meeting a few
years ago:

https://stpeter.im/files/i18n-into.pdf

Perhaps I need to offer that tutorial again? Or maybe turn it into a
small book?

> hyphens that exist in Unicode?

There is no complete solution to the problem of confusable characters.
Here is a fun example for you:

https://stpeter.im/journal/1420.html

However, normalization takes care of some (but not all) instances of
code points that look alike.

> Does it also involve removing Arabic and Hebrew points? (I think

There is no reason to *remove* characters from left-to-right scripts.
However, allowing them means you need to deal with interesting bidi
issues.

> Does it involve removing diacritics?

Not necessarily -- why can't my username be saintandré or whatever?
And *removing* diacritics might be problematic, if it implies that 'é'
would be transformed to 'e' (thus introducing the possibility of
additional false positives).

> Does it involve splitting combined characters (like U+00E6 into 'a'

Actually, there is no decomposition of U+00E6 æ into 'a' and 'e' in
Unicode. This might be counter-intuitive, but it's true.

> Is there a standard we can point to and say "do this before

I'd suggest looking carefully at SASLprep (RFC 4013) and its proposed
replacement:

If you have feedback on the latter, please post to the [email-masked]
list:

https://www.ietf.org/mailman/listinfo/precis

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Peter Saint-Andre <[email-masked]>
Date: Tue, 02 Jul 2013 07:21:23 -0600

On 7/2/13 7:18 AM, Peter Saint-Andre wrote:

> You might find it useful to look at the slides I put together for
> a two-hour tutorial on internationalization at an IETF meeting a
> few years ago:
> 

Oops, that is: https://stpeter.im/files/i18n-intro.pdf

Peter

- -- 
Peter Saint-Andre
https://stpeter.im/

---

From: Bjoern Hoehrmann <[email-masked]>
Date: Tue, 02 Jul 2013 15:52:14 +0200

* Yoav Nir wrote:
>The group can decide to mandate one behavior (entirely server-side) for 
>basicauth, and something else for others (like maybe negotiate the kind 
>of normalization to perform). But I'd rather we use the same kind of 
>processing for all methods unless there's a good reason to do so. 

The specification in question cannot really require server-side normali-
sation. It is entirely up to the server whether to authorize a client to
access its resources. The server might treat user names case-sensitively
or might reject passwords that are not in NFC form or do whatever else
it feels like. You cannot make a black box test that would demonstrate
non-compliance to such a requirement. This would rather note that use of
certain combinations of characters can result in unexpected failures and
various forms of normalisation and tolerances can be used to mitigate.

I think it would be out of scope of the document to do much more and it
would very likely delay and complicate deployment without a clear gain.

There is of course the option to make sure the draft defines that user
agents must treat certain unrecognized values for the charset parameter
or multiple `WWW-Authenticate` headers for the same Basic realm such
that servers have an option to opt into client-side normalization later,
if the group decides on some common options. So,

  WWW-Authenticate: Basic realm="example", charset="UTF-8/NFCv6+"

or

  WWW-Authenticate: Basic realm="example", charset="UTF-8", x="precis"
  WWW-Authenticate: Basic realm="example", charset="UTF-8"

for instance.

---

From: Yoav Nir <[email-masked]>
Date: Tue, 2 Jul 2013 14:02:31 +0000

On Jul 2, 2013, at 4:18 PM, Peter Saint-Andre <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 

Excellent resource. Yeah, you should.

> code points that look alike.

For our particular use case we need the results of normalization to be bit-for-bit equal, and to get the same result when typing the name or password in different browsers on PCs, Macs, all Linux distros, and all smartphones and tablets. As mentioned earlier in this thread, these are not always identical.

> However, allowing them means you need to deal with interesting bidi

There is. In French, André is correct while Andre is simply wrong. In Arabic and Hebrew, the diacritics are reading aids that were bolted on to an existing script. The text is fine with and without them, so I could use either יואב or יוֹאָב for my name, and both would be perfectly fine and in some sense equivalent. It's still fine to decide that these forms are not equivalent for our purposes, so if you used points for your username when you created it, you'll have to use points forever.

> additional false positives).

It all goes back to the keyboard the user uses to input username and password. I have no idea how to type the example above on Windows or on any phone. I just don't know the key combinations (yes, I could google it). So having your username be saintandré could leave you unable to log in from some platforms. The case could be made that we need stronger normalization than other use cases.