Subject: [http-auth] UTF-8, usernames, passwords

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Mon, 27 Oct 2014 10:12:10 -0600

Over in the PRECIS working group, we've been trying to harmonize the 
username and password definitions from "saslprepbis" (which is intended 
to apply to more than just SASL) with the updated spec for basic auth:

http://datatracker.ietf.org/doc/draft-ietf-precis-saslprepbis/
http://datatracker.ietf.org/doc/draft-ietf-httpauth-basicauth-update/

See for instance this thread:

http://www.ietf.org/mail-archive/web/precis/current/msg00705.html
http://www.ietf.org/mail-archive/web/precis/current/msg00704.html
http://www.ietf.org/mail-archive/web/precis/current/msg00861.html
http://www.ietf.org/mail-archive/web/precis/current/msg00869.html

Although we've gotten closer to harmony and I've been working to adjust 
draft-ietf-precis-saslprepbis as needed, we're not there yet.

USERNAMES

UTF-8: both saslprepbis and basicauth specify UTF-8 (for basicauth, when 
the new 'charset' parameter is used)

Unicode normalization: NFC

Width mapping: not specified in basicauth, whereas saslprepbis specifies 
that fullwidth and halfwidth characters MUST be mapped to their 
decomposition mappings (is this something that basicauth might specify?)

The major gap appears to be the allowable characters:

- saslprepbis limits username characters to those allowed by the PRECIS 
IdentifierClass, essentially letters and digits and ASCII7 characters, 
along with single space characters separating parts of a string.

- basicauth might (at least based on what I understand) allow a wider 
range of characters, such as multiple spaces in a row or even symbols 
outside the ASCII7 range (see discussion thread above for examples)

As to the space issue, we might be able to work around this by defining 
an application-layer construct in basicauth that reuses the "userpart" 
rule from saslprepbis.

However, for allowable characters if we want to allow basically any 
character in basicauth usernames then in PRECIS terms we'd need to make 
a profile of the FreeformClass, not the IdentifierClass. IMHO that would 
be less than ideal and likely violate the principle of least user 
astonishment ("why does this username work for the web but not email?").

PASSWORDS

Here saslprepbis and basicauth are closer together. In fact the only 
gaps might be:

- width mapping: saslprepbis says MUST NOT perform width mapping, 
whereas basicauth doesn't say anything about that

- spaces: in SASLprep and also saslprepbis, non-ASCII spaces are mapped 
to ASCII space, whereas basicauth doesn't say anything about that

For both usernames and passwords in basicauth, we might want to say 
something about bidi characters.

Is there a way that we can come to agreement on these issues? Do we 
perhaps need time at the next IETF meeting, or a conf call of some kind, 
to make progress?

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Yutaka OIWA <[email-masked]>
Date: Tue, 28 Oct 2014 09:52:54 +0900

Dear all,

As an author of another PRECIS draft for HTTP authentication
(http://tools.ietf.org/html/draft-oiwa-precis-httpauthprep-00),
I personally feel that limiting allowing characters to
IdentifierClass is feasible and reasonable.
It includes all "sensible IMO" characters for identifiers,
is binary compatible with all printable ASCII characters,
and is character-set compatible with ISO-8859-1 set.
It will not exclude any printable characters which is currently
officially allowed in the spec.

We should allow any number of "ASCII SPACE" U+0020
between printable characters for backward compatibility,
and should not forbid any printable ASCII characters
(except some schema-dependent "forbidden-by-protocol"
character like colon <:> in basic).
I strongly believe we should forbid any control characters.
In both Peter's and my proposals, SPACEs in the
beginning and the tail of the string is not allowed,
which can be a target to discuss.

I'm currently trying to seek for possible merger of
my proposal into saslprepbis, to reduce number of
profiles to implement.

-- 
Yutaka OIWA, Ph.D.
           Senior Researcher, Research Institute for Secure Systems (RISEC)
    National Institute of Advanced Industrial Science and Technology (AIST)
                      Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[440546B5] fp[7C9F 723A 7559 3246 229D  3139 8677 9BD2 4405 46B5]

2014-10-28 1:12 GMT+09:00 Peter Saint-Andre - &yet <[email-masked]>:
> username and password definitions from "saslprepbis" (which is intended to
> apply to more than just SASL) with the updated spec for basic auth:
>
> UTF-8: both saslprepbis and basicauth specify UTF-8 (for basicauth, when the
> new 'charset' parameter is used)
>
> IdentifierClass, essentially letters and digits and ASCII7 characters, along
> with single space characters separating parts of a string.
>
> - basicauth might (at least based on what I understand) allow a wider range
> of characters, such as multiple spaces in a row or even symbols outside the
> ASCII7 range (see discussion thread above for examples)
>
> As to the space issue, we might be able to work around this by defining an
> application-layer construct in basicauth that reuses the "userpart" rule
> from saslprepbis.
>
> character in basicauth usernames then in PRECIS terms we'd need to make a
> profile of the FreeformClass, not the IdentifierClass. IMHO that would be
> less than ideal and likely violate the principle of least user astonishment
> ("why does this username work for the web but not email?").
>
> PASSWORDS
>
> Here saslprepbis and basicauth are closer together. In fact the only gaps
> might be:
>
> - width mapping: saslprepbis says MUST NOT perform width mapping, whereas
> basicauth doesn't say anything about that
>
> - spaces: in SASLprep and also saslprepbis, non-ASCII spaces are mapped to
> ASCII space, whereas basicauth doesn't say anything about that
>
> Is there a way that we can come to agreement on these issues? Do we perhaps
> need time at the next IETF meeting, or a conf call of some kind, to make
> progress?
>
> Peter
>
> --
> Peter Saint-Andre
> https://andyet.com/
>
> _______________________________________________
> http-auth mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/http-auth

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Mon, 27 Oct 2014 19:04:10 -0600

> between printable characters for backward compatibility,

Actually that is allowed by the current rule in saslprepbis:

    username   = userpart [1*(1*SP userpart)]

> my proposal into saslprepbis, to reduce number of

Yes, that is a good goal.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Yutaka OIWA <[email-masked]>
Date: Tue, 28 Oct 2014 15:17:03 +0900

>   username   = userpart [1*(1*SP userpart)]

Yes.  I intended that two profile specs are consistent on this.

Sorry for possible confusions, and thank you for clarification.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Oct 2014 09:49:05 +0100

On 2014-10-27 17:12, Peter Saint-Andre - &yet wrote:
> ...

It's an interesting suggestion, but I just don't see how we can test it 
against reality -- right now people can put whatever they want into 
userids and passwords, and thus it's possible that these edge cases 
exist in practice.

I'd be ok with informative text that warns against certain character 
ranges (if we can do it quickly, are not blocked by the reference, and 
the rational actually makes sense :-).

Best regards, Julian

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 28 Oct 2014 09:24:33 -0600

> the rational actually makes sense :-).

I would like to avoid having two PRECIS profiles for usernames, one for 
HTTP usernames and one for non-HTTP usernames.

However, if the HTTPAUTH WG truly wants to say "anything goes" for 
usernames, then we're treating both HTTPAUTH usernames and HTTPAUTH 
passwords as we treat passwords in saslprepbis - i.e., as freeform 
strings. And if that's consistent with traditional usage for HTTPAUTH 
then so be it.

But in that case HTTPAUTH usernames aren't intended to be more 
restricted in any sense, they're not intended to be "safe" strings with 
a limited range of characters, they're not intended to reduce the 
possibility of user confusion caused by visually similar characters and 
characters that are more "expressive" (e.g., symbol characters), and 
IMHO they're not taking into account the lessons of i18n usability at all.

(We could possibly say that HTTPAUTH applications MAY use the PRECIS 
PasswordFreeformClass profile from saslprepbis for both HTTPAUTH 
usernames and HTTPAUTH passwords, but I leave that up to the HTTPAUTH WG.)

If the HTTPAUTH WG and the responsible AD and the IAB i18n directorate 
and the IETF community and the IESG are all fine with that approach, 
then I'm not going to raise objections, but I'm also not going to 
continue putting time and energy into finding a compromise or 
harmonizing saslprepbis usernames with HTTPAUTH "usernames", because 
despite having the same name these entities are fundamentally different 
from each other and it's silly to consider them the same in any way.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Oct 2014 16:28:41 +0100

> from each other and it's silly to consider them the same in any way.

Understood.

The main issue I have is that this WG seems to be very limited in it's 
capacity to review specs, discuss changes, and also doesn't seem to have 
any participation from HTTP client/server implementers.

(I'd love to be proved otherwise, btw)

With that in mind, I'm very very reluctant to add new normative 
requirements to something that most people consider "legacy code".

Best regards, Julian

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 28 Oct 2014 09:32:00 -0600

> requirements to something that most people consider "legacy code".

Understood as well. :-)

So, the most that I think we can do is to say "traditionally anything 
goes for HTTPAUTH usernames and passwords, but it you want to do 
something a bit more sensible from the perspective of 
internationalization then you might apply the username and password 
profiles from saslsprepbis" with merely an informational reference to 
the latter specification.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Julian Reschke <[email-masked]>
Date: Tue, 28 Oct 2014 16:35:10 +0100

> the latter specification.

Sounds reasonable to me.

Questions:

1) What is the schedule for this spec?

2) Who do you think the audience for this advice is? Not the User-Agent 
implementer, I guess, because she can't afford rejecting credentials 
that might be used in practice. HTTP server implementers might not have 
control over the userid/password system used. So how exactly would we 
phrase this?

Best regards, Julian

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 28 Oct 2014 09:51:38 -0600

> 1) What is the schedule for this spec?

Aside from the issue of coordination with the HTTPAUTH WG, I think it is 
ready for WGLC.

> control over the userid/password system used. So how exactly would we

Good point. It seems to be advice for operators, although operators 
can't put the advice into practice unless HTTP server implementers offer 
the feature.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Michael Sweet <[email-masked]>
Date: Tue, 28 Oct 2014 12:12:00 -0400

Julian,

> ...
> Understood.
> 
> The main issue I have is that this WG seems to be very limited in it's capacity to review specs, discuss changes, and also doesn't seem to have any participation from HTTP client/server implementers.
> 

Well, CUPS is a HTTP client/server implementation and I'm here... :)

> With that in mind, I'm very very reluctant to add new normative requirements to something that most people consider "legacy code".

While HTTP Basic authentication is still widely used for printing and other "embedded" solutions, the time for limiting the username strings was almost 25 years ago when it was first defined.  Any limits should be recommendations at this point.

_________________________________________________________
Michael Sweet, Senior Printing System Engineer, PWG Chair

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Tue, 28 Oct 2014 10:32:19 -0600

> Well, CUPS is a HTTP client/server implementation and I'm here... :)

That's good to know. :-)

> While HTTP Basic authentication is still widely used for printing and other "embedded" solutions, the time for limiting the username strings was almost 25 years ago when it was first defined.  Any…

Perhaps even only a suggestion, not a recommendation.

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Yutaka OIWA <[email-masked]>
Date: Wed, 29 Oct 2014 07:26:08 +0900

Dear Julian,

> 2) Who do you think the audience for this advice is?
> Not the User-Agent implementer, I guess, because
> she can't afford rejecting credentials that might be used in practice.

It may be not in a focus of httpauth WG (mainly PRECIS WG),
but I think IETF should finally convince User-Agent implementors
to handle it, because without that UTF-8 strings are very fragile for
interoperability.

Please also consider that PRECIS IdentifierClass will NOT
reject any printable strings which are currently officially
used without i18n extensions.

I understand the scope of Basic-auth-update which will
be compatible with existing implementations with
ad-hoc UTF-8 uses.  My personal guess is that
introduction of PRECIS will not be very harmful in reality:
as far as the spec does not enforce use of PRECIS profile
as MUST requirements, they will continue to send
forbidden characters (at least printable ones such as
dingbat figures) as is if they need, and they will continue to
work with current poor interoperability level.
PRECIS profile will add better interoperability between
supported clients and supported range of characters.

> HTTP server implementers might not have control over
> the userid/password system used. So how exactly would we phrase this?

Current httpauthprep have an implementation advices, and
if merged, I guess (I want) saslprepbis will have them, too.

My proposal is that only the clients with Unicode character
handling capabilities will do that, and servers SHOULD NOT
process it in any way (consider cases other than Basic auth).
Clients handling only octet-encoded credentials MAY
assume that these are already properly processed and
MAY omit any online processing.

2014-10-29 0:35 GMT+09:00 Julian Reschke <[email-masked]>:
> implementer, I guess, because she can't afford rejecting credentials that
> might be used in practice. HTTP server implementers might not have control
> over the userid/password system used. So how exactly would we phrase this?
>
> Best regards, Julian
>
>

---

From: Yutaka OIWA <[email-masked]>
Date: Wed, 29 Oct 2014 07:41:21 +0900

Dear Michael,

> While HTTP Basic authentication is still widely used for printing and
> other "embedded" solutions, the time for limiting the username strings
> was almost 25 years ago when it was first defined.
> Any limits should be recommendations at this point.

Specifically, PRECIS IdentifierClass will not restrict any printable strings
which are officially allowed in the current spec (ASCII or ISO-8859-1).
It will expand it to a sensible set of Unicode characters,
only when any new authentication scheme or extensions
(like Julian's for Basic auth) is deployed.
(We're explicitly rejecting some control characters
 like LF or CR, but I guess these are not real problems.)

So, it is the *first* definition for the handling of Unicode
characters in HTTP authentication, and it is the time to do it.
If we don't do it for right now,
UTF-8 interoperability will be very poor for next 25 years.

I am also carefully considering about existing implementations
which have ad-hoc handling of UTF-8 strings.
They are not aligned with existing specifications,
and they will be useful as it is, although
maybe it will not strictly aligned to new specifications,
if users force them to offense the new spec.
For this, good choices of normativeness level are important.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Oct 2014 16:51:04 +0100

> to handle it, because without that UTF-8 strings are very fragile for

I'm sorry but this really sounds like wishful thinking. Do you seriously 
believe that user agent implementors will start blocking certain 
characters when there's a slim chance that they might be used in 
practice already?

> used without i18n extensions.

The problem here is that due to the vagueness of RFC 2617, there 
currently is no limit to what can be used (well, maybe except control 
characters).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Oct 2014 16:55:15 +0100

> which are officially allowed in the current spec (ASCII or ISO-8859-1).

You are reading the current spec differently than many UA implementors.

> characters in HTTP authentication, and it is the time to do it.

It may be the first definition, but these characters are already in use 
in credentials, and we can't turn the clock back.

> For this, good choices of normativeness level are important.

This is something where the spec will have little impact on reality, no 
matter what amount of normativeness we put in. Explaining the problem 
space and giving recommendations seems like a good idea to me; 
everything else IMHO is pointless.

Best regards, Julian

---

From: Yutaka OIWA <[email-masked]>
Date: Thu, 30 Oct 2014 08:14:44 +0900

Dear Julian,

I'll separate some long answers to a few mails.

> matter what amount of normativeness we put in. Explaining the problem space
> and giving recommendations seems like a good idea to me; everything else
> IMHO is pointless.

For Basic and Digest, I agree "recommendation-level" is a good
choice, because these aim for "ad-hoc" compatibility
with working implementations without new extensions.
I think it's the correct level of "normativeness" for these.
If my word "normativeness" looks like limited to more stronger things,
I apologize for confusion.

I believe, at the same time, the "correct" thing to do
if we had had no historical restrictions, is a SHOULD level, and
I'll use it in SHOULD level for my own proposal, because it is
experimental and we can expect only new deployments without
existing encoded credentials.

---

From: Yutaka OIWA <[email-masked]>
Date: Thu, 30 Oct 2014 08:20:22 +0900

> It may be the first definition, but these characters are already in use in
> credentials, and we can't turn the clock back.

First, the IdentifierClass is still quite conservative in what is
actually allowed, so I am quite optimistic in believing that
the actually rejected credentials with a new rule is either
virtually nothing or (even if it is existing) quite rare.

I first proposed to PRECIS to use more broader set of
characters for HTTP authentications,
(although it was not correctly mentioned in the spec)
but I was once convinced during PRECIS WG discussions
to use the current set of allowed characters.
Peter should possibly have better examples of allowed and
disallowed strings than me.

Second, for Basic and Digest, we may use it only as a
definitions of good, safe characters and say something like
"for use of characters inside this set we can guarantee
interoperability between supported clients,
but use of characters outside of this set may cause some
unwanted interoperability problems by the nature of
Unicode character sets (but it will work as currently it is)."

---

From: Yutaka OIWA <[email-masked]>
Date: Thu, 30 Oct 2014 08:25:49 +0900

Dear Julian,

> You are reading the current spec differently than many UA implementors.

I understand (as I mentioned in many earlier mails) that
many UA implementors have already started to send
non-ISO-8859-1 characters in several ad-hoc ways, as
anyway users were inputting these characters to UI,
and they must do something on that.
I'm NOT ignoring that fact completely.
But it was and it is still working as ad-hoc, and
no interoperability is guaranteed at all.

As a Japanese EUC-environment user,
I really used un-interoperable situations for
for 15+ years (where most servers require SJIS inputs.)
Actually, Japanese were for-a-long-time having problems
on such things, so most of them are quite trained with
using ASCII-only identifiers (actually I guess we're
less requiring identifier i18n than European users).

Harmonizing to UTF-8 in both ad-hoc implementation levels
and well-defined spec level is really a good movement for me.
What I wanted to say is that the current situation is still
ad-hoc, and it is mostly working with a luck.
I'm really happy with basic-auth-update because only with it
such situations will have truely, guaranteed interoperability.

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Wed, 29 Oct 2014 17:27:22 -0600

> everything else IMHO is pointless.

Totally naïve question: can we tie better i18n behavior to use of the 
charset authentication parameter? Unmodified code won't advertise that 
param, but modified and updated code will. Most of the expectations 
around that param are already pretty heavy (UTF-8 and especially NFC), 
so adding a few more constraints might not be a significant burden.

But I am coming into this conversation from the outside, so I freely 
admit that I might be missing a whole lot of context...

Peter

-- 
Peter Saint-Andre
https://andyet.com/

---

From: Julian Reschke <[email-masked]>
Date: Thu, 30 Oct 2014 09:59:04 +0100

> so adding a few more constraints might not be a significant burden.

The expectations aren't heavy at all; these are implemented already.

The new constraints we speak of however are usually beyond the control 
of those who implement Basic auth (they usually do not implement 
userid/password storage but just interface to it).

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 31 Oct 2014 09:04:39 +0900

>> was almost 25 years ago when it was first defined.  Any limits should
>> be recommendations at this point.
>

I think there should be strong recommendations to tightly control the 
identifiers that are allowed, even if the protocol may allow more, and 
to carefully test any kind of processing (rejection, normalization,...) 
to make sure different parts of an infrastructure are in sync. 
Otherwise, things like 
https://labs.spotify.com/2013/06/18/creative-usernames/ will happen.

This should be part of a security section in every HTTP authentication 
related document that uses IDs or passwords or something similar.

Regards,   Martin.

---

From: Peter Saint-Andre - &yet <[email-masked]>
Date: Thu, 30 Oct 2014 20:05:46 -0600

> https://labs.spotify.com/2013/06/18/creative-usernames/ will happen.

Yes, that is one of my favorite examples. If only I had a way to 
communicate with that team to let them know that the XMPP nodeprep 
definition is not specified in an ancient I-D but in RFC 6122...

> related document that uses IDs or passwords or something similar.

Agreed.

Peter

-- 
Peter Saint-Andre
https://andyet.com/