Subject: Re: Invalid UTF-8 strings

From: Peter Åstrand <[email-masked]>
Date: Tue, 11 Dec 2001 16:14:44 +0100 (CET)

On Tue, 27 Nov 2001, Peter Åstrand wrote:

I'll answer a few of my own questions:

> 1. What actually is an invalid UTF-8 string? Is it any string with an
> invalid character, like 11000000 11000001 (0xC0 0xC1))?

I've been told:
10xxxxxx
110xxxxx 0xxxxxxx
110xxxxx 11xxxxxx
1110xxxx 0xxxxxxx
1110xxxx 11xxxxxx
1110xxxx 10xxxxxx 0xxxxxxx
1110xxxx 10xxxxxx 11xxxxxx
11110xxx 0xxxxxxx
11110xxx 11xxxxxx
11110xxx 10xxxxxx 0xxxxxxx
11110xxx 10xxxxxx 11xxxxxx
11110xxx 10xxxxxx 10xxxxxx 0xxxxxxx
11110xxx 10xxxxxx 10xxxxxx 11xxxxxx
...
1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 11xxxxxx

Plus, the stream should not end too early. 

> 2. Should operations check UTF-8 strings before returning them? E.g, 
> should READLINK verify the linkdata before returning? Or is it OK for a 

I suggest no. It would limit performance. However, I think checks should 
be done at creation time. This seems to be the intention of the NFS4 
protocol. However, some checks are missing, I think:

* CREATE should return NFS4ERR_INVAL if trying to create a symbolic link 
with linkdata not obeying UTF-8. Maybe the comment in the READLINK 
description needs to clarify this also. 

* SECINFO should return NFS4ERR_INVAL for non-UTF8 "name" argument. I
believe this is Issue66. (Btw, why does SECINFO have a name arg? To me, it
makes more sense have the (cfh) representing the file instead.)

* SETATTR, NVERIFY, VERIFY should also return NFS4ERR_INVAL on invalid 
attributes, for example attributes a non-UTF8 fattr4_owner. 

All these ops have NFS4ERR_INVAL as possible return type, but maybe the 
spec should clarify that this error should return for invalid attributes. 

/Peter

> But what about attributes, for example "owner". Should SETATTR accept
> setting the owner to an invalid UTF-8 string? Should GETATTR return
> invalid UTF-8 strings without errors, if such strings is stored on disk?
> 
> I've made a table, summarizing where UTF-8 is used. Have I understood the 
> policy for UTF-8 checking correctly? Should the "?" be "yes" or "no"?
> 
> ---
> 
> Data types derived from or containing utf8string:
> COMPOUND4args
> COMPOUND4res
> CB_COMPOUND4args
> CB_COMPOUND4res
> component4
> pathname4
> fs_location4
> fs_locations4
> READLINK4resok
> CREATE4args
> createtype4
> linktext4
> nfsace4
> fattr4_acl
> open_read_delegation4
> open_write_delegation4
> open_delegation4
> OPEN4resok
> fattr4_owner
> fattr4_owner_group
> fattr4_mimetype
> 
> Types used in attributes:
> fattr4_owner, fattr4_owner_group, fattr4_mimetype, fs_locations4, fattr4_acl
> 
> 
>          Check input			Check output
> ====================================================
> Procedures
> ==========
> COMPOUND:	no(tag)	                    no(tag)
> CB_COMPOUND:	no(tag)	                    no(tag)
> 			                    
> Operations		                    
> ==========		                    
> ACCESS		-	                    -
> CLOSE		-	                    -
> COMMIT		-	                    -
> CREATE		yes(objname, linkdata)      -
> DELEGPURGE	-	                    -
> DELEGRETURN	-	                    -
> GETATTR		-	                    ?(obj_attributes)
> GETFH		-	                    -
> LINK		yes(newname)		    -	
> LOCK		-	                    -
> LOCKT		-	                    -
> LOCKU		-	                    -
> LOOKUP		yes(objname)                -
> LOOKUPP		-	                    -
> NVERIFY		?(obj_attributes)	    -	
> OPEN		yes(open_claim.file)	    ?(delegation.*.permissions.who)
> OPENATTR	-			    -
> OPEN_CONFIRM	-			    -
> OPEN_DOWNGRADE	-			    -
> PUTFH		-			    -
> PUTPUBFH	-			    -
> PUTROOTFH	-			    -
> READ		-			    -
> READDIR		-			    ?(reply.entries.name, 
> 					      reply.entries.attrs)
> READLINK        -			    ?(linkdata) 
> REMOVE		yes(target)		    -
> RENAME		yes(oldname, newname)	    -
> RENEW		-			    -
> RESTOREFH	-			    -
> SAVEFH		-			    -
> SECINFO		yes(name)		    -
> SETATTR		yes(obj_attributes)	    -
> SETCLIENTID	-			    -
> SETCLIENTID_CONFIRM -			    -
> VERIFY		yes(obj_attributes)	    -
> WRITE		-			    -
> CB_GETATTR	-			    ?(obj_attributes)
> CB_RECALL	-			    -
> 
> 
> 

-- 
Peter Åstrand                Telephone: +46-13-21 46 00
Cendio Systems               E-mail: [email-masked]
Teknikringen 3
583 30 Linköping
Sweden

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 11 Dec 2001 07:48:10 -0800

This raises another interesting question.  What do we do about the
case in which you have a string that is a valid UTF-8 string but
which represents characters that don't fit in 16 bits?  These
would not be a problem for file systems which use native ASCII
names and simply store the UTF-8 but they definitely would be
a problem for file that store names as Unicode.

The spec now says that names are UTF8-encoded UCS strings but 
doesn't say much about what you do if the strings contain characters 
that are not valid UCS characters.  

The first case to consider is characters that fit in 16 bits
but have no valid assignments in UCS.  I think the spec should 
explicitly say that that these are valid and servers should not 
reject them.

The more interesting case concerns characters that are beyond the 
16-bit range.  The spec talks about how Unicode may not be big enough
(in the future), leaving it unclear about what servers and clients
are supposed to do now.

My preference for dealing with this is for the spec to say that for
this version of nfs-v4, UTF-8 should be used to encode 16-bit
characters and that anything else is invalid (NFS4ERR_INVAL).  It
would be still be noted that UTF-8 allows encoding of larger
characters and that if any such characters are added to UCS, the
issue can be addressed in a minor version.  Otherwise, we have a
lot of complicated interoperability issues to consider and deal with
when the event which may trigger them may in fact never happen.

-----Original Message-----
From: Peter Åstrand [mailto:[email-masked]]
Sent: Tuesday, December 11, 2001 10:15 AM
To: [email-masked]
Subject: Re: Invalid UTF-8 strings

I've been told:
10xxxxxx
110xxxxx 0xxxxxxx
110xxxxx 11xxxxxx
1110xxxx 0xxxxxxx
1110xxxx 11xxxxxx

/Peter

---

From: Erik Seaberg <[email-masked]>
Date: 11 Dec 2001 12:23:36 -0800

Dave Noveck <[email-masked]> writes:

> 16-bit range.  The spec talks about how Unicode may not be big
> enough (in the future), leaving it unclear about what servers and
> clients are supposed to do now.

Shouldn't filesystems that intend to encode Unicode filenames in 16
bits be using UTF-16 surrogate pairs (RFC 2781) to represent
codepoints between U+010000 and U+10FFFF?  I think ISO and unicode.org
have begun assigning codepoints outside the basic multilingual plane
to dead or invented languages.

On invalid encodings, a UTF-8 reader should also object to encodings
that are longer than necessary for a codepoint.  ISTR reading about
security flaws exploitable by encoding special US-ASCII substrings
(like "/" and "..") in multibyte encodings so that 8-bit pattern
matchers don't find them.

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 11 Dec 2001 12:48:47 -0800

> to dead or invented languages.

If they've actually started to do this, then I suppose we'll have 
to allow characters that don't fit in 16 bits.  Filenames in Cretan 
Linear B anyone? 

> matchers don't find them.

I don't understand the problem here.  Can you give an example of
an encoding that would be troublesome?

---

From: Erik Seaberg <[email-masked]>
Date: 11 Dec 2001 13:23:12 -0800

Dave Noveck <[email-masked]> writes:

> Filenames in Cretan Linear B anyone?

There seem to be assignments for Old Italic, Gothic, Deseret, and
music and math symbols.  And you never know, the Lord of the Rings
movie could spur a revival of Tengwar....

> I don't understand the problem [with overlong UTF-8 encodings].  Can
> you give an example of an encoding that would be troublesome?

A simple-minded implementation (that doesn't rely on the kernel for
authorization) might be configured to allow requests like
"/home/erk/foo" but reject requests like "/home/erk/../../etc/passwd".
If your request uses "\xC0\xAE\xC0\xAE" or something, anyone who
screens filenames before the final UTF-8 to UTF-16 conversion won't
see the ".." that's about to be passed on to the kernel.  I think a
few Web servers have been exploitable this way.

---

From: RJ Atkinson <[email-masked]>
Date: Mon, 31 Dec 2001 12:56:59 -0500

At 10:33 31/12/01, Peter Åstrand wrote:
>In my opinion, the protocol spec. should be changed to use the Unicode
>definition of UTF-8. This would limit NFS4 to codepoints between U+000000
>and U+10FFFF (with surrogate pairs, ~20 bits), but I think thats enough.

That proposal would preclude certain languages from being fully supported
that are currently fully supported, so is a non-starter, though I am sure 
the proposal works fine for Swedish.  The IETF practice is always to specify
ISO-10646 and ought not be changed peculiarly just for NFS.  We need to
make sure that the specs work for as many languages as possible, rather
than just for the Indo-European languages.  Using UTF-8 already biases
the playing field in a manner hostile to folks using ideogrammatic
characters.

Ran
[email-masked]

---

From: Erik Seaberg <[email-masked]>
Date: 31 Dec 2001 12:23:40 -0800

Peter Åstrand <[email-masked]> writes:

> There exists UTF-8 sequences that are impossible to represent even
> with surrogate pairs.

Can a NFS4 server impose local filesystem rules against client
requests?  If so, NFS4 over a UTF-16 filesystem simply never allows
codepoints beyond U+10FFFF in filenames, just as over a Unix
filesystem it never allows "/" or "\0" in filenames.  RFC 3010 stated
that ".." has no special semantics, but that seems to forbid clients
from using it in place of LOOKUPP, not oblige servers to map each and
every UTF-8 string to some locally legal filename.

> Should a Unix-kernel-client pass the octets given in open()
> unmodified over the wire, to the server?

I suppose the only other option is a libc wrapper that runs the
filename through mbstowcs() in the current locale and then wcstombs()
in a UTF-8 locale, just as Win32's CreateFileW() will have to convert
from UTF-16.  Do ANSI or POSIX have anything to say about I18N of
open() or fopen(), or is the filename arg just assumed to be in some
unknown single-byte encoding?

---

From: Erik Seaberg <[email-masked]>
Date: 31 Dec 2001 12:42:14 -0800

> At 10:33 31/12/01, Peter =C5strand wrote:

> that are currently fully supported, so is a non-starter, though I am sure

Didn't ISO and Unicode agree not to assign codepoints beyond U+10FFFF,
so that UTF-16 apps don't break?
<URL: http://www.unicode.org/unicode/faq/utf_bom.html#9 > seems to say so.

---

From: Peter Åstrand <[email-masked]>
Date: Mon, 31 Dec 2001 16:33:47 +0100 (CET)

> codepoints between U+010000 and U+10FFFF?  

Yes, but this does not solve the problem: There exists UTF-8 sequences
that are impossible to represent even with surrogate pairs. The four-octet
form is defined like this:

11110uuu 10uuzzzz 10yyyyyy 10xxxxxx

uuuuu must be 0uuuu or 10000; everything else is impossible to represent
with Unicode even with surrogate pairs. One example of such a UFT-8 string
is:

11110100 10100011 10100011 10100011 = F4 A3 A3 A3

This sequence is, as far as I understand, valid UTF-8 according to
ISO10646 but not according to Unicode, since ISO10646 and Unicode defines
different "flavors" of UTF-8. In addition to the requriment of uuuuu =
{0uuuu, 10000}, Unicode only allows sequences with 1-4 octets.

The NFS4 protocol specification explicitly says NFS4 should use the 
ISO10646 flavor: "The UTF-8 encoding of the UCS as defined by 
[ISO10646]...". This means problems for systems which uses Unicode file 
names. 

(Unicode provides sample code for checking Unicode-UTF-8 validity, se 
isLegalUTF8() in 
ftp://ftp.unicode.org/Public/PROGRAMS/CVTUTF/ConvertUTF.c)

Chapter 11.2 in the spec, Overview of Universal Character Set Standards,
seems a bit un-updated. It should probably mention surrogate pairs, UTF-32
etc.

Btw, It would be nice with some notes about implementing the UTF-8
support. For example, I wonder: Should a Unix-kernel-client pass the
octets given in open() unmodified over the wire, to the server? Eq.,
should the open() system call use an UTF-8 interface for NFS4 filesystems?
I believe so, but what should the kernel do if the user is trying to 
create a file called "ô£££", which happens to be the invalid Unicode-UTF-8 
sequence F4 A3 A3 A3? Should the client verify the UTF-8 seq. locally, 
before trying to send it to the server? But what error should open() 
return? EFAULT, EACCES or something else?

---

From: RJ Atkinson <[email-masked]>
Date: Mon, 31 Dec 2001 17:37:26 -0500

At 15:42 31/12/01, Erik Seaberg wrote:
>Didn't ISO and Unicode agree not to assign codepoints beyond U+10FFFF...

        Much though I wish it were not true, UNICODE and ISO-10646 aren't
identical (yet).  They are very close, but not (yet) quite the same
thing.  Convergence has been happening for a while, but seems to not
yet be complete.

        IETF practice is to specify ISO-10646 (and from the sidelines
encourage the UNICODE and ISO-10646 specs to converge to be identical
and fully complete linguistically).

Ran