Subject: Re: Invalid UTF-8 strings

From: David Robinson <[email-masked]>
Date: Tue, 11 Dec 2001 13:08:25 -0600 (CST)

There are really two different cases of invalid UTF-8 strings:

	1) Strings that can represent no valid UTF-8 characters.
	   For example: illegal prefixes, incorrect number of bytes

	2) Strings that are valid UTF-8 syntax but cannot be
	   represented in the machines native mechanisms.

An argument can be made that case #1 is actually an XDR encode
bug and should be checked and errored at the RPC/XDR layer
and RPC_CANTDECODEARGS (11) should be returned.

For case #2 this is similar to a client trying to send a V3
unsigned time value with the MSB (> Y2039) set to a system that 
internally only supports a 32-bit signed value. I think it is
a mistake to specify that v4 will only handle 16 bit characters
and if larger values are desired that we rev the protocol. Instead
we should allow any valid UTF-8 string, but allow any server to
respond with a well defined error code that indicates that
it cannot handle that value, not just NFSERR_INVAL which implies
a bad value from the protocol perspective but one that indicates a
a value that this system doesn't support.

	-David

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 11 Dec 2001 12:23:22 -0800

>	   represented in the machines native mechanisms.

This is not an invalid UTF-8 string but it is an encoding of
something that is not a valid UCS string, at least it can't be
a valid one today, since today UCS characters only have 16 bits.

> and RPC_CANTDECODEARGS (11) should be returned.

That argument can certainly be made but I don't see any point
in going out of your way to give the client no indication
of where (in what op) the problem is.  Besides in this case,
the XDR says this is an opaque string, and the fact that it
is a UTF-8 string is part of the V4 protocol, not the XDR,
so it makes sense for the error to happen at that level. 

> and if larger values are desired that we rev the protocol.

But why is it a mistake?  Is there someone out there who needs
today, or will in the foreseeable future have a need for 32-bit
characters?  Are any such characters defined by anybody?  We have
many situations in which we are forced to deal with potential
interoperability problems.  They are a real pain for people to 
deal with but sometimes we have to do it.  Let's not go looking
for them, however.  If we follow your approach somebody will
be able to have filenames with 32-bit characters (the numeric value
of some counter say) without any conversion and then have it just
work on some servers and not on others, and then find out it
doesn't work on lots of servers.  If there was some need for this
facility, which many servers couldn't support, then we have to deal
with the issue.  But if there isn't, why create potential
interoperability issues. 

> Instead

I can live with that (NFS4ERR_NOTUCS2 perhaps) but what about the
clients?  Are they going to be forced to deal with the possibility 
of getting back characters that are longer than 16 bits?  I imagine
that some clients aren't set up to handle them very easily.

---

From: "Yoder, Alan" <[email-masked]>
Date: Tue, 11 Dec 2001 14:20:49 -0800

> characters?  Are any such characters defined by anybody?  

   What about the UCS4 32-bit Unicode standards
being pushed by various Oriental organizations?
A google of ucs4 reveals that the Sather programming
language even has native support for this data
type, so someone somewhere is taking it seriously.

;>

Alan

===============================================================
Alan G. Yoder                                    [email-masked]
Network Appliance, Inc.                           
Sunnyvale, CA                                      408-822-6919
===============================================================

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 12 Dec 2001 06:39:31 -0800

> type, so someone somewhere is taking it seriously.

Apparently.  So there's no way to hold the line at
16 bits.  Given that, I think David Robinson's approach
to this issue is the right one.