Subject: [nfsv4] RE: charsets

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 7 Dec 2004 12:59:26 -0500

For people new to the thread (those not on [email-masked]),
the background here is that Bruce is wondering exactly how mandatory
the checks for valid UTF-8 on names received from the client
have to be.

J Bruce Fields wrote:
> On Tue, Dec 07, 2004 at 11:46:28AM -0500, Noveck, Dave wrote:
> > Section 11.4 discusses this but all it says is "should".
> > 
> > The other stuff in this chapter is expressed in terms of 
> > stringprep profiles and those might imply something stronger,
> > to one who was knowledgable in that area.
> 
> Yeah, that's where I got lost.
> 
> So I suppose this would be better asked on the nfsv4 ietf list, 

I added it to the cc list for this reply.

> but is *anyone* actually respecting that "should"?

If you include stuff under devlopment, then the answer is that I am, 
in my private workarea.  But in that code, I am not just enforcing
the restriction, I have put in substantive internationalization
support (see below).

> Every existing nfsv4 server implementation also permits either local
> access or access via other protocols, right?  And it doesn't look like
> rejecting non-utf8 filenames works well in that situation.

It does if the client sends UTF-8.

It sounds like you are supposing that clients will send names using
whatever encoding they are using for v2v3 and if that is the case they
won't take kindly to UTF-8 enforcment.

Now if you remove the enforcement then you have something that will 
work if the clients do something compatible with that but note that 
you have diverged from the v4 spec in more ways than simply saying 
"I'd rather not" to section 11.4.  If a client does a readdir then 
you would be returning non-UTF-8 strings for the filenames.  At that 
point, I'd have to say that you are simply ignoring what the spec says 
about strings being UTF-8 and the details of which modal auxiliaries 
are used where in the spec is beside the point.  The current v2v3 
situation is that the encoding of such strings is not dealt with 
as part of the protocol definition and that means you have to have 
clients agree on their encoding.  V4 chose not to go that way.

So my approach to this is that if a v2v3 filename was creating with 
a given encoding in effect then you return the UTF8 equivalent.  I'm
assuming that that is what clients expect and need.  Comments?

For us, this is analogous to the situation on NFSv2v3-CIFS interoperability.
If you create a file within NFSv2v3 and it is looked at from CIFS,
CIFS will see the Unicode equivalent of the name sent by NFS.  The
protocol does not define this mapping but the administrator does 
on volume-by-volume basis. 

One other point is that the contents of symlinks are also defined in
the spec as UTF-8.  This implies that the contents should be checked for 
UTF-8-ness and that clients are expecting that the contenst will in fact
be UTF-8 strings.  As this implies that you might have to do translation
between v2v3 and v4 views of symlink data, I'd like to verify this before 
I go off and code the relevant support. 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 7 Dec 2004 13:12:18 -0500

On Tue, Dec 07, 2004 at 12:59:26PM -0500, Noveck, Dave wrote:
> "I'd rather not" to section 11.4.  If a client does a readdir then you
> would be returning non-UTF-8 strings for the filenames.  At that
> are used where in the spec is beside the point.

Yeah, actually I think our real problem is that we're doing some UTF-8
checking on filenames we receive, while still happily sending out
non-UTF-8 in readdir responses.

> assuming that that is what clients expect and need.  Comments?

Realistically, our server can't expect to know how any given filename
was encoded.  Absent such knowledge, I assume there's no way we can
reliably perform a conversion to UTF-8.

--b.

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 7 Dec 2004 13:18:45 -0600

Folks, RFC3530 got one thing wrong when it comes to I18N: it left the
normalization form of filenames unspecified.  This is a serious problem
for anyone who wishes to implement full I18N support.

NFSv4.1 should fix that.  It'd be a simple fix.

As I see it _ideal_ NFSv4 servers should:

 - validate stringprep of new file/directory names, error if not
   properly stringprepped

 - only send back stringprepped file/directory names

The stringprep must include a normalization form (it's a bug that
RFC3530 didn't).  Without it NFSv4 servers must actually do the string
preparation and normalization to the server's choice of canonical
normalization to avoid duplicate but different names in the filesystem.

AIU it takes more code, CPU cycles and memory to do normalization than
it does to validate normalization.

Clients should be responsible for actually doing the stringprep.

The ideal is hard to meet though where there exist legacy filesystems --
what codeset was used to encode this filename?  who knows?  At best one
might know the most common locale used by users of that legacy
filesystem and convert to stringprepped UTF-8 accordingly; at worst
heuristics have to be applied to each name in the filesystem to detect
the codeset used to encode it.

> the checks for valid UTF-8 on names received from the client

IMO they have to be, but the spec needs to be fixed also.  How many
NFSv4 implementors actually implement I18N though?  I have no idea.

> won't take kindly to UTF-8 enforcment.

No, they won't.

> assuming that that is what clients expect and need.  Comments?

You need a way to determine the codeset used by the legacy clients.

> on volume-by-volume basis. 

That's as good way to 'determine' the codeset used by legacy clients as
you'll get without resorting to applying [probably necessarily
imperfect] heuristics to actual filenames.

Nico
--