Subject: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

From: "Noveck, David" <[email-masked]>
Date: Mon, 6 Jan 2014 06:53:10 -0500

David Black and I have been looking at the internationalization-related implications of how the current section 5.10 deals with case-insensitivity and non-case preserving behavior.  It seems that the section needs some major redrafting.  If you think it can get by with minor clarification and text editing, consider GREEK-SMALL-LETTER-FINAL-SIGMA, LATIN-CAPITAL-LETTER-I-WITH-DOT-ABOVE, LATIN-SMALL-LETTER-DOTLESS-I, OHM-SIGN, ANGSTROM-SIGN.

In any case, there is an issue that arises from the idea of rfc3530bis's approach to internationalization being that it is based on current inter-operating implementations.  The fact that this case-related stuff is essentially part of the internationalization stuff (despite being in a different chapter) means that we have to make some reasonable effort  to find out whether such implementations exist, and how they behave if so.

So I'd like the group's responses to the following questions:

1.       Do you know of NFSv4 servers (or server/filesystem pairs) that implement non-case-preserving behavior?  If so, does it up-case or down-case?  Also, does it deal with internationalization-related issues at all.

2.       Do you know of NFSv4 servers (or server/filesystem pairs) that implement case-insensitive lookups?

3.       Do you have clients that have actually interacted with either of these?

For those who either know the internals of such a server or are able to test how it reacts currently:

1.       Does it treat GREEK-SMALL-LETTER-FINAL-SIGMA as having an upper-case equivalent of GREEK-CAPITAL-LETTER-SIGMA?

2.       If so, and it is a non-case-preserving-implementation  which down-cases, what is the lower-case equivalent of  GREEK-CAPITAL-LETTER-SIGMA?  Is it context-dependent?

3.       How does it deal with the issue of dotted vs. dotless I in Turkish?  Is this configurable?

4.       How are OHM-SIGN and ANGSTROM-SIGN dealt with?   Are they treated as having no lower-case equivalents, as suggested by the current text (in the -29)?  Are they matched up with GREEK-SMALL-LETTER-OMEGA and LATIN-SMALL-LETTER-A-WITH-RING-ABOVE, as Unicode suggests?  Or something else?

---

From: "McDonald, Alex" <[email-masked]>
Date: Mon, 6 Jan 2014 15:20:52 +0000

The issues are dealt with here:  http://www.unicode.org/faq/casemap_charprop.html. Mappings (or more accurately, foldings for comparison) are from uppercase to lowercase.

*         Do you know of NFSv4 servers (or server/filesystem pairs) that implement non-case-preserving behavior?  If so, does it up-case or down-case?  Also, does it deal with internationalization-related issues at all.

"Case folding in Unicode is based on the lowercase mapping." Any server that folds to up-case would not be Unicode compliant.

*         Does it treat GREEK-SMALL-LETTER-FINAL-SIGMA as having an upper-case equivalent of GREEK-CAPITAL-LETTER-SIGMA?

That is an unnecessary and invalid treatment given mandatory case folding to lower case.

*         If so, and it is a non-case-preserving-implementation  which down-cases, what is the lower-case equivalent of  GREEK-CAPITAL-LETTER-SIGMA?  Is it context-dependent?

It should be based on ftp://ftp.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt

# C: common case folding, common mappings shared by both simple and full mappings.

03C2; C; 03C3; # GREEK SMALL LETTER FINAL SIGMA

For special casings in ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt, there's the note:

# Note: the following cases are not included, since they would case-fold in lowercasing

# 03C3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK SMALL LETTER SIGMA
# 03C2; 03C3; 03A3; 03A3; Not_Final_Sigma; # GREEK SMALL LETTER FINAL SIGMA

*         How does it deal with the issue of dotted vs. dotless I in Turkish?  Is this configurable?

>From ftp://ftp.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt

# The status field is:

# C: common case folding, common mappings shared by both simple and full mappings.

# F: full case folding, mappings that cause strings to grow in length. Multiple characters are separated by spaces.

# S: simple case folding, mappings to single characters where different from F.

# T: special case for uppercase I and dotted uppercase I

#    - For non-Turkic languages, this mapping is normally not used.

#    - For Turkic languages (tr, az), this mapping can be used instead of the normal mapping for these characters.

#      Note that the Turkic mappings do not maintain canonical equivalence without additional processing.

#      See the discussions of case mapping in the Unicode Standard for more information.

#

# Usage:

#  A. To do a simple case folding, use the mappings with status C + S.

#  B. To do a full case folding, use the mappings with status C + F.

#

#    The mappings with status T can be used or omitted depending on the desired case-folding

#    behavior. (The default option is to exclude them.)

#

I would recommend that the option be to specifically and explicitly ignore T status field foldings.

I don't understand the last point with reference to OHM-SIGN and ANGSTROM-SIGN.

Alex McDonald
Standards & Industry Associations Group
CTO Office
NetApp
+44 7795 046686 Mobile Phone
[email-masked]<mailto:[email-masked]>
twitter: @alextangent<http://twitter.com/alextangent>
Follow us: [Description: Description: Facebook] <http://www.facebook.com/NetApp?REF_SOURCE=ems-facebook> [Description: Description: Twitter] <http://twitter.com/#netapp?REF_SOURCE=ems-twitter> [Description: Description: Linked In] <http://www.linkedin.com/groups/NetApp-111681/about?REF_SOURCE=ems-linkedin> [Description: Description: YouTube] <http://www.youtube.com/user/NetAppTV?REF_SOURCE=ems-youtube> [Description: Description: Slideshare] <http://www.slideshare.net/NetApp?REF_SOURCE=ems-slideshare> [Description: Description: Community] <https://communities.netapp.com/welcome?REF_SOURCE=ems-cty>

From: nfsv4 [mailto:[email-masked]] On Behalf Of Noveck, David
Sent: 06 January 2014 11:53
To: nfsv4 list ([email-masked])
Subject: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

---

From: "Noveck, David" <[email-masked]>
Date: Mon, 6 Jan 2014 11:24:14 -0500

Thanks for your input, but I'm trying to find out the behavior of existing implementations.  In that context, it would help to point out behavior that you know exists, that is invalid or otherwise Unicode non-compliant.  I'm aware that many possibly implementations are not Unicode compliant but, I'd like to be able to triage those into known-to-exist, known-not-to-exist, and not-known-to-exist.

From: nfsv4 [mailto:[email-masked]] On Behalf Of McDonald, Alex
Sent: Monday, January 06, 2014 10:21 AM
To: Noveck, David; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

# The status field is:

#

# Usage:

#

#

Alex McDonald
Follow us:  <http://www.facebook.com/NetApp?REF_SOURCE=ems-facebook>  <http://twitter.com/#netapp?REF_SOURCE=ems-twitter>  <http://www.linkedin.com/groups/NetApp-111681/about?REF_SOURCE=ems-linkedin>  <http://www.youtube.com/user/NetAppTV?REF_SOURCE=ems-youtube>  <http://www.slideshare.net/NetApp?REF_SOURCE=ems-slideshare>  <https://communities.netapp.com/welcome?REF_SOURCE=ems-cty>

From: nfsv4 [mailto:[email-masked]] On Behalf Of Noveck, David
Sent: 06 January 2014 11:53
To: nfsv4 list ([email-masked]<mailto:[email-masked]>)
Subject: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

---

From: "McDonald, Alex" <[email-masked]>
Date: Mon, 6 Jan 2014 17:36:43 +0000

I realize that, but was wanting (and perhaps failed) to point out that since the internationalization section needs some major redrafting, it will be better to draft something that (a) is standardizable with relative ease in less than 600 pages and (b) doesn't require real life interoperability failures before we realize it doesn't work.

Hardball version; anything that doesn't conform to Unicode folding rules will have to change (with the usual restrictions to ensure practicality, such as Turkish code points, and perhaps others where the rules allow - who needs Klingon?) This should be the call for that change and a discussion based on that standard.

Accounting in the bis for those that don't follow the Unicode case folding rules won't help, since there is no alternative to them (or at least, no proven alternative) if you want interoperability. Are the current issues not evidence of that?

Alex McDonald

From: Noveck, David [mailto:[email-masked]]
Sent: 06 January 2014 16:24
To: McDonald, Alex; nfsv4 list ([email-masked])
Subject: RE: Internationalization and rfc3530bis-29's section 5.10

From: nfsv4 [mailto:[email-masked]] On Behalf Of McDonald, Alex
Sent: Monday, January 06, 2014 10:21 AM
To: Noveck, David; nfsv4 list ([email-masked]<mailto:[email-masked]>)
Subject: Re: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

# The status field is:

#

# Usage:

#

#

Alex McDonald

---

From: "Black, David" <[email-masked]>
Date: Mon, 6 Jan 2014 12:49:32 -0500

> I realize that, but was wanting (and perhaps failed) to point out that since the internationalization section needs some major redrafting,
> it will be better to draft something that (a) is standardizable with relative ease in less than 600 pages and (b) doesn't require real life
> interoperability failures before we realize it doesn't work.

That's is a fine thing to do for NFSv4.2, and could be done via use of the work in the précis WG that gets all of this right wrt Unicode.  OTOH, this draft (rfc3530bis) is an update to NFSv4.0 that needs to emphasize compatibility with what is deployed in the way of NFSv4.0 implementations.

> Hardball version; anything that doesn't conform to Unicode folding rules will have to change (with the usual restrictions to ensure practicality,
> such as Turkish code points, and perhaps others where the rules allow - who needs Klingon?) This should be the call for that change and a
> discussion based on that standard.

That's fine for NFSv4.2, but non-starter for NFSv4.0 due to deployed implementations.  Writing a new version of the NFSv4.0 spec to which none of the existing implementations comply is not a good idea, IMHO - that's what v4.2 is for ...

Thanks,
--David

From: nfsv4 [mailto:[email-masked]] On Behalf Of McDonald, Alex
Sent: Monday, January 06, 2014 12:37 PM
To: Noveck, David; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

Alex McDonald

From: Noveck, David [mailto:[email-masked]]
Sent: 06 January 2014 16:24
To: McDonald, Alex; nfsv4 list ([email-masked]<mailto:[email-masked]>)
Subject: RE: Internationalization and rfc3530bis-29's section 5.10

·         Do you know of NFSv4 servers (or server/filesystem pairs) that implement non-case-preserving behavior?  If so, does it up-case or down-case?  Also, does it deal with internationalization-related issues at all.

·         Does it treat GREEK-SMALL-LETTER-FINAL-SIGMA as having an upper-case equivalent of GREEK-CAPITAL-LETTER-SIGMA?

·         If so, and it is a non-case-preserving-implementation  which down-cases, what is the lower-case equivalent of  GREEK-CAPITAL-LETTER-SIGMA?  Is it context-dependent?

·         How does it deal with the issue of dotted vs. dotless I in Turkish?  Is this configurable?

# The status field is:

#

# Usage:

#

#

Alex McDonald

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 7 Jan 2014 17:16:45 +0000

On Jan 6, 2014, at 9:49 AM, Black, David <[email-masked]<mailto:[email-masked]>> wrote:

> it will be better to draft something that (a) is standardizable with relative ease in less than 600 pages and (b) doesn’t require real life
> interoperability failures before we realize it doesn’t work.

That’s is a fine thing to do for NFSv4.2,

That is a fine thing to do in a stand alone document that does not tie a cement block around the neck of NFSv4.2.

We agreed after the IESG sent 3530-bis back that we would tackle documenting deployed implementations in the 3530-bis framework.

And in line with our new desire to not wrap everything in with the minor versioning, we would encourage members to pick up the good fight
in a new document that all of the minor versions could be adapted towards.

and could be done via use of the work in the précis WG that gets all of this right wrt Unicode.  OTOH, this draft (rfc3530bis) is an update to NFSv4.0 that needs to emphasize compatibility with what is deployed in the way of NFSv4.0 implementations.

> Hardball version; anything that doesn’t conform to Unicode folding rules will have to change (with the usual restrictions to ensure practicality,
> such as Turkish code points, and perhaps others where the rules allow – who needs Klingon?) This should be the call for that change and a

That’s fine for NFSv4.2, but non-starter for NFSv4.0 due to deployed implementations.  Writing a new version of the NFSv4.0 spec to which none of the existing implementations comply is not a good idea, IMHO - that’s what v4.2 is for ...

Thanks,
--David

I realize that, but was wanting (and perhaps failed) to point out that since the internationalization section needs some major redrafting, it will be better to draft something that (a) is standardizable with relative ease in less than 600 pages and (b) doesn’t require real life interoperability failures before we realize it doesn’t work.

Hardball version; anything that doesn’t conform to Unicode folding rules will have to change (with the usual restrictions to ensure practicality, such as Turkish code points, and perhaps others where the rules allow – who needs Klingon?) This should be the call for that change and a discussion based on that standard.

Accounting in the bis for those that don’t follow the Unicode case folding rules won’t help, since there is no alternative to them (or at least, no proven alternative) if you want interoperability. Are the current issues not evidence of that?

Alex McDonald
Follow us: <http://www.facebook.com/NetApp?REF_SOURCE=ems-facebook><http://twitter.com/#netapp?REF_SOURCE=ems-twitter><http://www.linkedin.com/groups/NetApp-111681/about?REF_SOURCE=ems-linkedin><http://www.youtube.com/user/NetAppTV?REF_SOURCE=ems-youtube><http://www.slideshare.net/NetApp?REF_SOURCE=ems-slideshare><https://communities.netapp.com/welcome?REF_SOURCE=ems-cty>

Thanks for your input, but I’m trying to find out the behavior of existing implementations.  In that context, it would help to point out behavior that you know exists, that is invalid or otherwise Unicode non-compliant.  I’m aware that many possibly implementations are not Unicode compliant but, I’d like to be able to triage those into known-to-exist, known-not-to-exist, and not-known-to-exist.

•         Do you know of NFSv4 servers (or server/filesystem pairs) that implement non-case-preserving behavior?  If so, does it up-case or down-case?  Also, does it deal with internationalization-related issues at all.

“Case folding in Unicode is based on the lowercase mapping.” Any server that folds to up-case would not be Unicode compliant.

•         Does it treat GREEK-SMALL-LETTER-FINAL-SIGMA as having an upper-case equivalent of GREEK-CAPITAL-LETTER-SIGMA?

•         If so, and it is a non-case-preserving-implementation  which down-cases, what is the lower-case equivalent of  GREEK-CAPITAL-LETTER-SIGMA?  Is it context-dependent?

For special casings in ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt, there’s the note:

•         How does it deal with the issue of dotted vs. dotless I in Turkish?  Is this configurable?

# The status field is:

#

# Usage:

#

#

I don’t understand the last point with reference to OHM-SIGN and ANGSTROM-SIGN.

Alex McDonald
Follow us: <http://www.facebook.com/NetApp?REF_SOURCE=ems-facebook><http://twitter.com/#netapp?REF_SOURCE=ems-twitter><http://www.linkedin.com/groups/NetApp-111681/about?REF_SOURCE=ems-linkedin><http://www.youtube.com/user/NetAppTV?REF_SOURCE=ems-youtube><http://www.slideshare.net/NetApp?REF_SOURCE=ems-slideshare><https://communities.netapp.com/welcome?REF_SOURCE=ems-cty>

In any case, there is an issue that arises from the idea of rfc3530bis’s approach to internationalization being that it is based on current inter-operating implementations.  The fact that this case-related stuff is essentially part of the internationalization stuff (despite being in a different chapter) means that we have to make some reasonable effort  to find out whether such implementations exist, and how they behave if so.

So I’d like the group’s responses to the following questions:
1.       Do you know of NFSv4 servers (or server/filesystem pairs) that implement non-case-preserving behavior?  If so, does it up-case or down-case?  Also, does it deal with internationalization-related issues at all.
2.       Do you know of NFSv4 servers (or server/filesystem pairs) that implement case-insensitive lookups?
3.       Do you have clients that have actually interacted with either of these?

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 7 Jan 2014 19:55:15 +0000

I agree with Tom's assessment.  If we need/want something else for this particular topic, let's see a separate I-D covering the details at it applies to "future versions" to focus the details and energy and then determine where/when it best fits.

From: nfsv4 [mailto:[email-masked]] On Behalf Of Haynes, Tom
Sent: Tuesday, January 7, 2014 9:17 AM
To: Black, David
Cc: McDonald, Alex; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

> interoperability failures before we realize it doesn't work.

That's is a fine thing to do for NFSv4.2,

Thanks,
--David

Alex McDonald

# The status field is:

#

# Usage:

#

#

Alex McDonald

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jan 2014 17:20:03 -0600

If an existing filesystem is case-insensitive, ...

....and the NFSv4 fileserver is layered such that it cannot change the
undelying filesystem's case-folding...

...how relevant can any case mappings we specify be?

The NFSv4 server could fold case itself, knowing that the filesystem
does too.  But if the filesystem can be accessed through non-NFSv4
paths... then bad.

IMO this is *exactly* like the normalization case: we can't quite
specify a required behavior.  The behavior belongs squarely in a
component that neither the NFSv4 client nor the NFSv4 server can
safely do anything about.

Why aren't we saying this in the new RFC?  Fear of the IESG?

The more I think about it, the more it seems to me that normalization
and case behavior are simply out of scope for NFSv4 *the protocol*,
and that the RFC should say exactly that.  It would be fine to have an
implementation/interop. advice section describing *filesystem*
behaviors that exist and/or that are useful.

Nico
--

---

From: "Black, David" <[email-masked]>
Date: Tue, 7 Jan 2014 19:06:50 -0500

Ok, Nico, you're the second person who's responded to Dave's question about 
what existing NFSv4.0 implementations do by offering advice on how to rewrite
the standard.  Not only is that not what Dave asked for, but that work's
already in progress, and I expect that the result will look a lot like what
you and Alex would like to see ... but it's important to understand what
deployed implementations actually do, as this is about the rfc3530bis
draft, which is NFSv4.0 [yes, dot-zero!].

So, does anyone know what actual deployed NFSv4.0 implementations actually do???

Thanks,
--David

> -----Original Message-----
> From: nfsv4 [mailto:[email-masked]] On Behalf Of Nico Williams
> Sent: Tuesday, January 07, 2014 6:20 PM
> To: Noveck, David
> Cc: nfsv4 list ([email-masked])
> [email-masked]

---

From: Nico Williams <[email-masked]>
Date: Tue, 7 Jan 2014 19:22:09 -0600

> draft, which is NFSv4.0 [yes, dot-zero!].

I did answer the question, and I answer it more in more detail below.

My answer is that the question is meaningless (see below).  NFSv4
should say neither require nor recommend any specific behavior as
NFSv4 can't really tell filesystems what to do here.

> So, does anyone know what actual deployed NFSv4.0 implementations actually do???

They leave it to the filesystem, just like every nearly[*] other
remote filesystem protocol implementation does.  Next question!

;)

You must have meant "does anyone know what actual case-insensitive
filesystems used via NFSv4 do?", which is the same as "does anyone
know what actual case-insensitive filesystems?".  So let's address
that too.

You'd have to do a survey of filesystems, and that's harder than a
survey of NFSv4 server implementations.

The results may well be all over the place.  IIUC for FAT32 the answer
depends on the code page the filesystem was built with (but FAT32 can
probably be reasonably left out of any such survey survey).

Even then, if this is supposed to factor out, why ask that question of
filesystems?  Wouldn't it be more useful to ask what all sorts of
filesystem and non-filesystem things do with
GREEK-SMALL-LETTER-FINAL-SIGMA and friends?  For example, we can look
at IDNA and other Internet standards.

We could start a survey by looking here:
http://en.wikipedia.org/wiki/Comparison_of_file_systems as there are
some useful links there, but no detailed case mapping description on
that page.

Nico

[*] For *other* protocols, such as SMB, or in NFSv4 when trying to
present a case-sensitive filesystem as case-insensitive, server
implementations may implement case insensitivity at the fileserver
layer when used to share a case-sensitive filesystem.  The reverse is
not generally feasible, at least on general-purpose systems.

---

From: Mike Kupfer <[email-masked]>
Date: Tue, 07 Jan 2014 18:01:28 -0800

> So I'd like the group's responses to the following questions:
[...]

I'll do a little research to see what Solaris does with your second set
of questions, but it may take me a several days to complete it.

cheers,
mike

---

From: "McDonald, Alex" <[email-masked]>
Date: Wed, 8 Jan 2014 16:27:21 +0000

> > Ok, Nico, you're the second person who's responded to Dave's question
> > about what existing NFSv4.0 implementations do by offering advice on
> > how to rewrite the standard.  Not only is that not what Dave asked
> > for, but that work's already in progress, and I expect that the result
> > will look a lot like what you and Alex would like to see ... but it's
> > important to understand what deployed implementations actually do, as
> > this is about the rfc3530bis draft, which is NFSv4.0 [yes, dot-zero!].
> 
> My answer is that the question is meaningless (see below).  NFSv4 should
> say neither require nor recommend any specific behavior as
> They leave it to the filesystem, just like every nearly[*] other remote
> filesystem protocol implementation does.  Next question!
> 
> ;)
> 
> filesystems used via NFSv4 do?", which is the same as "does anyone know
> what actual case-insensitive filesystems?".  So let's address that too.
> 
> You'd have to do a survey of filesystems, and that's harder than a survey of
> NFSv4 server implementations.
> 
> filesystems?  Wouldn't it be more useful to ask what all sorts of filesystem
> and non-filesystem things do with GREEK-SMALL-LETTER-FINAL-SIGMA and
> friends?  For example, we can look at IDNA and other Internet standards.
> 
> some useful links there, but no detailed case mapping description on that
> page.
> 
> Nico
> 
> [*] For *other* protocols, such as SMB, or in NFSv4 when trying to present a
> case-sensitive filesystem as case-insensitive, server implementations may
> implement case insensitivity at the fileserver layer when used to share a
> case-sensitive filesystem.  The reverse is not generally feasible, at least on
> general-purpose systems.

Hmm. I'm not sure that surveying filesystems will be any more instructive. 

Non-authoritative, but used as an example.

Windows is UTF-16 from NT onwards (not UCS-2; NT is UCS-2). NTFS (and probably ReFS) is case preserving but case insensitive by default; this is also the default behavior of the NFSv4 server shipped with 2012 that uses NTFS as its source filesystem. It also allows a non-default case preserving and case sensitive option for NFS, but requires a global OS kernel flag change to support this. There is also a user specifiable translation file (sic) "containing mapping information for replacing characters in the names of files when moving them from Windows-based to UNIX-based file systems."

In other words, the answer to case sensitivity for Windows NFSv4 is a mixture of protocol, filesystem, OS and user choice. How does this information help complete 5.10 from a protocol standpoint?

---

From: "Noveck, David" <[email-masked]>
Date: Wed, 8 Jan 2014 13:39:35 -0500

Ø  IMO this is *exactly* like the normalization case: we can't quite specify a required behavior.

It should be.  My idea is that in both cases, we work from existing inter-operating implementations and have to accept those as a given.

Note that, while -29 specifies a number of normalization-related choices (do normalization, ignore normalization, do normalization-insensitive lookups), it does not just wash its hands of the matter.  It gives the server implementation (including NFSv4 server proper and server-side filesystem) a set of choices (all MAYs) but does not say "do whatever you feel like doing".  The main reason is that clients need to know what servers might do and be prepared for that set of behaviors.  Also, we have a set of clients and servers that have evolved ways of working together given the existing range of choices and new implementations need some guidance.  If you want to export a filesystem that uses, for example, NKFC normalization, then neither I, the working group, nor the IESG can stop you nor should they.  But you should be aware that clients/applications may have problems with your server accessing that filesystem.

I'm trying to get enough information to do  the same sort of thing for the case-related stuff and will do so if possible.

Ø  Why aren't we saying this in the new RFC?

We didn't address this in the -29 because I was only looking at chapter 12 and didn't really realize that 5.10 really needs  to be addressed at the same time.

Once I actually read  the existing 5.10 text, I realized we have to address this somehow before moving this document forward.

Ø  Fear of the IESG?

I'm not scared of the IESG.  I'm scared of this process going on forever.

-----Original Message-----
From: nfsv4 [mailto:[email-masked]] On Behalf Of Nico Williams
Sent: Tuesday, January 07, 2014 6:20 PM
To: Noveck, David
Cc: nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

....and the NFSv4 fileserver is layered such that it cannot change the undelying filesystem's case-folding...

The NFSv4 server could fold case itself, knowing that the filesystem does too.  But if the filesystem can be accessed through non-NFSv4 paths... then bad.

IMO this is *exactly* like the normalization case: we can't quite specify a required behavior.  The behavior belongs squarely in a component that neither the NFSv4 client nor the NFSv4 server can safely do anything about.

The more I think about it, the more it seems to me that normalization and case behavior are simply out of scope for NFSv4 *the protocol*, and that the RFC should say exactly that.  It would be fine to have an implementation/interop. advice section describing *filesystem* behaviors that exist and/or that are useful.

Nico

--

nfsv4 mailing list

---

From: "Black, David" <[email-masked]>
Date: Wed, 8 Jan 2014 14:46:43 -0500

> In other words, the answer to case sensitivity for Windows NFSv4 is a mixture
> of protocol, filesystem, OS and user choice. How does this information help
> complete 5.10 from a protocol standpoint?

The internationalization portion of the rfc3530bis draft is being rewritten
with a primary goal of interoperability with deployed implementations - see
the first sentence of section 12.1 in the -29 version of the rfc3530bis draft.
Section 5.10 on character case just got pulled into this effort, because what's
currently in that section is problematic (to put it politely).

So, confirmation of how a Windows NFSv4.0 server behaves wrt case is helpful
information on what deployed implementations actually do.

Among the reasons that this is of interest is that Unicode is not the basis for
all NFSv4.0 internationalization - there are NFSv4.0 servers out there (in this
context, "servers" means the whole kit and caboodle, including the physical
filesystem) that can deal with character sets other than Unicode and 7-bit ASCII.
It'll be useful to understand whether there are any such servers that try to do
case insensitivity for character sets other than Unicode, 7-bit ASCII, or the
7-bit ASCII subset of other character sets.  I hope there aren't because the
notion of case beyond 7-bit ASCII is character-set-specific (e.g., it's not
even consistent among the 8859-* character sets) but "bad ideas" have been known
to have been coded and deployed ...

Thanks,
--David

> From: McDonald, Alex [mailto:[email-masked]]
> Sent: Wednesday, January 08, 2014 11:27 AM
> To: Nico Williams; Black, David
> Subject: RE: [nfsv4] Internationalization and rfc3530bis-29's section 5.10
> 
> Windows is UTF-16 from NT onwards (not UCS-2; NT is UCS-2). NTFS (and probably
> ReFS) is case preserving but case insensitive by default; this is also the
> default behavior of the NFSv4 server shipped with 2012 that uses NTFS as its
> source filesystem. It also allows a non-default case preserving and case
> sensitive option for NFS, but requires a global OS kernel flag change to
> support this. There is also a user specifiable translation file (sic)
> "containing mapping information for replacing characters in the names of files
> when moving them from Windows-based to UNIX-based file systems."
> 
> In other words, the answer to case sensitivity for Windows NFSv4 is a mixture
> of protocol, filesystem, OS and user choice. How does this information help
> complete 5.10 from a protocol standpoint?
>

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jan 2014 13:53:26 -0600

> In other words, the answer to case sensitivity for Windows NFSv4 is a mixture of protocol, filesystem, OS and user choice. How does this information help complete 5.10 from a protocol standpoint?

Indeed.  Consider Spanish and French.  Until recently the rule in
Spanish was that accents are removed when up-casing -- the Royal
Spanish Academy dropped that rule recently but it's still widely used.
 So Spanish speakers may expect a case-insensitive filesystem to treat
"SE" as equivalent to both "se" and "sé", and, therefore, "se" as
equivalent to "sé"; "SÉ" is permitted (since the rule change) but
still infrequent.  French has not had such a rule to my knowledge,
therefore those three (and "SÉ") would all be distinct from each
other.

Fortunately that rule got dropped, so now we can treat "se" and "SE"
as equivalent to each other but distinct from "sé" and "SÉ".

How many such language-specific (as opposed to script-specific)
differences in case-folding might there be lurking that we don't know
about?  Can we hope to satisfactorily address them all?  Why should we
try?

This is from the Unicode 6.3.0 SpecialCasing.txt file:

"
# ================================================================================
# Language-Sensitive Mappings
# These are characters whose full case mappings depend on language and
perhaps also
# context (which characters come before or after). For more information
# see the header of this file and the Unicode Standard.
# ================================================================================

# Lithuanian

# Lithuanian retains the dot in a lowercase i when followed by accents.
"

So language-specific differences in case mapping do exist.

Even ignoring those, see this text from the Unicode 6.3.0 CaseFolding.txt file:

"
# The data supports both implementations that require simple case foldings
# (where string lengths don't change), and implementations that allow
full case folding
# (where string lengths may grow). Note that where they can be supported, the
# full case foldings are superior: for example, they allow "MASSE" and
"Maße" to match.
"

So there's at least two types of acceptable case foldings: "simple"
and "full".  Which one should we mandate or recommend, if any?  What
subset of SpecialCasing.txt should we mandate or recommend, if any?

I think we should recommend one set of case mappings, but with a note
indicating that a) it's likely that implementations will diverge, b)
that future versions of Unicode might change that mapping anyways (in
part because language-specific rules may change, as in the Spanish
case I mentioned above), c) the interop consequences of divergent case
mappings.

I would also prefer to see a note about the fileserver/filesystem
distinction -I think it's difficult to understand these issues without
understanding that- but others seem to prefer not discussing that
distinction in much detail.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jan 2014 14:10:30 -0600

> information on what deployed implementations actually do.

Maybe I'm just too pedantic.  But IMO this question should be phrased
in terms of filesystems, not NFSv4 server implementations.  I
understand your "kit and caboodle" point, but there are NFSv4 server
implementors who do not deal in "kit and caboodle".  It strikes me as
really inappropriate to expect that "NFSv4 server" == "soup to nuts".

It's unfair to those implementors who cannot be responsible for an
entire stack to demand that they be.  Imposing requirements they
cannot meet would be inappropriate and a very bad idea anyways.

> filesystem) that can deal with character sets other than Unicode and 7-bit ASCII.

If you look only at case-insensitive HFS+, NTFS, and ZFS, and maybe
the NAS filesystems... that is, if you exclude FAT*, then we're
probably talking about filesystems that can deal entirely in Unicode,
or which if they allow non-Unicode names, they don't then apply any
case mappings to non-Unicode names.  That's a guess.

> to have been coded and deployed ...

I'm pretty sure there's no such filesystems.

Nico
--

---

From: "Black, David" <[email-masked]>
Date: Wed, 8 Jan 2014 15:21:39 -0500

> really inappropriate to expect that "NFSv4 server" == "soup to nuts".

I understand that, but the protocol spec is written based on a "view from
the wire" independent of who does what beyond the "end of the wire".

> cannot meet would be inappropriate and a very bad idea anyways.

I'm not sure what the "unfair" point is about, but avoiding the latter is
one of the major goals of the rewrite of the rfc3530bis draft's i18n text -
see section 12.1.

Thanks,
--David

> From: Nico Williams [mailto:[email-masked]]
> Sent: Wednesday, January 08, 2014 3:11 PM
> To: Black, David

---

From: "Noveck, David" <[email-masked]>
Date: Wed, 8 Jan 2014 15:30:03 -0500

Ø  It's unfair to those implementors who cannot be responsible for an entire stack to demand that they be.

It would be but nobody has demanded anything of the sort.

Nevertheless, the client sees behavior that reflects the behavior of the NFSv4 server code and the filesystem.   I think the spec can address the fact that these two components exist but, from the clients' point of view what he cares about is the behavior that he will see.

Ø   Imposing requirements they cannot meet would be inappropriate and a very bad idea anyways.

Agree, but want to point out that nobody has done that.   If someone proposes a requirement that cannot be met, that is the time to either get rid of it entirely or downgrade it to a recommendation.

-----Original Message-----
From: nfsv4 [mailto:[email-masked]] On Behalf Of Nico Williams
Sent: Wednesday, January 08, 2014 3:11 PM
To: Black, David
Cc: McDonald, Alex; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Internationalization and rfc3530bis-29's section 5.10

On Wed, Jan 8, 2014 at 1:46 PM, Black, David <[email-masked]<mailto:[email-masked]>> wrote:

> The internationalization portion of the rfc3530bis draft is being

> rewritten with a primary goal of interoperability with deployed

> implementations - see the first sentence of section 12.1 in the -29 version of the rfc3530bis draft.

> Section 5.10 on character case just got pulled into this effort,

> because what's currently in that section is problematic (to put it politely).

> So, confirmation of how a Windows NFSv4.0 server behaves wrt case is

> helpful information on what deployed implementations actually do.

Maybe I'm just too pedantic.  But IMO this question should be phrased in terms of filesystems, not NFSv4 server implementations.  I understand your "kit and caboodle" point, but there are NFSv4 server implementors who do not deal in "kit and caboodle".  It strikes me as really inappropriate to expect that "NFSv4 server" == "soup to nuts".

It's unfair to those implementors who cannot be responsible for an entire stack to demand that they be.  Imposing requirements they cannot meet would be inappropriate and a very bad idea anyways.

> Among the reasons that this is of interest is that Unicode is not the

> basis for all NFSv4.0 internationalization - there are NFSv4.0 servers

> out there (in this context, "servers" means the whole kit and

> caboodle, including the physical

> filesystem) that can deal with character sets other than Unicode and 7-bit ASCII.

If you look only at case-insensitive HFS+, NTFS, and ZFS, and maybe the NAS filesystems... that is, if you exclude FAT*, then we're probably talking about filesystems that can deal entirely in Unicode, or which if they allow non-Unicode names, they don't then apply any case mappings to non-Unicode names.  That's a guess.

> It'll be useful to understand whether there are any such servers that

> try to do case insensitivity for character sets other than Unicode,

> 7-bit ASCII, or the 7-bit ASCII subset of other character sets.  I

> hope there aren't because the notion of case beyond 7-bit ASCII is

> character-set-specific (e.g., it's not even consistent among the

> 8859-* character sets) but "bad ideas" have been known to have been coded and deployed ...

Nico

--

nfsv4 mailing list

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jan 2014 15:05:21 -0600

> the wire" independent of who does what beyond the "end of the wire".

But we can't very well ignore reality.

The reality is that there's a very clear and bright boundary between
"fileserver" and "filesystem".  If we write requirements and
recommendations that ignore that reality then we shouldn't be
surprised if implementors end up "ignoring" them.

This view that the description of the bytes on the wire must capture
everything else is naive unless capturing all those other details
includes noting that they vary widely because of the separation
between fileserver and filesystem.  If it's the latter then we're
wasting our time unless the goal is to document reality, but it'd be
far easier to document that distinction between fileserver and
filesystem than it is to document what all the case-insensitive
filesystems use for case mapping!

> one of the major goals of the rewrite of the rfc3530bis draft's i18n text -

It's unfair to impose requirements that one class of implementors
cannot hope to meet.  It'd be worse than that if that class of
implementors happens to cover (as it does) all those implementing on
general purpose OSes.  And it'd be utterly useless to impose such
requirements if -as I suspect- that class of implementors happens to
have all existing implementors in it.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jan 2014 15:21:22 -0600

On Wed, Jan 8, 2014 at 12:39 PM, Noveck, David <[email-masked]> wrote:
> Note that, while -29 specifies a number of normalization-related choices (do
> normalization, ignore normalization, do normalization-insensitive lookups),
> it does not just wash its hands of the matter.  It gives the server
> implementation (including NFSv4 server proper and server-side filesystem) a
> set of choices (all MAYs) but does not say “do whatever you feel like
> doing”.  The main reason is that clients need to know what servers might do
> and be prepared for that set of behaviors.  Also, we have a set of clients
> and servers that have evolved ways of working together given the existing
> range of choices and new implementations need some guidance.  If you want to
> export a filesystem that uses, for example, NKFC normalization, then neither
> I, the working group, nor the IESG can stop you nor should they.  But you
> should be aware that clients/applications may have problems with your server
> accessing that filesystem.

I agree, mostly.  In practice if you normalize to NFKC on CREATE (and
LOOKUP) no one is likely to notice.  If you normalize to NFD or NFKD
people will notice (and they have noticed).

Also, I don't see how the choice of normalization behavior affects
*clients*.  It affects applications and users.  Once more, I think
it's important to distinguish the different parts of the stack.

> I’m trying to get enough information to do  the same sort of thing for the
> case-related stuff and will do so if possible.

I see.  But with normalization we were lucky: there were only two
known cases of filesystems that did anything about normalization.
Many more deal with case insensitivity.  We're likely to find at least
as many choices as filesystems that have case insensitivity, even if
you exclude FAT32.

> Once I actually read  the existing 5.10 text, I realized we have to address
> this somehow before moving this document forward.

I think the text should have informative text describing what
"case-insensitive" means, with references to the relevant Unicode
standards, and noting that there are several choices within Unicode
itself, and also noting that there are laguage-specific choices too,
not just codeset-specific choices.

> I’m not scared of the IESG.  I’m scared of this process going on forever.

Characterizing all the case-insensitive behaviors will certainly take time...

Nico
--

---

From: "Black, David" <[email-masked]>
Date: Wed, 8 Jan 2014 18:07:05 -0500

Nico - small excerpts to make one thing clear:

> > I'm not sure what the "unfair" point is about [... snip ...]
> 
> cannot hope to meet.

I absolutely agree, period

I've been there and have the scars in getting that sort of situation
right on more than one occasion as a WG chair, even though I have no
such role here.

It looks like you're mis-interpreting "server" as "NFSv4.0-server-only"
and then proceeding with a vigorous strawman demolition exercise.  Please
re-read section 12.2 of the -29 version of the rfc3530bis draft, which
is intended to recognize that boundary.  If you have suggestions for how
to make the fileserver vs. filesystem clearer in that section of the
draft, please suggest text as opposed to complaining that the boundary
is being ignored - it is not being ignored.

Beyond that, It feels like the supposed "requirements" are being invented
out of thin air, as neither Dave Noveck nor I have posted any proposed
requirements for case handling, and moreover Dave's original email
pointed out the primacy of existing implementations in figuring out
how to write the text:

"In any case, there is an issue that arises from the idea of rfc3530bis’s
approach to internationalization being that it is based on current
inter-operating implementations."

To which I followed up with:

"Writing a new version of the NFSv4.0 spec to which none of the existing
implementations comply is not a good idea,"

This is an effort to gather information on what actual implementations do
towards a goal of coming up with text that doesn't cause problems for
existing deployed NFSv4.0 implementations.

Thanks,
--David

> Sent: Wednesday, January 08, 2014 4:05 PM
> To: Black, David

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jan 2014 18:21:58 -0600

For ZFS you can look at Illumos source.  It's difficult to decipher
what the case mappings are, as the case mappings were compiled into
four levels of tables, each indexed by the corresponding byte of the
UTF-8 encoding of the given codepoint.  Decompiling these tables is
probably too much work; someone at Oracle can instead look at the
original tables that were used to compile from.  In case you want to
see the compiled tables, they are here:
http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/sys/u8_textprep_data.h

and the code using them is here:
http://src.illumos.org/source/xref/illumos-gate/usr/src/common/unicode/u8_textprep.c#462

It might be easier (and certainly more general) to write a program to
go through the Unicode CaseFolding.txt and SpecialCasing.txt files and
try all the interesting upper/lower case pairs on a live filesystem.
That could be used to measure many filesystem implementations.

Nico
--

---

From: Tom Talpey <[email-masked]>
Date: Wed, 08 Jan 2014 19:33:20 -0800

> information on what deployed implementations actually do.

There is no Windows NFSv4.0 server. There are NFSv2, NFSv3 and NFSv4.1
Windows servers, but v4.0 is not implemented.

---

From: Nico Williams <[email-masked]>
Date: Wed, 8 Jan 2014 21:48:17 -0600

> That could be used to measure many filesystem implementations.

I've a script, but I need to port it to work on OS X.  I'll post it
maybe tonight, maybe tomorrow.

---

From: Nico Williams <[email-masked]>
Date: Thu, 9 Jan 2014 23:57:20 -0600

bash printf \u functionality is rather unreliable.  I re-wrote the thing
in a better language.  I've reports for ZFS, HFS+, and NTFS, all
obtained locally, unfiltered by any NFSv4 service.  Posting them to the
list seems unlikely to be useful.  I'll send them to David Noveck, David
Black, and anyone who asks.

To save you the suspense: all three filesystems have different case
mapping tables.  Forget about standardizing that; point to Unicode and
say "pick one of common, simple, and full" and "if you must support
special case mappings make that configurable on a filesystem basis".

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Fri, 10 Jan 2014 00:04:40 -0600

Nico                                                                                                                                               
--