Subject: [nfsv4] rfc3530bis -30: Approach to case insensitivity (5.10)

From: "Black, David" <[email-masked]>
Date: Thu, 9 Jan 2014 18:01:31 -0500

Since there's been such intense interest in this ;-), I thought I'd
offer some initial thoughts on what to do about section 5.10 on 
case insensitivity.

The current text in section 5.10 is definitely wrong, as it bases
case insensitivity on the (descriptive) names of characters, although
its goal of describing how to build table-driven case mapping seems
good.  There are no requirements keywords (e.g., "MUST", "SHOULD")
in that section, and so it's probably a good idea to keep things
that way in the new text (which will likely be moved into a section
12 somewhere with a forward pointer from 5.10).

For UTF-8-only servers and physical filesystems, the following sentence
from section 4.1.3 of draft-ietf-precis-framework-13 (with RECOMMENDED
changed to lower case) looks like a good starting point:

   If case preservation is not desired, it is RECOMMENDED to use Unicode
   Default Case Folding as defined in Chapter 3 of the Unicode Standard
   [UNICODE].

Beyond that, there is some useful discussion of locale-specific case
mapping (e.g., for dotted and dotless 'i' in Turkish) in section 2.3
of draft-ietf-precis-mappings-05.  Overall, I'd prefer to write something
short to basically say that NFSv4.0 case insensitivity for Unicode
should be implemented via case folding as specified in the Unicode
standard, go look at that standard for more details ;-).

A consideration in doing that is the versions of the Unicode standard
(e.g., Christoph has pointed out an implementation that is still at
Unicode 3.2, vs. the current Unicode 6.3 version).  Some text may be
in order to say something about use of prior versions of the Unicode
standard.

When multiple character sets are possible (e.g., server and physical
filesystem filename implementations are 8-bit-clean), the concept of
character case beyond 7-bit ASCII is character-set-specific.  Section
12.8 of the -30 draft already recommends against some character-set-
specific string processing when the character set is not known; that
could be enhanced with a specific recommendation against case
insensitivity when the character set is not known.

I'd ask for comments, but I suspect that such an invitation will not
be necessary ;-).  Information on what actual implementations do is
still of interest.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Nico Williams <[email-masked]>
Date: Thu, 9 Jan 2014 19:13:31 -0600

Referencing UCS-4 is definitely wrong.  Please let's reference a
specific version of Unicode.  Copying PRECIS as to this seems like a
good place to start, yes.

My advice:

 - RECOMMEND that implementors of case-insensitive filesystem
behaviors implement all the common and simple case mappings from some
Unicode version (pick one).

 - Permit (MAY, OPTIONAL) case-insensitive filesystem behavior using
full and special (language-specific) case mappings.

 - Note that the exact set of case mappings that in use may vary
(though generally for any one filesystem case mappings are not subject
to change after filesystem creation).

Nico
--

---

From: "Black, David" <[email-masked]>
Date: Thu, 9 Jan 2014 20:25:55 -0500

> good place to start, yes.

Thanks.  I think the latest version of Unicode should be referenced with
allowance for use of equivalent functionality in other versions.

> to change after filesystem creation).

Working from PRECIS, I'd prefer to write text about case folding rather
than case mapping.  See answer to second question here:

http://www.unicode.org/faq/casemap_charprop.html

--David

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 10 Jan 2014 10:13:18 +0100

> be necessary ;-).  Information on what actual implementations do is

Btw, I'd love to get a hand on Nico's tool.  In Linux we have various
filesystems with case insentivie behaviour, and there are lots of differences
in how and when they perform case folding.  I was trying to nudge people
towards common behaviour when XFS CI support was in progress, but it didn't
get very far.  Many of those filesystems are NFS exportable, so you could
see them in the wild, even if they aren't common configurations.

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 10 Jan 2014 10:19:14 +0100

> full and special (language-specific) case mappings.

For many of the legacy filesystems from other OSes that Linux supports
we have a common library of case mappings that came from various defacto
or real standards, mostly old DOS code pages and ISO 88??? standards.

Some communities still use them heavily as utf8 has horrible inefficiencies
for a few of the charsets.

While various of the filesystems can be NFS exported I'm curious if people
consider them important enough to bother in the standard, as I'm pretty
sure they run afould the NFSv4 spec in various other ways anyway.

> to change after filesystem creation).

That's unfortunately not true for the PC-legacy filesystems, which do not
store the mapping table on disk, but as a mount parameter.  While changing
the character set at runtime is a bad idea it's easily possible at least
for the Linux implementations of FAT, JFS or UDF to pick a few that
are NFS exportable.

But as set above I'm not sure if these are really something that should
be worried about in a standard.

---

From: "McDonald, Alex" <[email-masked]>
Date: Fri, 10 Jan 2014 15:06:07 +0000

> 
> A consideration in doing that is the versions of the Unicode standard (e.g.,
> Christoph has pointed out an implementation that is still at Unicode 3.2, vs.
> the current Unicode 6.3 version).  Some text may be in order to say
> something about use of prior versions of the Unicode standard.
> 

Once the requirements are decided and what parts of Unicode apply, then there are Unicode policies with respect to this that can be referred to directly;  http://www.unicode.org/policies/policies.html#Stability

Alex McDonald
Standards & Industry Associations Group
CTO Office
NetApp
+44 7795 046686 Mobile Phone
[email-masked]
twitter: @alextangent
Follow us:

---

From: Nico Williams <[email-masked]>
Date: Fri, 10 Jan 2014 11:29:13 -0600

> or real standards, mostly old DOS code pages and ISO 88??? standards.

Yes, that has to be allowed, but remember that I'm drawing a big
distinction between file *server* and filesystem.  Recommendations here
should be for new filesystems.  Legacy content is what it is; legacy has
to be permitted.

> for a few of the charsets.

That's not really true.  Much has been written about the supposed
inefficiencies of UTF-8, and they are not that great.  In the case of
filesystems I seriously doubt that UTF-8 vs. anything else can make much
of a difference in benchmarks (even ones specifically designed to
measure this).  (Normalization and case folding are more likely to cause
performance problems, but even then, in actual usage it shouldn't be a
problem because there are many opportunities for optimization.)

> sure they run afould the NFSv4 spec in various other ways anyway.

They need to be permitted.

> That's unfortunately not true for the PC-legacy filesystems, which do not

I said "generally" -- a weasel word, I know.

> for the Linux implementations of FAT, JFS or UDF to pick a few that

Oh, I was expecting FAT to be the only one...  Anyways...

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Fri, 10 Jan 2014 11:35:35 -0600

> see them in the wild, even if they aren't common configurations.

Oh, interesting.  I didn't know that.

Anyways, code inlined below.  I had a bash script first, but much
depended on the version of bash installed, so I re-wrote it in C (why
not Python?  I guess once we were talking re-write in a sytems
programming language C seemed to be the one that would give me the most
control).

You'll get some warnings on Cygwin due to my usage of unsigned char.

You need to set stdin to be CaseFolding.txt from Unicode, and the
current directory needs to be one where the program can create and
unlink files.  There's no dependencies on anything other than the C
run-time.

Enjoy.

#include <assert.h>
#include <ctype.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>

static int
parse_chars(unsigned long int *cp, size_t nc, size_t *ncp, const unsigned char *buf)
{
    long long int c;
    const unsigned char *p;
    char *e;

    *ncp = 0;
    for (p = buf; p != NULL && p[0] != ';' && nc > 0; nc--) {
        errno = 0;
        c = strtoll(p, &e, 16);
        if (c < 0 || c > 0x10FFFF || errno != 0)
            return 0;
        *(cp++) = c;
        p += (unsigned char *)e - p;
        (*ncp)++;
    }
    return 1;
}

static void
encode(unsigned char *p, size_t sz, unsigned long int *cp, size_t nc)
{
    *p = '\0';
    for (; sz > 0 && nc > 0; nc--, cp++) {
        if (*cp < 0x80) {
            *(p++) = *cp;
            sz--;
        } else if (*cp < 0x800) {
            *(p++) = 0xC0 | (*cp >> 6);
            *(p++) = 0x80 | (*cp & 0x3F);
            sz -= 2;
        } else if (*cp < 0xFFFF) {
            *(p++) = 0xE0 | (*cp >> 12);
            *(p++) = 0x80 | ((*cp >> 6) & 0x3F);
            *(p++) = 0x80 | (*cp & 0x3F);
            sz -= 3;
        } else {
            *(p++) = 0xF0 | (*cp >> 18);
            *(p++) = 0x80 | ((*cp >> 12) & 0x3F);
            *(p++) = 0x80 | ((*cp >>  6) & 0x3F);
            *(p++) = 0x80 | (*cp & 0x3F);
            sz -= 4;
        }
        *(p++) = '\0';
    }
    assert(sz > 0);
}

int
main(int argc, char **argv)
{
    unsigned char buf[128];
    unsigned char to_create[64];
    unsigned char to_stat[64];
    unsigned char *p;
    unsigned long int c[4];
    struct stat st;
    int fd;
    unsigned long lineno = 0;
    size_t nc;

    while (fgets(buf, sizeof(buf), stdin) != NULL) {
        lineno++;
        p = strchr(buf, '\n');
        if (p == NULL) {
            fprintf(stderr, "Warning: line %lu too long\n", lineno);
            continue;
        }
        *p = '\0';
        p = buf;
        if (!isalnum(*p))
            continue;       /* Comments and empty lines */

        if (!parse_chars(c, sizeof(c)/sizeof(c[0]), &nc, p)) {
            fprintf(stderr, "Warning: couldn't parse uppercase char in "
                    "line %lu (%s)\n", lineno, buf);
            continue;
        }
        encode(to_create, sizeof(to_create), c, nc);
        assert(to_create[0] != '\0');
        if ((p = strpbrk(p, ";")) == NULL ||
            (p = strpbrk(p + 1, ";")) == NULL) {
            fprintf(stderr, "Warning: malformed line %lu (%s)\n", lineno, buf);
            continue;
        }
        if (!parse_chars(c, sizeof(c)/sizeof(c[0]), &nc, p + 1)) {
            fprintf(stderr, "Warning: couldn't parse lowercase char in "
                    "line %lu (%s)\n", lineno, buf);
            continue;
        }
        encode(to_stat, sizeof(to_stat), c, nc);
        assert(to_stat[0] != '\0');

        fd = open(to_create, O_CREAT | O_EXCL | O_WRONLY, 0600);
        if (fd == -1) {
            fprintf(stderr, "Warning: failed to create %s: %s (%d)\n",
                    to_create, strerror(errno), errno);
            continue;
        }
        close(fd);
        if (stat(to_stat, &st) == -1)
            printf("NOT FOLDED: %s -> %s (%s)\n", to_create, to_stat, buf);
        else
            printf("FOLDED: %s -> %s (%s)\n", to_create, to_stat, buf);
        unlink(to_create);
    }
}

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 10 Jan 2014 19:41:22 +0100

> problem because there are many opportunities for optimization.)

For filenames themselves I'm pretty sure it doesn't make a practical
difference.  But remember that in traditional Unix filesystems semantics
the file name is just a byte stream, and the whole iocharset convention
is in place primarily so that for the case insenstive filesystem that
filesystem can be made aware of the character set the user uses for anything
else on the system.  So while utf8 vs other charset might not make a
difference for the FAT using external volume it might make a difference
for other workloads, and people try to have their filesystem fit in with
the rest.

I'm not really interested in discussing the actual efficiency btw, just
saying that there is a big enough userbase still using other character set of
this, possibly just perceived, inefficieny.

---

From: Nico Williams <[email-masked]>
Date: Fri, 10 Jan 2014 12:52:58 -0600

> this, possibly just perceived, inefficieny.

They're using those for legacy reasons.  We need to support that, full
stop.  So we're in violent agreement, methinks :)