Subject: RE: Case sensitivity on sftp servers

From: Richard Whalen <[email-masked]>
Date: Fri, 5 Sep 2003 17:14:41 -0400

I think that this will work.

-----Original Message-----
From: Joseph Galbraith [mailto:[email-masked]]
Sent: Friday, September 05, 2003 4:38 PM
To: Richard Whalen
Cc: 'Joseph Galbraith'; [email-masked]
Subject: Re: Case sensitivity on sftp servers

> Though I agree that specifying case sensitivity may be useful, I think
that
> specifying it on a server wide basis is too broad.
> 
> The server may offer several file systems and some of these may not
preserve
> case, while others may preserve case but not be case sensitive, and still
> other may preserve case and be case sensitive.
> 
> The server may also have the option of allowing the client to specify
> whether or not file operations are case sensitive on a file system that
> preserves case.
> 
> It may be necessary to specify a method of obtaining the file systems
> available and their characteristics (Read Only, Case Preserving, Case
> Sensitive, etc.)

You raise a good point--- hmmm... let's try this on
for size:

new attrib bit:

SSH_FILEXFER_ATTR_FILESYSTEM
	string filesystem-name

SSH_FILEXFER_ATTR_FILESYSTEM is only sent by the server
if the client requested it by including in the flags
mask during readdir or stat operation and the file
in question is a directory.

filesystem-name is a handle that can be passed back
to the server using the "read-filesystem-attributes"
It need not be human readable.

-- 
byte   SSH_FXP_EXTENDED
uint32 request-id
string "read-filesystem-attributes"
string filesystem-name

byte SSH_FXP_EXTENDED_REPLY
uint32 request-id
uint32 file-system attribute flags
	CASE_SENSITIVE
	CASE_PRESERVITIVE
	CASE_OPTIONAL
string-utf8 invalid-chars
uint32 reserved-name-count
string reserved-name[1..reserved-name-count]

And, a new flag bit in in the modes during open to
specify that the filename should be case sensitive
or case insensitive.  If the server can't comply,
it return OP_UNSUPPORTED.

What do you think?

I think I keep the "default-file-attributes" and then
the filesystem only need be sent if it is different from
the default.

- Joseph

---

From: Martin Pool <[email-masked]>
Date: Mon, 8 Sep 2003 12:20:44 +1000

> It need not be human readable.

Alternatively, you could make case sensitivity directly be an
attribute of the directory, omitting a layer of indirection.  (Of
course this layer might be useful for discovering e.g. free space
later.)

> it return OP_UNSUPPORTED.

I take it this is intended to control only the way the server matches
against existing filenames, and not to influence the way new files are
created?

So with case sensitivity on

  OPEN("README", WRITE|CREAT|EXCL|CASE_INSENSITIVE) 

in a directory containing "readme" ought to fail with
FILE_ALREADY_EXISTS even on a case-sensitive filesystem, and

  OPEN("README", READ|CASE_SENSITIVE)

in a directory containing "readme" ought to fail with NO_SUCH_FILE
even on a case-insensitive filesystem?

Do you perhaps want a ternary for sensitive/insensitive/don't-care?
Some clients might be happy to use whatever the server wants.

If you give the client this option when opening files, then presumably
it should have the option for all other operations that include
filenames, including remove, rename, stat, etc.

Since it affects many operations and clients are unlikely to want to
change partway through, perhaps it should be a per-connection option?

I would like to mention that in Samba and CIFS, experience has shown
that negotiated case-sensitivity in a protocol was buggy and
surprisingly hard to implement reliably efficiently.  For example,
many Windows applications use varying case for the same file even when
NT tells them that they are not on a case-insensitive filesystem.
Doing case-insensitive compares in Unicode is not trivial and there
are implementations that get it wrong.

Adding this bit doubles or triples the testing domain for many
operations.  I would suggest not adding this option unless you're
really sure it's needed.

-- 
Martin

---

From: "Dan O'Reilly" <[email-masked]>
Date: Sun, 07 Sep 2003 20:42:32 -0600

>even on a case-insensitive filesystem?

No, it wouldn't necessarily do so on VMS, which supports multiple versions
of files.

Think outside the box, Grasshopper...<grin>

>operations.  I would suggest not adding this option unless you're

On UNIX, maybe not.  On Windows, maybe not.  But they're not the only
systems...

------
+-------------------------------+----------------------------------------+
| Dan O'Reilly                  |  "There are 10 types of people in this |
| Principal Engineer            |   world: those who understand binary   |
| Process Software              |   and those who don't."                |
| http://www.process.com        |                                        |
+-------------------------------+----------------------------------------+

---

From: Martin Pool <[email-masked]>
Date: Mon, 8 Sep 2003 14:34:53 +1000

On  7 Sep 2003 Dan O'Reilly <[email-masked]> wrote:

> No, it wouldn't necessarily do so on VMS, which supports multiple
> versions of files.

How would it behave on VMS, sensei?

-- 
Martin

---

From: "Dan O'Reilly" <[email-masked]>
Date: Mon, 08 Sep 2003 08:17:08 -0600

>How would it behave on VMS, sensei?

Well, specifically with respect to the example creating the file, it
would simply create a new version of the file, leaving the old one intact
(depending on if a version limit for the file/directory was set, and the
depth to which it is set).

------

---

From: Joseph Galbraith <[email-masked]>
Date: Mon, 08 Sep 2003 14:20:18 -0600

> course this layer might be useful for discovering e.g. free space

Also, if the filesystem is different, the other characteristics
of the filesystem may also change (reserved filenames and illegal
characters.

> against existing filenames, and not to influence the way new files are

No.  WinNT has the ability, at least under NTFS to
optionally support case.  I don't know about VMS?
And maybe others.

[...]

> Some clients might be happy to use whatever the server wants.

This would be a good idea: CASE_NATIVE.

> operations.  I would suggest not adding this option unless you're

I think I would tend to agree; while MS CreateFile() api does support
this, many other APIs [such as DeleteFile()] don't.

What I want is the ability to know whether or not Readme
and README are the same file.  And perhaps I don't really
care about illegal characters and names.

So, let me summerize our proposals:

Case sensitivity proposal #1, alias: the-first-cut
--------------------------------------------------
"[email-masked]" --
    o deal with only the system native case-sensitivity.
    o can't deal with multiple filesystem handling case differently.
    o can't deal with filesystems where case sensitivity is optional.

Case sensitivity proposal #2, alias: the-whole-enchillada
---------------------------------------------------------
"filesystem-name" + "read-filesystem-attributes"
    o deal with case sensitivity on a per directory bases,
      so multiple filesystems are handled.
    o Case sensitivity can be specified during open
    o Probably needs to specify sensitivity during delete/rename/etc.,etc.
    o Probably hard to implement and get reliable (based on Samba/CIFS experience.)

Case sensitivity proposal #3, alias: the-lean-mean-case-sensitive-machine
-------------------------------------------------------------------------
(And also deals with other issues on my list):

New attrib bit: SSH_FILEXFER_ATTR_FLAGS

The 'flags' field is any combination of the following bits:
     SSH_FILEXFER_ATTR_FLAGS_READONLY
         Advisory readonly bit.  This bit is not part of the
         access control information on the file, but is rather
         an advisory field indicating that the file should not
         be written.

         For example, some source code control systems use this
         bit to indicate that the file is not locked, and therefor
         should not be modified.  When the file is locked, the bit
         is cleared.

     SSH_FILEXFER_ATTR_FLAGS_SYSTEM
         The file is part of operating system.

     SSH_FILEXFER_ATTR_FLAGS_HIDDEN
         File should not be shown to user unless specifically
         requested.  For example, most unix systems should set
         this bit if the filename begins with a 'period'.

     SSH_FILEXFER_ATTR_FLAGS_CASE_INSENSITIVE (directory only)
         Files & directory names in this directory should be compared with-out
         regard to case.

         It is recommended that where possible, the filesystem be allowed to
         do comparisons.  For example, if a client wished to prompt a user before
         overwriting a file, it should not compare the new name with the previously
         retrieved list of names in the directory.  Rather, it should first try to
         create the new file by specifying SSH_FXF_CREAT without the SSH_FXF_TRUNC
         flag.  Then, if this fails and returns SSH_FX_FILE_ALREADY_EXISTS, it
         should prompt the user and then retry the create specifying SSH_FXF_TRUNC.

         The case insensitive flag should only be used to implement name globbing
         or other filename based filtering systems.

         Unless otherwise specified, filenames are assumed to be case sensitive.

     SSH_FILEXFER_ATTR_FLAGS_ARCHIVE
         The file should be included in backup / archive operations.

     SSH_FILEXFER_ATTR_FLAGS_ENCRYPTED
         The file should be encrypted.  If the filesystem doesn't support
         encryption, the operation must return UNSUPPORTED.

     SSH_FILEXFER_ATTR_FLAGS_COMPRESSED
         The file should be compressed.  If the filesystem doesn't support
         compression, the operation must return UNSUPPORTED.

     SSH_FILEXFER_ATTR_FLAGS_SPARSE_OPTION
     SSH_FILEXFER_ATTR_FLAGS_SPARSE_REQUIRED
         The file should be sparse.  If SPARSE_REQUIRED is specified, the operation
         MUST fail if it is not supported.

My preference at this point leans toward #3.  Do any other implementors
supporting other operating systems have bits that would be appropriate
for this field?

- Joseph

---

From: Martin Pool <[email-masked]>
Date: Tue, 9 Sep 2003 10:45:32 +1000

On  8 Sep 2003 Dan O'Reilly <[email-masked]> wrote:

> > > >NO_SUCH_FILE even on a case-insensitive filesystem?
> > >
> would simply create a new version of the file, leaving the old one
> intact(depending on if a version limit for the file/directory was set,
> and the depth to which it is set).

Doesn't that contradict the meaning of SSH_FXF_EXCL?

  Causes the request to fail if the named file already exists.

Anyhow, what does that have to do with case sensitivity?

-- 
Martin

---

From: Martin Pool <[email-masked]>
Date: Tue, 9 Sep 2003 11:27:46 +1000

On  8 Sep 2003 Joseph Galbraith <[email-masked]> wrote:

> > I take it this is intended to control only the way the server
> > matches against existing filenames, and not to influence the way new
> > files are created?
> 
> optionally support case. 

Yes, the filesystem supports it, but as far as I know it cannot
usefully be accessed from applications except from in the crippled
POSIX mode.  See e.g.

  http://techsupt.winbatch.com/TS/T000001036004F26.html

In any case this is a per-filesystem option, not something that can be
specified when creating a file.  

It's hard to imagine a filesystem where you can specify that when you
create a file, though perhaps it could be supported when creating a
directory.

> I don't know about VMS?  And maybe others.

I think VMS is always case-insensitive.  

OS X is an interesting case because it commonly uses both
case-sensitive (UFS) and case-preserving (HFS) filesystems.  However
in neither case does the application get to choose which semantics it
wants.  All you can do is find out what a particular
directory/filesystem will do.  (Linux is similar, although the
case-insensitive filesystems are less often used.)

> > many Windows applications use varying case for the same file even
> > when NT tells them that they are not on a case-insensitive
> > filesystem. Doing case-insensitive compares in Unicode is not
> > trivial and there are implementations that get it wrong.
> > 

Yes, creating files that you can't rename or delete is not very
useful.

> and README are the same file.

> And perhaps I don't really care about illegal characters and names.

Perhaps it's easier to have clients just try it and see?  That's
simpler, more reliable and probably not too expensive.

>          this bit if the filename begins with a 'period'.

On Unix this would presumably be a readonly bit?

>          Files & directory names in this directory should be compared
>          with-out regard to case.
> 
>          It is recommended that where possible, the filesystem be
>          allowed to do comparisons.  For example, if a client wished
>          to prompt a user before overwriting a file, it should not
>          compare the new name with the previously retrieved list of
>          names in the directory.  Rather, it should first try to
>          create the new file by specifying SSH_FXF_CREAT without the
>          SSH_FXF_TRUNC flag.

I like this recommendation, but should it use SSH_FXF_EXCL as well?

>          Then, if this fails and returns
>          SSH_FX_FILE_ALREADY_EXISTS, it should prompt the user and
>          then retry the create specifying SSH_FXF_TRUNC.
> 
>          The case insensitive flag should only be used to implement
>          name globbing or other filename based filtering systems.
> 
>          The file should be encrypted.  If the filesystem doesn't
>          support encryption, the operation must return UNSUPPORTED.
> 
>          The file should be compressed.  If the filesystem doesn't
>          support compression, the operation must return UNSUPPORTED.
> 
>          The file should be sparse.  If SPARSE_REQUIRED is specified,
>          the operation MUST fail if it is not supported.
> 
> My preference at this point leans toward #3.  Do any other
> implementors supporting other operating systems have bits that would
> be appropriate for this field?

#3 looks good to me too.

Does putting all these bits in a single attribute cause a problem when
setting the attributes?  Some clients might not have permission to set
some bits; or the bits might be readonly.  I suppose the server has to
just ignore those bits if they already have the specified values.

If you are collecting a list of bits, you might like to include some
Linux attribute bits, some of which are on other systems too.  I don't
think they're a priority for remote access, but access over filexfer
be useful for backups:

  SSH_FILEXFER_ATTR_FLAGS_APPEND_ONLY

       The file can only be opened for writing in append mode.

  SSH_FILEXFER_ATTR_FLAGS_IMMUTABLE

       The file cannot be deleted or renamed, no hard link can be
       created to this file and no data can be written to the file.

       This bit implies a stronger level of protection than
       SSH_FILEXFER_ATTR_FLAGS_READONLY, the file permission mask or
       ACLs.  Typically even the superuser cannot write to immutable
       files, and only the superuser can set or remove the bit.

  SSH_FILEXFER_ATTR_FLAGS_SYNC

       When the file is modified, the changes are written
       synchronously to the disk.

-- 
Martin

---

From: Joseph Galbraith <[email-masked]>
Date: Tue, 09 Sep 2003 08:53:48 -0600

> specified when creating a file.  

Actually, the Win32 CreateFile() api takes a flag:

FILE_FLAG_POSIX_SEMANTICS
     Indicates that the file is to be accessed according
     to POSIX rules. This includes allowing multiple files
     with names, differing only in case, for file systems
     that support such naming. Use care when using this
     option because files created with this flag may not
     be accessible by applications written for MS-DOS or
     16-bit Windows.

So it is accessable from Win32 (not just from the posix subsystem)
and can be specified when creating the file.  I'm not sure when this
flag was introduced; it may have been with windows 2000.

> create a file, though perhaps it could be supported when creating a

Unfortunately, I don't think Win32 offers support
for creating case-sensitive directories.  Or for
any other case-sensitive operations.  Though if
you are willing to go straight to the NT API
(as opposed to the Win32 API) I'm quite sure
that you can do any file operation you want in
either a case sensitive fashion or case insensitive.

(Provided of course, you are working with an NTFS
partition or some other filesystem that can support
both.)

> On Unix this would presumably be a readonly bit?

Yes, I would think so.

> I like this recommendation, but should it use SSH_FXF_EXCL as well?

Yes, you are correct.

> just ignore those bits if they already have the specified values.

Well, it means you probably have to read the mask before
you write it (like mode bits if you only want to change
the other permissions.)

>        synchronously to the disk.

Excellent; I'll add these-- being able to do good backups is one
of the applications that has been on my mind.

Thanks,

Joseph

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 9 Sep 2003 08:33:27 -0700

> create a file, though perhaps it could be supported when creating a

The namespace where case sensitivity matters is the directory's, so at
first glance it makes no sense to specify case behaviour on file opens.

But I can see case-behaviour-in-file-open as a short cut for checking
the case behaviour of the containing directory (e.g., "preserve this new
file's name's case or fail if you can't" -> short for "does the
directory where this file is to be created preserve case?  If yes, then
create the file, if not fail"; "open the file that corresponds to name
'Xyz' with case-sensitive matching" -> "does 'Xyz's parent directory
preserve case, and if so, does it have a case-sensitive namespace?  If
yes open file 'Xyz', else fail").

Is such a short cut justified?  I think not.

> case-insensitive filesystems are less often used.)

Bingo.  I think the client needs to be able to determine the case rules
of each directory and otherwise leave it at that (plus, maybe, the case-
behaviour-in-file-open conditional).

> > and README are the same file.

Perhaps the protocol should offer a file and/or file handle equality
operation, particularly since there's no "inode" or otherwise "unique"
internal identifier attribute (nor can there always be).  And/or maybe
we could use a glob operation, so the client could glob("Readme").

> simpler, more reliable and probably not too expensive.

Agreed.

Nico
--

---

From: Martin Pool <[email-masked]>
Date: Wed, 10 Sep 2003 12:45:34 +1000

> FILE_FLAG_POSIX_SEMANTICS

Fine, you can create "README" and "Readme".  But attempting to copy,
delete or rename them, pass them to libraries, or open them from
another program might get an error, one file at random, or both
depending on how you do it. This does not seem very useful, and reports
on the web indicate that people porting from Posix did not find it
helpful.  This is born out by the Samba experience.

Basically every report I know of indicates that trying to be
case-sensitive on NT is just thoroughly painful even though it's
theoretically supported.  I don't think there's any sense in adding
specific support to the protocol.  If the option is in the protocol
then users might try it, but they're likely to be confused and
disappointed.

Of course if somebody has done case-sensitive NT in production and was
happy with the results then I withdraw.

> you write it (like mode bits if you only want to change

OK, just checking.

-- 
Martin

---

From: Martin Pool <[email-masked]>
Date: Tue, 23 Sep 2003 11:05:09 +1000

In the threads about rename and case sensitivity last month, we seemed
to reach consensus amongst the people who posted.  Will the revised
draft paragraphs be added to the standard?

-- 
Martin

---

From: Joseph Galbraith <[email-masked]>
Date: Tue, 23 Sep 2003 09:16:39 -0600

> draft paragraphs be added to the standard?

Yes.  I'm working on a number of revisions to the
draft.  I'd hoped to have it out by last week-- now
I hope to have it out sometime this week :-)

- Joseph