Subject: A proposal for OPEN

From: Joseph Galbraith <[email-masked]>
Date: Mon, 08 Sep 2003 17:37:46 -0600

As we've worked with SFTP v4, we've found the open
command a little awkward in some cases because the
client can't really specify what access it wants
and so we are left guessing how to open the file.

For example, we typically open the file with
READ_DATA and/or WRITE_DATA access, but not
with WRITE_ACL.  If the client does a fsetstat
and includes an ACL, we will fail because we
didn't open the file with enough access.  But,
if we opened the file for WRITE_ACL, we might
fail to open the file and the client never
would have attempted to write an acl anyway.

I'm proposing the following new open command
for v5:

byte   SSH_FXP_OPEN
string filename
uint32 access_desired
uint32 access_flags
attrib open_attrs

access_desired
    Any of the ace-mask flags from section 5.7.

access_flags
    The access_flags is two seperate bit fields.  Bit values
    in the 0xFFFF0000 mask are required, and if they can not
    be supported, the operation MUST fail.

    Bit values in the 0x0000FFFF mask are requested, but if they
    can not be supported the operation SHOULD continue.

    Clients MUST NOT send any undefined bits in the access_flags
    field.  Servers should ignore any bits not specified here.

    Future IETF action may add additional bits without
    reving the version number.

    The following required fields are defined:
        ACCESS_DISPOSITION    = 0x00070000
            CREATE_NEW        = 0x00000000
            CREATE_OVERWRITE  = 0x00010000
            OPEN_EXISTING     = 0x00020000
            OPEN_OR_CREATE    = 0x00030000
            TRUNCATE_EXISTING = 0x00040000
        ACCESS_APPEND_DATA    = 0x00080000

    The following required fields are defined:
        ACCESS_SHARE_READ    = 0x00000001
        ACCESS_SHARE_WRITE   = 0x00000002
        ACCESS_SHARE_DELETE  = 0x00000004

    Several flags in the SSH_FILEXFER_ATTR_FLAGS
    in the open_attrs also control aspects of file
    creation; this flags have no affect during
    file open (to change these values for existing
    files, clients MUST use setstat or fsetstat)

     SSH_FILEXFER_ATTR_FLAGS_READONLY
         Advisory readonly bit should be set.  (This does not
         affect the current operation, only future operation.)

     SSH_FILEXFER_ATTR_FLAGS_SYSTEM
         The file should be marked as part of operating system.

     SSH_FILEXFER_ATTR_FLAGS_HIDDEN
         File should be hidden from the user.

     SSH_FILEXFER_ATTR_FLAGS_ARCHIVE
         The file should be included in backup / archive operations.

     SSH_FILEXFER_ATTR_FLAGS_ENCRYPTED
         The file should be encrypted.  If the filesystem doesn't support
         encryption, the operation must return UNSUPPORTED.

     SSH_FILEXFER_ATTR_FLAGS_COMPRESSED
         The file should be compressed.  If the filesystem doesn't support
         compression, the operation must return UNSUPPORTED.

     SSH_FILEXFER_ATTR_FLAGS_SPARSE_OPTION
     SSH_FILEXFER_ATTR_FLAGS_SPARSE_REQUIRED
         The file should be sparse.  If SPARSE_REQUIRED is specified, the operation
         MUST fail if it is not supported.

-----
    I've chosen the DISPOSITION value over the previous
    CREAT/TRUNC/EXCL bit mask because it:

    a. takes the same three bits
    b. removes the ability to specify poorly defined
       combinations (TRUNC|EXCL)
    c. I think it is easier to understand.
    d. Has room for expanision if we need it

What do people think?

Thanks,

Joseph

---

From: Jeffrey Hutzelman <[email-masked]>
Date: Mon, 08 Sep 2003 23:41:36 -0400

>     reving the version number.

This isn't really self-consistent.  Are bits in the 0xffff0000 group 
critical or not?  Given that new bits can be defined without bumping the 
version number, it's important that the behaviour of a server depend on 
whether the server supports a given flag, and not whether the server was 
implemented before or after a given flag was defined.

Given that you've indicated you want to be able to add flags without 
bumping the version, and that you've gone out of your way to indicate which 
potential future flags would be critical and which would not, I'd say you 
do want servers to reject critical flags of which they're unaware.

I'd suggest something like this:

The access_flags is two separate bit fields.  Bit values in the 0xffff0000 
mask are required, while bit values in the 0x000ffff mask are requested. 
Clients must not send any undefined bits in the access_flags field.

Bit values in the 0xffff0000 mask are required; if a flag in this range is 
set and cannot be supported by the server, the operation MUST fail.

Bit values in the 0x0000ffff mask are requested; if a flag in this range is 
set and its meaning is not known to the server, the operation MUST continue 
as if the flag had not been set.  If such a flag is set and its meaning is 
known to the server, but cannot be supported, the server MAY report an 
appropriate error or allow the operation to continue as if the flag had not 
been set.

Future IETF action may add additional bits without any need to change the 
protocol version number.

[ I think something slightly stronger is warranted -- the available space 
is small, and should probably only be used for "mainstream" revisions of 
the protocol and not private extensions. ]

>         ACCESS_APPEND_DATA    = 0x00080000

The meanings of these are more or less obvious, but they should be spelled 
out, I think.  In terms of traditional UNIX open flags, I think these mean:

CREATE_NEW = O_CREAT | O_EXCL
CREATE_OVERWRITE = O_CREAT | O_TRUNC
OPEN_EXISTING = 0
OPEN_OR_CREATE = O_CREAT
TRUNCATE_EXISTING = O_TRUNC

ACCESS_APPEND_DATA = O_APPEND

Is this correct?

Note that the semantics of ACCESS_APPEND_DATA are not entirely clear, 
especially when used with CREATE_NEW, CREATE_OVERWRITE, or 
TRUNCATE_EXISTNG.  Does this merely indicate that the file position pointer 
is positioned to the EOF when the file is opened, or does it mean that 
every write is performed atomically at the EOF, without changing the file 
position pointer and without any chance of conflict with another appending 
process?

If the former, then the flag has no effect in cases where the file will 
have 0 length after the open, and we shoulld drop it entirely and instead 
defined dispositions APPEND_EXISTING and APPEND_OR_CREATE.

If the latter, this needs to be spelled out, and it should be clear that if 
a server cannot actually support the atomic-append behaviour, it MUST fail 
the operation.  Of course, that leaves you with no means of getting a 
non-atomic append if that is sufficient.

>         ACCESS_SHARE_DELETE  = 0x00000004

The meanings of these are not obvious to me, probably because they're drawn 
from some Windows terminology with which I'm not familiar.

>     d. Has room for expanision if we need it

Yes; this sounds quite reasonable.  The interactions of O_CREAT, O_EXCL, 
and O_TRUNC will be familiar to UNIX folk, but not to others, and the 
combination of TRUNC and EXCL is indeed ill-defined.

-- Jeff

---

From: Joseph Galbraith <[email-masked]>
Date: Tue, 09 Sep 2003 09:28:56 -0600

> This isn't really self-consistent.

Arghh... of course, you are right.  I failed to go back and
rework that section.

> The access_flags is two separate bit fields.  Bit values in the 
> 0xffff0000 mask are required, while bit values in the 0x000ffff mask are 
> requested. Clients must not send any undefined bits in the access_flags 
> field.
> 
> Bit values in the 0xffff0000 mask are required; if a flag in this range 
> is set and cannot be supported by the server, the operation MUST fail.
> 
> Bit values in the 0x0000ffff mask are requested; if a flag in this range 
> is set and its meaning is not known to the server, the operation MUST 
> continue as if the flag had not been set.  If such a flag is set and its 
> meaning is known to the server, but cannot be supported, the server MAY 
> report an appropriate error or allow the operation to continue as if the 
> flag had not been set.
> 
> Future IETF action may add additional bits without any need to change 
> the protocol version number.

Excellent.

> [ I think something slightly stronger is warranted -- the available 
> space is small, and should probably only be used for "mainstream" 
> revisions of the protocol and not private extensions. ]

Yes, this sounds reasonable.

> The meanings of these are more or less obvious, but they should be 
> spelled out, I think.  In terms of traditional UNIX open flags, I think 
> these mean:
> 

Yes.  I'll spell it out in the draft.

> Note that the semantics of ACCESS_APPEND_DATA are not entirely clear
 > [...]

You raise a good point.  How about:

ACCESS_APPEND_DATA_ATOMIC = 0x00080000
ACCESS_APPEND_DATA        = 0x00100000

ACCESS_APPEND_DATA_ATOMIC
ACCESS_APPEND_DATA
     Data is always written at the end of the file.  The
     offset field of the SSH_FXP_WRITE requests are ignored.

     If the ATOMIC flag is used, then data must be written
     atomically to the end of the file without any chance of
     conflict with another appender.

     If the APPEND_DATA flag is used, atomic append is not
     required, but the server MAY still use atomic appends
     if it so chooses.

     Clients should balance the possibility that a server
     may not support atomic appends against the possibility
     of other writers conflicting.  If the file is not opened
     with SHARE_WRITE, there is no need for atomic appends.

> The meanings of these are not obvious to me, probably because they're 
> drawn from some Windows terminology with which I'm not familiar.

   ACCESS_SHARE_READ
       This flag indicates that client is willing to allow
       other readers of the file to simultaneously co-exist.

   ACCESS_SHARE_WRITE
       This flag indicates that client is willing to allow
       other writers of the file to simultaneously co-exist.

   ACCESS_SHARE_DELETE
       This flag indicates that client is willing to allow
       the file to be deleted while it is accessing it.

   For example, to display a log file that is being updated
   by a daemon process, the client would open the file for
   READ_DATA access, and specify that it was will to allow
   other writers (the daemon process) by specifying
   ACCESS_SHARE_WRITE.

   Previous versions of the protocol did not support these
   options, and so the behavior was implementation dependant.
   Clients that do not care about this, or wish to continue
   to leave the decision up to the server should specify
   all three sharing bits.

   If the client requests an exclusive access (does not
   set one of the share bits), but the server can not
   provide the such locking, it SHOULD fail the request.

   However, if the client is willing to share access, but
   the server does not support this kind of sharing, it
   SHOULD allow the request to succeed anyway.

Another alternative would be to reverse the fields (I'm
familiar with the windows varient, but remember a time
when I found them frustrating.)  Would people find the
following easier to parse:

   ACCESS_READ_LOCK
   ACCESS_WRITE_LOCK
   ACCESS_DELETE_LOCK
     The client wishes to be the lock out other reader, writers,
     writers or deleters.

     For example, the open will fail if another process has
     opened the file for WRITE access and the client requests
     a write lock.

     To display a log file being written to by a daemon process,
     a client would specify READ_DATA accesses, and would specify
     no locking.

     If the server can not provide the locking requested, it
     MUST fail the request.

Now that I've written it, I think maybe I prefer the LOCK variant.
What do other people think?

- Joseph

---

From: Derek Fawcus <[email-masked]>
Date: Thu, 25 Sep 2003 02:15:39 +0100

> following easier to parse:

Well if you want to add locking,  I prefer the above style.

But you also have the issue of mandatory vs advisory locks...

DF

---

From: Joseph Galbraith <[email-masked]>
Date: Thu, 25 Sep 2003 11:43:23 -0600

> Well if you want to add locking,  I prefer the above style.

Yes; I had settled on these.

> But you also have the issue of mandatory vs advisory locks...

I've heard these terms before, but I'm not sure I've had them
clearly defined for me before.  Let me see if I'm anyplace close
on what they mean :-)

   Mandatory Lock
   ==============
   Once I am granted a mandatory lock, I own it until
   I release it.  Others trying to access the file in
   a way that conflicts with my lock will receive an
   error.

   Advisory Lock
   =============
   Once I am granted an advisory lock, I own it until
   either I release it or the server notifies me that
   it is breaking my lock.  Others trying to access the
   file in a way that conflicts with my lock will result
   in the server breaking my lock.

Is this any place close to what these terms mean?

Thanks,

Joseph

---

From: Derek Fawcus <[email-masked]>
Date: Thu, 25 Sep 2003 18:54:44 +0100

> clearly defined for me before.  Let me see if I'm anyplace close

Well I was think more of the unix type advisory vs mandatory locks,
whereby with advisory locks,  locking is only implemented if requested,
i.e.

   process 1 opens the file,  requests a lock,  does i/o

   process 2 opens the file,  does i/o

if the kernel only implements advisory locks,  then process 2 can
(because it has no knowledge of the locking) interfere with the
i/o of process 1.  Furthermore process 1 never gets to find out.

So on a system that only implemented advisory locking,  having
a file accessed by SFTP (with locks) would not protect against
a local process manipulating the file.  This local process could
well be another SFTP instance,  or could be as simple as someone
truncating the file from a shell prompt with '> filename'.

Mandatory locks are as you describe - once locked other's can't
interfere.

DF