Subject: Sftp rename: 2 proposals

From: Joseph Galbraith <[email-masked]>
Date: Fri, 05 Sep 2003 11:04:19 -0600

I made this proposal, but after not being able to find it, I realized
I screwed up sending it and it didn't ever make it to the list:

All right, how about the following change:

    Files (and directories) can be renamed using the SSH_FXP_RENAME
    message.  Its data is as follows:

         uint32     id
         string     oldpath [UTF-8]
    	string     newpath [UTF-8]
         uint32     flags

    where `id' is the request identifier, `oldpath' is the name of an
    existing file or directory, and `newpath' is the new name for the
    file or directory.

    flags is 0 or a combination of

      SSH_FXP_RENAME_OVERWRITE  0x00000001
      SSH_FXP_RENAME_ATOMIC     0x00000002
      SSH_FXP_RENAME_NATIVE     0x00000004
      SSH_FXP_RENAME_COPY_OK    0x00000008

    If flags does not include SSH_FXP_RENAME_OVERWRITE,
    and there already exists a file with the name
    specified by newpath, the server MUST respond
    with SSH_FX_FILE_ALREADY_EXISTS.

    If flags includes SSH_FXP_RENAME_ATOMIC, and
    the destination file already exists, it is
    replaced in an atomic fashion.  I.e., there
    is no observable instance in time where the
    destination file does not exist.  SSH_FXP_RENAME_ATOMIC
    implies SSH_FXP_RENAME_OVERWRITE.

    There may, however, be some span of time where
    both the source and destination file exist.
    SSH_FXP_RENAME_ATOMIC has no bearing on the
    source file.

    If flags includes SSH_FXP_RENAME_ATOMIC and the
    server cannot replace the destination in an
    atomic fashion, then the server MUST respond
    with SSH_FX_OP_UNSUPPORTED.

    Because some servers cannot provide atomic rename,
    clients should only specify atomic rename if correct
    operation requires it.  If SSH_FXP_RENAME_OVERWRITE
    is specified, the server MAY preform an atomic
    rename even if it is not requested.

    If flags includes SSH_FXP_RENAME_NATIVE, the server
    is free to do the rename operation in whatever
    fashion it deems appropriate.  Other flag values
    are considered hints as to desired behavior, but
    not requirements.

    If flags includes SSH_FXP_RENAME_COPY_OK, the
    server MAY preform a copy / delete operation
    in order to complete the operation.  If an atomic
    rename is also requested, the server may only
    preform this sequence if it can do so in an
    atomic fashion for the destination file.

    The server may fail rename requests in other
    situations, for example if `oldpath' and `newpath'
    point to different file systems on the server.

    The server will respond to this request with a
    SSH_FXP_STATUS message.

The other proposal is this (from Martin Pool, [email-masked]):

 > I think the fact that SSH_FXP_RENAME is impossible to reliably
 > implemented on Unix is a serious problem.  Secondarily it is a shame
 > that it prevents use of the atomic-rename Unix idiom.
 >
 > I would like to suggest that the phrasing be changed to
 >
 >   The server MAY return SSH_FX_FILE_ALREADY_EXISTS if there exists a
 >   file with the name specified by newpath.
 >
 > Clients which wish to always replace the path without knowing the
 > server filesystem can precede the rename operation with an attempt to
 > delete the destination file.  That does not seem to be an undue
 > burden.
 >
 > Clients which know the server is Unix and which wish to take advantage
 > of atomic replace can do so.

I believe this proposal has a couple of issues:

o It is not possible to know if the server is unix (and I
   think, in general, it is a bad idea to make decisions
   based on OS type.)

o If the client does _not_ wish to have the file replaced,
   it must stat the file and wait for the response before
   preceeding.

o As mentioned above, if the client wishes the file to
   always be overwritten, it must delete the destination,
   and then do the rename, but even that may _still_ fail
   if something comes along and recreates the file in the
   interim.  (Imagine a system where the file must exist,
   and so there is a monitor program which recreates the
   file if it is deleted.  Farfetched?  Maybe.  Possible?
   Definitely.)

I think it is far better for the client to be able to specify
the behavior they want on the rename.  Most clients are probably
happy with whatever the server does natively.

-- 

I believe both of these proposals require sftp v5; mine
certainly does, but Martin's changes the semantics of
an existing operation sufficiently that I believe
we should change the version number.

- Joseph

---

From: [email-masked] (Martin Pool)
Date: Sun, 7 Sep 2003 10:56:31 +1000

>    existing file or directory, and `newpath' is the new name for the

I would be happy to see this proposal accepted rather than mine.  I do
think it is important the problem be addressed in some way.

>    implies SSH_FXP_RENAME_OVERWRITE.

s/instance/instant/?

Perhaps rather than "does not exist" say something like "where the
name does not refer to either the old or the new file", as the unix
rename operation is described.  Having a moment where the file appears
empty or incomplete would be wrong, but this might happen for a
copy/delete operation.

Perhaps it should mention that "observable" includes other processes
or SSH clients, not just this client.

>    SSH_FXP_RENAME_ATOMIC has no bearing on the

I'm not sure about this last sentence.  If you just mean that both
files may exist for a moment, it is redundant with the previous
sentence.

>    atomic fashion for the destination file.

s/preform/perform/

I know what copy/delete is, but I don't understand the meaning or
intention of this flag.  Do you mean that e.g. cross-filesystem moves
would be allowed only if this flag was specified, and they would fail
otherwise?  

So presumably most clients would turn SSH_FXP_RENAME_COPY_OK on?  Who
would turn it off?  Clients worried about the time it would take to
copy the file?

>    The server will respond to this request with a

[on my proposal]

>   think, in general, it is a bad idea to make decisions

I wasn't suggesting that the client ought to send "are you unix?".
However, if a human is using sftp interactively they *may* know what
the remote filesystem is, and want to take advantage of that.  

>   it must stat the file and wait for the response before

That is a drawback.  And there is a small race here as well: some
other process might create a new file in the interim.

>   file if it is deleted.  Farfetched?  Maybe.  Possible?

But this problem also exists on server systems where the native rename
does not overwrite.  Rename-and-replace on Windows (I think) has to be
implemented by deleting the file and then renaming.  So there is still
a race window, although since it does not include a network round trip
it is smaller.

> happy with whatever the server does natively.

Yes, I agree.  I like that your proposal makes the "don't care" value
obvious.

-- 
Martin