Subject: Re: FYI: I-D ACTION:draft-dusseault-http-patch-02.txt

From: Lisa Dusseault <[email-masked]>
Date: Wed, 7 Jul 2004 18:16:35 -0700

I'm putting together another draft before the July deadline, let's see 
if we can get closer to agreement this time.  I appreciate where the 
comments came with constructive suggestions; in those cases it made it 
much easier to deal with the comments.

On May 21, 2004, at 3:45 PM, Roy T. Fielding wrote:

>
>> http://www.ietf.org/internet-drafts/draft-dusseault-http-patch-02.txt
>
> I cannot support the following changes to the PATCH method, since
> they serve no useful purpose and add complexity to HTTP.
>
> The IM header field serves no other purpose than to duplicate
> what content-type is already defined to perform, and has the
> added complexity ...

Yeah, I had misgivings myself putting that into draft-02.  I'll see 
what it takes to register the MIME type for at least one of vcdiff or 
gdiff.

> There is no reason to require that the resource already exists
> (such a requirement is nonsensical in any case, since in HTTP the
> resource is the temporal invariance found in mapping of URI to content,
> not some file in storage).  The requirement should be that the PATCH
> media type define whether an existing representation is necessary
> or not, which is trivial for the server to discover when applying
> the patch (it is a conflict, like any other conflict).

Some patch formats don't allow for being applied to a "null file".  
Rather than allow the delta format to dictate whether this is allowed 
and choose an error in case it might not, I thought it made the spec 
simpler to say that PATCH cannot be used to apply a delta to a 
"previously unmapped URL".  This means that the error response can be 
depended on and server and client implementation ought to be simpler.  
It's a real corner-case to want to PATCH a new, empty resource, and it 
may easily be an unintended action on the part of the client -- the 
client may believe that the resource exists (yes, If-Match can be used 
to confirm this, but PATCH can make it easier).

If it's the wording that results in what you believe is nonsensical, is 
there some other wording?  At times the WebDAV group has used "unmapped 
URL" instead of "resource that does not exist".  Would that fix the 
problem?

>
> There is no reason to require that PATCH be subject to the same
> access controls as PUT.  The methods are orthogonal and implementers
> do not need to be constrained.  Yes, that is a logical implication
> for use of WebDAV ACLs, but that doesn't make it a requirement
> on all implementations.

I've amended this with "If the server supports WebDAV ACLs...".  It's a 
SHOULD requirement so that implementations can diverge if they need to. 
  Another option would be to say that servers MUST either make PATCH 
follow the same access controls as PUT, or they MUST define a new 
permission named 'patch'... but I don't see this really being used in 
practice.  The intent here is to give clients some idea whether to 
enable functions that use PATCH after querying the server to see what 
the current user's privileges are.  The explicit information on how to 
work with WebDAV ACLs  was suggested by Chris Sharp on his review of 
the -00 draft.

>
> "The PATCH request MUST only be used in a context which ensures
> that only one user may apply a patch at a time ..." assumes a
> particular application (content replication) that is not necessary
> as a requirement on the protocol. In particular, this requirement
> disallows the primary purpose of the original PATCH method, which
> was to support collaborations among multiple authors working on
> different parts of the same representation.  Yes, conflicts are
> possible in such situations, but they can be handled by the process
> of applying each PATCH and responding with appropriate error codes --
> they do not need to be legislated out of existence by overzealous
> standards.  In fact, most reliable differencing mechanisms include
> content verification via context or hashing; there is no need to
> require that the simple HTTP mechanisms of Etag and If-Match be used
> when the client is capable of making that choice on their own, and
> the server is capable of handling that choice based on the existing
> HTTP/1.1 protocol and orthogonal to the definition of PATCH.
> Requirements within the core protocol must be reserved for design
> decisions that are necessary for interoperability, not those that
> are dependent on a particular application of that protocol.

The intent of this language was to be extra safe about having the 
client apply the PATCH method to a known starting-point.  Some delta 
formats could corrupt the files if they are applied to the wrong 
version.  Rather than leave this up to the delta format defined, it 
seemed more consistent, and not too constraining, to make this 
constraint for all delta formats.

Despite what I believe to be the correctness of the intent, I could go 
along with weakening this -- however, I suspect not everybody in the 
mailing list would agree.  Julian Reschke suggested that the use of 
strong ETags (or locks) be a MUST to reduce chances of corrupting files 
by applying a delta to the wrong version.  Julian can confirm whether 
that indicates support for this restriction.

>
> The PATCH response should not be limited to 204 -- there are many
> applications that might want to know whether fuzzy matching was
> used to apply the patch.  The spec should define that the response
> indicates what the server did.  An improvement would be to include
> any metadata in the response that was either modified or might
> indicate modification, but that is better done within the body of
> the response rather than mixing response metadata with that of the
> patched representation.

It's not limited to 204, that's just the "basic success code".  Many 
methods describe one or more success codes without limiting the 
implementation from using other success codes.

I could make the language more like RFC2616 in an attempt to define 
what the server did:

	  A successful response SHOULD use the 204 No Content status code, and
	      this response indicates that the server successfully applied the 
delta.
	  Note that 200 OK is not used in this case because 200 OK implies a 
body in the
	  response, and 201 Created is not used because the resource must 
already
	  exist.

If that doesn't capture it better I'll need specific text suggestions.

>
> Use of DELTAV status reporting is completely unnecessary.  The HTTP
> status code is more than capable of indicating the reason for failure,
> and explicit indication of what parts of the PATCH could not be
> applied will have to be specific to the supported diff format in
> any case.  A more reasonable mechanism would be to define a
> corresponding error format for each differencing format, since
> understanding of that format will be necessary for a user agent
> to explain what changes (not what parts of the diff file) have
> been rejected.  We can't define a universal XML response because
> most diffs cannot be enclosed within XML without additional
> encoding (such as base64), nor can we assume that line numbers
> are meaningful to binary differencing formats, though I suppose we
> could define a uniform way of counting the number of delta commands
> per format and uniformly indicate errors by reference to those
> numbers.

It is not necessary, but I wouldn't describe it as completely 
unnecessary.  It should be easy enough for the server to implement and 
for the client to use or ignore.  I disagree strongly that it would be 
"more reasonable" to define a corresponding error format for each 
differencing format.  I feel that the error codes given will be broadly 
useful for a number of differencing formats.

>
> Servers SHOULD NOT support the VCDIFF format until its IP restrictions
> are clarified and made available royalty-free for all uses of HTTP,
> at a minimum, and not just use within HTTP/1.1 as defined in 2001.

OK by me.  Easiest just not to mention VCDIFF.

>
> There is no reason for a server to advertise support for this
> method via OPTIONS *.  This is a resource-specific implementation
> question and does not need any more support from OPTIONS than
> is already provided for all methods.  Clients should never be
> encouraged to make global requests on a server for information
> that is resource-specific; servers don't work that way and all
> it does is recommend that the client waste bandwidth.

That's now a SHOULD, not a MUST -- but from others, are there any 
objections if I remove the description of OPTIONS * entirely?  Or 
should I say that "Allow-Patch" never appears in OPTIONS * responses?

Lisa

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 8 Jul 2004 03:16:27 +0100

> Some patch formats don't allow for being applied to a "null file".  

Wait a minute.  All patch formats can be applied to a null file.

By definition: it's equivalent to applying the patch to an empty file.

If the server needs to run a program which won't operate on a
non-existent file, then it's up to the server to create an empty file
before running the program.  An implementation detail.

Now, if the patch format cannot do anything useful with an empty file
but report an error, fair enough.  But that is not unique to empty
files: some patch formats cannot do anything with 1-byte files, or
files which don't start with "ELF", or whatever their limitation.

An empty file should not be singled out in HTTP among all the possible
file contents which a particular patch format cannot affect.

If an empty file is not special, what is the justification for
treating a non-existent one any differently?

(The Etag is different, the metadata is different, but neither of
those have anything to do with patch formats).

> and choose an error in case it might not,

If the delta format can patch an empty file, the server can take care
of patching non-existent ones.  If the delta format can't patch an
empty file, then this "simplification" to the spec. doesn't succeed in
its aim: the patcher will still issue errors.

> to confirm this, but PATCH can make it easier).

It happens often with the real "patch" program.  But perhaps more
usefully, PATCH is providing a mechanism to upload changes in new and
interesting formats, and hopefully a way to determine when those
formats can be used.

Some of those interesting formats might be specialised for particular
types of content.  For example an XML patcher needn't use
byte-oriented deltas - and as well as patching it's effectively an
XML-specific compression format.  It would be silly to prevent formats
like that from being used to create non-existent resources from
scratch, especially if they're capable of patching empty resources to
get the same effect.  You'll end up with clients doing an empty PUT
followed by PATCH.

Also, currently what mechanism is there for discovering that you can
send a file compressed to be uncompressed at the server?  Yet, if you
could use PATCH after determining that a delta format is supported (I
presume there'll be a way to discover supposed delta formats), then a
compressed delta format is a great way to do that.  Again, you'll see
clients doing an empty PUT followed by PATCH.

So...  wouldn't the spec be even simpler _and_ more useful if it said
PATCH applied to a resource which doesn't exist first creates an empty
representation and then patches it?  And as advice to implementers: if
a patching program requires a file to exist, the server should create
an empty file before calling the patching program.

However, if you don't agree with my argument, there's another:

PATCH allows a client to patch a resource if the resource's version is
exactly the same as the client expects (using If-Match).

If the client expects the resource doesn't exist, then it'll have to
use PUT with If-None-Match: *.  But is that widely implemented?  It
seems risky to depend on it, because HTTP/1.0 offers PUT and ignores
If-None-Match.  I know there are servers which advertise HTTP/1.1 but
ignore If-None-Match.

The same mechanism which a client uses to discover PATCH is available
on a server is just as useful for atomically creating a resource as
for atomically patching a resource.

-- Jamie

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 9 Jul 2004 08:48:56 -0700

I don't have any principled objections to saying that PATCH formats 
MUST be applicable to empty files and the server SHOULD NOT fail the 
PATCH request simply because the Request-URI was previously unmapped.

The only situation I object to is if different delta formats work 
differently, so that some of them may be applied to unmapped URIs and 
some may not -- that seemed too complicated when we could so easily 
treat them all the same way.

What do others think -- should PATCH always be able to work on unmapped 
URIs, or never?

Lisa

---

From: Scott Lawrence <[email-masked]>
Date: Fri, 09 Jul 2004 11:56:46 -0400

> What do others think -- should PATCH always be able to work on unmapped 

Why not leave it to the server?  If the particular patch format requires
an existing resource and it's not there, the server can return a 4xx
error.

-- 
Scott Lawrence

---

From: Julian Reschke <[email-masked]>
Date: Fri, 09 Jul 2004 17:57:47 +0200

Lisa Dusseault wrote:

> 
> I don't have any principled objections to saying that PATCH formats MUST 
> be applicable to empty files and the server SHOULD NOT fail the PATCH 
> request simply because the Request-URI was previously unmapped.
> 

I think that PATCH should treat unmapped URLs the same way as empty 
files. Whether a patch format is applicable to empty files then is a 
completely independant concern.

Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 9 Jul 2004 17:29:50 +0100

> an existing resource and it's not there, the server can return a 4xx

I agree with this.  What a patch format can operate on needs to be
specified per patch format.  That leaves scope for
application-specific patch formats in future.

It makes sense to say an unmapped URI SHOULD be treated the same as an
empty representation if there is no patch-format-specific reason to do
otherwise.

By definition, a patch only applies correctly to a file with a certain
known pre-existing content.  So a client always knows when a patch
format can be be used, and when it can't, given known server
capabilities and known pre-existing representation.

Because of this it seems fine to allow particular patch formats to
only apply to certain representations.  For example, an XML or
executable patcher -- or one which only patches ZIP files.  Why not?
The client, when computing the patch, already knows if the format is
appropriate, so it's fine and not at all confusing to return a 4xx
error if a patch is applied to the wrong kind of representation --
just the same as if a patch is applied e.g. to the wrong size.

So it's inappropriate to say a patch format MUST apply to empty
representations.  Because: its inappropriate to say a patch format
MUST apply to all byte streams, or anything else; there's nothing
wrong with patch formats that apply to only a subset.  However if a
patch format does apply to empty representations, it makes sense that
the same format SHOULD be usable to patch a non-existent one into
existence.

Imho,
-- Jamie

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 9 Jul 2004 09:42:31 -0700

As you point out, not all deltas apply to all formats.  That makes it 
harder for the server to create an empty representation to patch.  How 
would it work for, say, GDIFF?  What kind of file would the server 
create?  Would it be a text/plain file or some other MIME type?  
Currently PATCH assumes that the client and the server already know the 
MIME type of the file being patched so there's no reason to send the 
MIME type to create the new resource.  Allowing an unmapped URL to be 
PATCHed would require the client to send the MIME type to create, 
wouldn't it?

Lisa

---

From: Scott Lawrence <[email-masked]>
Date: Fri, 09 Jul 2004 12:56:48 -0400

> create?  Would it be a text/plain file or some other MIME type?  

Ultimately, servers are going to have to choose a patch application
method based on the content-type of the PATCH content.  That method will
know what it can patch, and will have to indicate an error should it be
applied inappropriately - empty or non-existent resources are edge cases
that don't need to be discussed at all in the description of PATCH.

-- 
Scott Lawrence

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 9 Jul 2004 10:24:06 -0700

That could be reasonable for as-yet-undefined patch formats, but I'd 
like at least one patch format to work "out-of-the-box" with the PATCH 
proposal -- a single generic patch format that doesn't need an 
additional specification in order to know how to use it.  If that's 
gdiff (my current thought), then the spec would have to say what the 
server should do when receiving a PATCH request with a gdiff body to an 
unmapped URL.  Perhaps it's sufficient for the spec to say that for 
gdiff, this should fail, because it's not clear what MIME type resource 
to create.

Lisa

---

From: Scott Lawrence <[email-masked]>
Date: Fri, 09 Jul 2004 13:44:40 -0400

On Fri, 2004-07-09 at 13:24, Lisa Dusseault wrote:
> unmapped URL.  

Iff you feel that needs to be said, then I think that how to handle
non-existent resources is properly part of the MIME type registration of
the gdiff format, not part of the description of the PATCH method.

-- 
Scott Lawrence

---

From: Jeffrey Mogul <[email-masked]>
Date: Fri, 09 Jul 2004 11:44:34 -0700

I tend to think of "edge cases" not as details that could be left
out of the specification, but as test cases for whether the specification
makes sense.  I think we would normally prefer to see a simple
specification that adequately covers the edge cases (without special
treatment) rather than one that doesn't, or one that ends up with
layers of complexity to cover edge cases.

At the risk of sounding like a broken record, this seems to me
like a clear justification for separating "content-type" and
"instance-manipulation".  That is, in the PATCH request, use
IM to describe what kind of delta format (or range update, or
whatever) is being applied, and use Content-Type to indicate
the, umm, Content-type of the resource being updated.

In the normal case (the resource already exists), the Content-type
header of the request is either redundant, or (if you're paranoid) a
way of doing type-checking.  In the edge case (no resource yet),
the Content-type header tells the server what to create.  Seems
simple to me :-)

-Jeff

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 9 Jul 2004 12:03:41 -0700

That's certainly one way to model the interaction -- that the delta 
data is a manipulation of some instance.  However, I'm not sure yet why 
we can't model the patch as simply the entity->instance that the client 
wishes to send to the server.

If we were to use PUT, and model the IM header as indicating what the 
delta format in the body was and how to apply it to the resource 
already there, then I agree we'd have to model patching as an instance 
manipulation.  But re-using PUT seems a little scary and I haven't even 
had anybody suggest that.

Note that modeling the delta as an instance manipulation -- and using 
IM header -- would possibly create a dependency on RFC 3229.  But it's 
not really the same as 3229 because that spec doesn't define the IM 
header as a client request header -- so we'd also have to define what 
the server must do with the IM header and which instance manipulations 
it had to support.

Generally, the entity-instance-manipulation model isn't as clearly 
defined for client requests as it is for server responses -- nor should 
it be, and it may simply not be needed there as much.

Lisa

---

From: Justin Chapweske <[email-masked]>
Date: Fri, 09 Jul 2004 14:16:09 -0500

Well said.
   
> makes sense.

-- 
Justin Chapweske - Founder, Onion Networks
http://onionnetworks.com/
651-340-8787
 
Transfer large files 10x faster than FTP with Onion Networks' WAN
Transport(tm).  http://onionnetworks.com/products_wantransport.php

---

From: Roy T.Fielding <[email-masked]>
Date: Fri, 9 Jul 2004 12:38:36 -0700

On Friday, July 9, 2004, at 09:42  AM, Lisa Dusseault wrote:
> Currently PATCH assumes that the client and the server already know 
> the MIME type of the file being patched so there's no reason to send 
> the MIME type to create the new resource.  Allowing an unmapped URL to 
> be PATCHed would require the client to send the MIME type to create, 
> wouldn't it?

I don't know about GDIFF (it isn't a commonly used format). However,
an empty file has no media type on its own -- most likely it would
obtain the media type information either from the patch format itself
or via some out-of-band mechanism (such as the name under which the
URI is served).  Most servers actually define the media type according
to the binding used to retrieve it, rather than an aspect of the file
itself, and thus telling it the media type on a PUT or PATCH would
be for the sole purpose of allowing the server to reject the content
under that URI.

What is important for interoperability is that the server can indicate
to the client when an unexpected condition isn't met -- that way, the
standard doesn't have to define which condition is right for all
applications.  So, I would write that PATCH on an unbound URI may,
at the server's discretion, either result in an empty representation
being created prior to application of the patch or an error code from
the server indicating that the method is not allowed on an unbound URI.
I can envision situations in which I would want the server to respond
different ways for different resources on a single server, so it really
needs to be a resource-specific error response (4xx).  Likewise, the
client can require a pre-existing empty file simply by using the
If-Match header field, so the behavior can be restricted by the client
if so desired.

....Roy

---

From: Lisa Dusseault <[email-masked]>
Date: Fri, 9 Jul 2004 12:53:53 -0700

> the server indicating that the method is not allowed on an unbound URI.

I find that client implementations generally act in ways that they find 
the server can be predictable.  With what you describe, a client 
implementor can write this code

   Try PATCH on known empty resource
     trap error {
      send PUT
    }

Or:

   Send PUT on known empty resource

Gee, which is simpler... I've seen clients go through huge numbers of 
extra requests on all servers they talked to, because of some server 
that couldn't handle the more efficient case.  IOW, clients care more 
about limiting the branches in their code (maintainability) than 
performance, in many cases.

So the predictable thing for PATCH would be to either to have 
consistent and predictable behavior for the method with all patch 
types, OR at least to have consistent and predictable behavior for a 
given patch type.

Lisa

---

From: Jamie Lokier <[email-masked]>
Date: Fri, 9 Jul 2004 22:33:28 +0100

> types, OR at least to have consistent and predictable behavior for a 

Consistent and predictable behaviour for a given patch type make
sense, although there does need to be an error code for when a client
sends the wrong thing (mistake or bug).

Doesn't the client have to do some negotiating to determine the
available patch types, btw?  That'd be enough information that the
client shouldn't have to try different methods after that.

-- Jamie

---

From: Roy T.Fielding <[email-masked]>
Date: Fri, 9 Jul 2004 15:33:37 -0700

On Friday, July 9, 2004, at 12:53  PM, Lisa Dusseault wrote:
> I find that client implementations generally act in ways that they 
> find the server can be predictable.  With what you describe, a client 
> Gee, which is simpler...

That depends on the application.  There is nothing preventing the client
from making that decision on its own, so there is no reason for the
protocol to standardize one over the other.  Likewise, some servers
may choose to allow PATCH but not PUT (for resources that only allow
a certain type of modification, for example).  The protocol does not
need to be restricted to the common case.

> types, OR at least to have consistent and predictable behavior for a 

You can't require that a server accepts a request.  Whether it is PATCH
or PUT, empty or existing, the only thing the protocol can do is state
the syntax and meaning of the messages.  A reasonable thing to specify
is that if a server is willing to accept a PATCH on a URI that has no
current representations, then its behavior must be consistent with the
patch being applied to an empty representation.

The client can choose which method to use on its own, based on its own
environment and its own constraints.

....Roy

---

From: Jeffrey Mogul <[email-masked]>
Date: Fri, 09 Jul 2004 17:51:17 -0700

I don't think RFC3229 in any way prohibits the use of IM headers
on requests.  In fact, it says:

         Nothing in this specification specifically precludes the use of
         a delta encoding for the body of a PUT request.  However, no
         mechanism currently exists for the client to discover if the
         server can interpret such messages, and so we do not attempt to
         specify how they might be used.

However, there's no reason why you couldn't define a new header
name to carry instance-manipulation values in a request ("R-IM",
for example), and then define whatever semantics you need for it.
The specification for IM in RFC3229 is only a few paragraphs
(plus some examples) so this is not a big deal.

I would think, in any case, that no matter what mechanism you
choose, you either need a way for the client to discover what
delta-encoding formats the server supports (analogous to
Accept-range, but please don't create another Accept-*
response header name!) or for the server to gracefully recover
from a client that tries a format it doesn't grok.  That is,
I think this is orthogonal to whether you use something like
the RFC3229 approach.

You might still decide, for other reasons, not to use the
instance/instance-manipulation approach, but don't let RFC3229
get in your way if you decide in favor of it.

-Jeff

P.S.: RFC3229 is a Proposed Standard.  So it can be changed,
if you think that's the right approach.

---

From: Lisa Dusseault <[email-masked]>
Date: Mon, 12 Jul 2004 12:05:17 -0700

On Jul 9, 2004, at 12:38 PM, Roy T.Fielding wrote:

> itself, ...

This may not now be relevant to the PATCH discussion, but I wanted to
check my understanding here.  Here's the statement I would have made:

   "Most servers have nearly complete independence between the
   media type of a resource and the binding used to retrieve it. 
Interactions
   between the two are mostly limited to resource creation time when if 
the
   client erroneously omits the Content-Type header, or if the content is
   importedthrough a mechanism other than HTTP, the server may have to
   guess a default MIME type to assign and may use the binding 
(particularly
   the filename extension) to make a good guess."

Obviously this is nearly the opposite of what you said Roy.  When you 
said
"according to the binding" what part of the binding did you mean? The
namespace or the file extension, or other?  If by namespace, do you have
an example of this?

I've been involved in several HTTP/WebDAV server implementations and
we always implemented the MIME type and binding(s) as independent bits
of resource data which both needed to be stored.  Xythos' 
interoperability
experience supported that choice.  When we worked with clients uploading
files without providing a MIME type (or providing a generic MIME type 
as the
Mac OS X client sometimes does) we found interoperability problems as
other clients relied on the MIME type, not the file extension.  And the 
server
couldn't always make a good guess based on the extension, and certainly
had no help based on any other part of the binding as this was in a file
sharing or document management environment where the namespace was
almost all arbitrary.

Lisa