Subject: PATCH vs multipart/byteranges vs Content-Range

From: Julian Reschke <[email-masked]>
Date: Sat, 16 Feb 2008 12:30:15 +0100

Hi,

I was in the process of putting together an example of using 
multipart/byteranges as a payload for the PATCH method when I realized 
that RFC2616 requires that format to at least contain two parts (partial 
responses in RFC2616 do not need multipart when there's only a single part).

So, to overwrite a single range of bytes, one would need a dummy (empty) 
part, unless...

...it's ok to use Content-Range as a request header.

Looking at 
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.16>:

"The Content-Range entity-header is sent with a partial entity-body to 
specify where in the full entity-body the partial body should be applied."

So, to append a line of text to a representation with a current 
Content-Length of 1234 bytes, would it be ok to use

---
PATCH /test.txt HTTP/1.1
Host: example.com
Content-Type: text/plain
Content-Range: 1234-1245/1234
Content-Length: 12

0123456789
---

(where the line to be appended ends in CRLF, thus counting 12 bytes).

Questions:

1) Would it be OK to use Content-Range on PATCH (understanding people 
previously were reluctant to use it on PUT because servers may ignore 
it, thus truncate a resource)?

2) Should instance-length be the length *before* or *after* applying the 
  request (I'd assume *before*). Or should it just use "*"? (*)

3) Should we mention this in the PATCH spec; minimally requiring servers 
to either reject requests with Content-Range headers, or to process the 
payload as a partial replace?

BR, Julian

(*) This would not be a problem for any partial update that doesn't 
change the length.

---

From: "Brian Smith" <[email-masked]>
Date: Sat, 16 Feb 2008 08:06:34 -0800

Julian Reschke wrote:
>
> So, to overwrite a single range of bytes, one would need a dummy
(empty) part, unless...
> ...it's ok to use Content-Range as a request header.

Content-Range is an entity header, not a request header. As such, it
describes the entity enclosed in the message, not anything at the
Request URI.

> "The Content-Range entity-header is sent with a partial 
> entity-body to specify where in the full entity-body the 
> partial body should be applied."

> So, to append a line of text to a representation with a 
> current Content-Length of 1234 bytes, would it be ok to use
> 
> ---
> PATCH /test.txt HTTP/1.1
> Host: example.com
> Content-Type: text/plain

This means "I am sending 12 bytes of a 1234 byte patch, starting at the
end and ranging to an offset that is past the end." In PATCH, the
entity-body is the patch, so a "partial entity-body" means you are only
sending part of a patch. The server is likely to reject such a request
because (a) text/plain isn't a MIME type for a patch format, and (b) it
still needs the other 1234 bytes of the patch to know what to do.

If it was a PUT, it would mean "I am sending 12 bytes of the 1234 byte
entity, starting at the end and continuing 12 bytes past the end," which
seems to be what you are trying to do. However, even this has a
presumption that the first 1234 bytes would have been the same as what
the server already has, and that presumption is not really justified by
the definition of Content-Range. In other words, when you issue a range
request in a GET, you are saying "I only care about a specific part of
the response," which is different than "We agree on the content outside
the range; just send me the part we don't agree on yet." 

> 1) Would it be OK to use Content-Range on PATCH 
> (understanding people previously were reluctant to use it on 
> PUT because servers may ignore it, thus truncate a resource)?

RFC 2616 doesn't define semantics for Content-Range except when it is
used in a response. The specification says that the server must not
ignore Content-Range when it receives it in a request, but it doesn't
say what it should do with it. Effectively, a server should reject all
requests with a Content-Range header unless a protocol layered on top of
HTTP is being used, which defines semantics of such requests. Perhaps
HTTPbis can define semantics for Content-Range in requests but it would
have to be completely optional to be compatible with RFC 2616.

>   request (I'd assume *before*). Or should it just use "*"? (*)

instance-length should be the total length of the complete patch, or
"*".

> 3) Should we mention this in the PATCH spec; minimally 
> requiring servers to either reject requests with 
> Content-Range headers

Servers are already required by RFC 2616 to reject any request with a
Content-Range header unless it knows what to do with it. 

> or to process the payload as a partial replace?

You would need a new header, that works like Content-Range, but applies
to the resource at the request URI instead of the entity body. But, even
if you had such a header, you have no way of choosing which
representation(s) of the resource it applies to. That is problematic
because byte-ranges are very representation-dependent; a byte-range for
a compressed variant is going to be quite different than the byte range
for the uncompressed variant.

By the way, I really like the idea of a patch format based on
multipart/byte-ranges; I even think that such a format should be a
SHOULD requirement for servers that implement PATCH.

- Brian

---

From: "Brian Smith" <[email-masked]>
Date: Sat, 16 Feb 2008 08:13:08 -0800

Brian Smith wrote:
> > 3) Should we mention this in the PATCH spec; minimally requiring 
> > servers to either reject requests with Content-Range headers
> 
> Servers are already required by RFC 2616 to reject any 
> request with a Content-Range header unless it knows what to 
> do with it. 

Actually, this is wishful thinking on my part. HTTPbis should have that
requirement that Content-* cannot be ignored for any method that uses a
request body (especially POST), but RFC 2616 only has the requirement
for PUT. Can a new issue be created for that?

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 16 Feb 2008 17:18:22 +0100

Brian Smith wrote:
 > ...

...that's why it is common wisdom not no apply write operations to 
content-negotiated resources.

> for the uncompressed variant.

That's correct, but it applies to any kind of write operation, such as 
PUT or PATCH.

> SHOULD requirement for servers that implement PATCH.

It's certainly a candidate.

It just would be nice if it would allow one-part entities. Maybe we 
should fix this to make it more useful for PATCH?

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 16 Feb 2008 17:22:17 +0100

> for PUT. Can a new issue be created for that?

Why would that have anything to do with the presence of a request body?

BR, Julian

---

From: "Brian Smith" <[email-masked]>
Date: Sat, 16 Feb 2008 08:32:05 -0800

Julian Reschke wrote:
> Brian Smith wrote:
> > Actually, this is wishful thinking on my part. HTTPbis should have 
> > that requirement that Content-* cannot be ignored for any 
> > method that uses a request body (especially POST), but RFC 2616
> > only has the requirement for PUT. Can a new issue be created for
> > that?
> 

A server processing a GET or DELETE can ignore all Content-* headers in
the request, because the the request body doesn't have any semantics for
GET and DELETE, and Content-* describe the request's entity.

POST, PUT, and PATCH cannot ignore Content-* headers in the request
because those Content-* headers are needed to understand the request
body enough to process it--especially Content-Range and
Content-Encoding.

Another issue is "What does Content-*" mean? Does it mean all headers
prefixed with "Content-", or just those headers prefixed by "Content-"
that are defined in RFC 2616?

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Sat, 16 Feb 2008 17:40:14 +0100

> GET and DELETE, and Content-* describe the request's entity.

Ah. Was confusing Content-* with Accept-O. Oh well.

> body enough to process it--especially Content-Range and

Yep.

> that are defined in RFC 2616?

I have raised that question some time ago in the context of PATCH, and I 
think the answer was: all matching that pattern. I do agree we should 
clarify that. (Mark?)

BR, Julian

---

From: "Subbu Allamaraju" <[email-masked]>
Date: Sat, 16 Feb 2008 10:53:25 -0800

> because (a) text/plain isn't a MIME type for a patch format, and (b) it

You mean, text/plain is invalid for bytes? Why would text/plan be invalid
for a patch format?

> SHOULD requirement for servers that implement PATCH.

Really? As you pointed out above, Content-Range is not specified for
requests. Secondly, IMHO, patch format does not need to be constrained to
byte-ranges.

Regards,
Subbu

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 16 Feb 2008 11:58:43 -0800

On Feb 16, 2008, at 3:30 AM, Julian Reschke wrote:
> multipart/byteranges as a payload for the PATCH method when I  
> realized that RFC2616 requires that format to at least contain two  
> parts (partial responses in RFC2616 do not need multipart when  
> there's only a single part).

I would not recommend using any of the byte range features for PATCH.
IMO, byte ranges need a careful review of existing implementations
and removal of those features that make denial-of-service a problem.
That may include deprecation of multipart/byteranges.

PATCH is a very simple method -- look at the content-type to see the
format of the body, which contains instructions on how to apply a
desired state change.  The intent (when I added it back in 1995) was
to define patch media types that contained a first section containing
changes to metadata and then later sections containing the diffs.
The decision on which patch media type to support would largely
depend on the nature of the resource (and the ability of folks who
wanted them to get off their duff and define the types).  I started
on a definition of a patch format for YAML late last year, but it
was derailed by the partitioning work.

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Sun, 17 Feb 2008 09:21:11 +1300

Hi

Just thinking about the structure this PATCH command is taking.  From my 
understanding the functional requirements include

* ability to create a new entity on the server based on a previous 
entity patching in 1+ difference ranges.
* transactional - if multiple ranges are specified in the patch, they 
must all be done or none of them
* must only patch resource if the entity is the same on the server as 
was the base of the derivation of the patch ranges on the client.
* must go through proxies.

I'm finding it difficult to see the overall structure - what a request 
may look like in total, but it appears as if all the byte-ranges for 
replacement are intended to be supplied in a single Content-Range header 
in the request message.

If that's not the case, then please ignore what follows, perhaps 
otherwise consider the following.

Since multipart messages allow headers per part, put the headers 
relating to each individual patch action into the part headers.  The 
message headers can then be greatly simplified, allowing much easier 
proxy traversal.

For instance (please excuse improper syntax of boundaries, but you 
should get the idea)

PATCH SomeURI HTTP/1.1
Content-Type: multipart/patchspec; boundary="someboundary"
If-Match: "someEtag"
Content-Length: <length of entire set of patch commands>

--someboundary--
Patch-Method: replace; range=0-100

... // data

--someboundary--
Patch-Method: replace; range=200-300

... // data

--someboundary--
Patch-Method: truncate; from=400

etc.

this would replace the sections 0-100, and 200-300 and then truncate the 
file from 400.

now, yes of course this is effectively layering a higher level protocol 
(PATCH) on top of HTTP.  I don't see this as a problem since, a) PATCH 
looks to be new anyway, and b) it's much easier for a proxy to deal with 
this without having to know about byte ranges.  Many proxied deprecate 
byte ranges in order to enforce data scanning of the entire entity.  For 
instance if you're scanning for virii at a proxy, you can't allow people 
to retrieve a file in byte ranges, since you can't scan it, so the 
common result is to remove Accept-Ranges from all server responses, and 
remove range tags from requests.

Julian Reschke wrote:
> that RFC2616 requires that format to at least contain two parts 
> (partial responses in RFC2616 do not need multipart when there's only 
> a single part).
>
> 2) Should instance-length be the length *before* or *after* applying 
> the  request (I'd assume *before*). Or should it just use "*"? (*)
>
> servers to either reject requests with Content-Range headers, or to 
> process the payload as a partial replace?
>
> BR, Julian
>

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Robert Sayre <[email-masked]>
Date: Sat, 16 Feb 2008 15:53:11 -0500

On Feb 16, 2008, at 2:58 PM, Roy T. Fielding wrote:
> desired state change.

Yes, and it needs to be distinct from POST so you can send the server  
a patch that isn't to be applied, but stored at a URI of the server's  
choosing.

>  I started

Here's a patch format for JSON in JSON:
http://blog.mozilla.com/rob-sayre/2008/02/15/restful-partial-updates/

- Rob

---

From: "Brian Smith" <[email-masked]>
Date: Sat, 16 Feb 2008 13:18:11 -0800

Subbu Allamaraju wrote:
> You mean, text/plain is invalid for bytes? Why
> would text/plan be invalid for a patch format?

text/plain isn't necessarily invalid, but in order to use that MIME
type, the server would have to parse the request entity just to
determine the format of the patch: is it a Darcs format patch? diff -u
format? Git? something else?

> Really? As you pointed out above, Content-Range is not specified
> for requests. Secondly, IMHO, patch format does not need to be
> constrained to byte-ranges.

By "patch format based on multipart/byteranges", I didn't mean to use
multipart/byteranges as-is. As Roy pointed out, any kind of standardized
patch format--one that is recommended for servers to support--should
have a syntax for updating headers along with the actual entity. And,
any such general-purpose patch format would have to have a way of
representing insertions and deletions, which multipart/byteranges cannot
do as it is defined today.

- Brian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Sat, 16 Feb 2008 13:43:35 -0800

> http://blog.mozilla.com/rob-sayre/2008/02/15/restful-partial-updates/

Damn, I knew I should have finished my blog first ... a blog wave has
started and I'm still getting into my trunks.

A bit of background: At RailsConf EU, I asked DHH what it was that
Rails needed most from HTTP.next and the answer was partial updates
exactly as you describe.  My answer was to use PATCH, of course, but
the standards issue is still that there are no registered diff formats.
So, I said to myself, I know how to fix that ...

YAML is a better format for contextual diffs.  I think your json.sync
format is close, but way too verbose (we are supposed to be saving
bandwidth here, not being human-readable).  Maybe I should start a
wiki for diff formats and just start registering them.

....Roy

---

From: James M Snell <[email-masked]>
Date: Sat, 16 Feb 2008 14:45:16 -0800

Having some place where the various diff formats can be collected would 
definitely be a good thing.

- James

Roy T. Fielding wrote:
> [snip]

---

From: Robert Sayre <[email-masked]>
Date: Sat, 16 Feb 2008 18:39:05 -0500

On Feb 16, 2008, at 4:43 PM, Roy T. Fielding wrote:
> I think your json.sync
> bandwidth here, not being human-readable).

Well, it's tricky, because a conflict will not save bandwidth, so it's  
best to include enough information to avoid as many as possible. Users  
don't like them anyway. I agree it's a bit of a failure for changes in  
long strings of text. Worse is better, or something. A nice extension  
might be a "value" key with an object field containing text offsets.

Regarding the key names and things, I'm clearly not trying hard. But I  
think JSON-as-patch-format has advantages that make it desirable. Easy  
to sniff character set, ascii escapes for control characters  
(mandatory in Mozilla's JSON encoder, and Crockford's), optional ascii  
escapes for unicode, and Web ubiquity. It's possible to implement in  
JS, for older browsers and browsers that lag intentionally. Bandwidth  
can be further crunched by gzip or macros in the transfer protocol  
(nudge, nudge).

> Maybe I should start a

Sounds good. FWIW, there are still a few edge cases left to deal with  
in the JSON sync thing, and I don't have time to work on them. I would  
also like to sit down and prove it, like Ramsey did, but that's going  
to be tedious and time-consuming. I won't get to it until summer. I  
can document this in an I-D if other people are willing to help me  
take it to the finish line.

- Rob

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 28 Feb 2008 11:26:22 +1100

Now <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/102>.

Also, looking at the text for PUT:

> The recipient of the entity MUST NOT ignore any Content-* (e.g.  
> Content-Range) headers that it does not understand or implement and  
> MUST return a 501 (Not Implemented) response in such cases.

I wonder if 'recipient' is the right term to use; what does it mean  
for an intermediary to understand Content-*?

On 17/02/2008, at 3:13 AM, Brian Smith wrote:

> that

--
Mark Nottingham     http://www.mnot.net/

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 28 Feb 2008 11:29:06 +1100

Now <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/103>.

I very nearly made this an editorial issue, but I thought that it  
would be good to get general agreement that Content-* is *not*  
constrained to just those headers defined by RFC2616. Anyone have a  
problem with that?

---

From: "Brian Smith" <[email-masked]>
Date: Wed, 27 Feb 2008 18:05:32 -0800

Mark Nottingham wrote:
> I very nearly made this an editorial issue, but I thought 
> that it would be good to get general agreement that Content-* 
> is *not* constrained to just those headers defined by 
> RFC2616. Anyone have a problem with that?

1. Anybody can create a new must-understand header by naming it
"Content-<something>." Is this desirable?

2. We cannot create must-understand headers without using that naming
scheme. It seems odd.

3. Are headers prefixed with Content-* always entity headers?

- Brian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 28 Feb 2008 16:15:21 +1100

That's an interesting way to state the problem; I hadn't thought about  
it in terms of mU, but of course that's what it is currently for PUT  
(and PUT only, at this point, although we have a companion issue for  
that).

Certainly, if this pair of issues were resolved in the most general  
fashion, it would make Content-* look like HTTP's mU mechanism (at  
least for requests with a body), and knowing how some people out there  
think (e.g., the WS crowd), this could easily be abused.

I think, however, that the key here is knowing what 'understand'  
means. For PUT, it pretty much means "store it and regurgitate it",  
with the additional caveat that if the server does any additional  
processing, it had better know the implications of things like Content- 
Type and Content-Encoding.

OTOH, POST falls into that latter category completely; all it does is  
processing. Even then, I'm not sure what "understand" really means,  
since a POST processor may not care a sausage for what Content- 
Language or even Content-Type means, depending on what it plans to do  
with the request.

See also <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/79>.

My .02 is that Content-* pretty clearly means all headers that have a  
Content- prefix, not just those defined by RFC2616. However, it  
doesn't follow that the "understand" rule for PUT should be  
automatically copied to POST and other request methods with bodies;  
and, I have a suspicion we'll be rewriting that language as part of  
#79 anyway.

---

From: "Brian Smith" <[email-masked]>
Date: Thu, 28 Feb 2008 05:07:24 -0800

Mark Nottingham wrote:
> means. For PUT, it pretty much means "store it and 
> regurgitate it", with the additional caveat that if the 
> server does any additional processing, it had better know the 
> implications of things like Content- Type and Content-Encoding.
> 
> OTOH, POST falls into that latter category completely; all it 
> does is processing. Even then, I'm not sure what "understand" 
> really means, since a POST processor may not care a sausage 
> for what Content- Language or even Content-Type means, 
> depending on what it plans to do with the request.

I think the intent is that headers like Content-MD5 (recalculate MD5 if
you encode/decode using Content-Encoding) and Content-Disposition (check
for path problems) need special processing when you "regurgitate" them.
Also, it usually doesn't make sense for the server to accept a request
with Content-Language: ko and then return the same entity with
Content-Language: jp or vice versa. But, it might make sense for it to
just not include a Content-Language header, which is what a lot of
servers do.

The requirement is relevant to POST and any other method. Perhaps the
requirement should be written as "Clients and origin servers SHOULD
ignore headers they don't understand and SHOULD NOT send headers they
don't understand (do not 'regurgitate' anything)", and moved to some
place that describes how headers should be processed generally. This
requirement is definitely not specific to PUT, considering that it often
applies to POST (whenever it is used as "PUT at server-assigned
location"), and will probably apply to PATCH too. And, it applies to
clients as well as servers; a client shouldn't send a header it receives
from a server unless it understands that header.

> Content- prefix, not just those defined by RFC2616. 

If you accept the above suggestion, then the special treatment of the
"Content-" prefix is no longer needed, so the "Content-*" issue will be
moot.

- Brian

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 28 Feb 2008 21:45:09 +0100

Brian Smith wrote:
 
> headers like Content-MD5 (recalculate MD5 if you 
> encode/decode using Content-Encoding) 
[...]
> need special processing when you "regurgitate" them.

What "special processing" apart from what is stated in
RFC 1864 do you have in mind ?

| To generate the value of the Content-MD5 field, the MD5 
| algorithm is computed on the canonical form of the MIME
| entity's object.  In particular, this means that the 
| sender applies the MD5 algorithm on the data immediately
| after conversion to canonical form, before applying any 
| content-transfer-encoding, and that the receiver also 
| applies the MD5 algorithm on the canonical form, after
| undoing any content-transfer-encoding.

For any questions about which Content-* header fields are
relevant for HTTP I'd check the header field registries:
http://www.iana.org/assignments/message-headers/perm-headers.html
http://www.iana.org/assignments/message-headers/prov-headers.html

2616bis could add a normative reference to RFC 3864, and
maybe get rid of long boring lists covered by RFC 4229 (?)

 Frank

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 29 Feb 2008 07:58:42 +1100

FYI:
   http://tools.ietf.org/wg/httpbis/trac/ticket/40

---

From: Robert Sayre <[email-masked]>
Date: Thu, 28 Feb 2008 16:09:57 -0500

On Feb 28, 2008, at 12:15 AM, Mark Nottingham wrote:

> My .02 is that Content-* pretty clearly means all headers that have  
> a Content- prefix, not just those defined by RFC2616. However, it  

Content-* is in RFC 2045. RFC 2616 has a reference to RFC 2045 in  
section 1.1, and in a long appendix covering differences from MIME.

- Rob

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 29 Feb 2008 10:22:31 +1100

That's a pretty big change, considering the previous text;
> MUST return a 501 (Not Implemented) response in such cases.
Doing away with this would likely cause problems for existing  
implementations (and WebDAV?).

For non-PUT methods, the default is already to ignore the headers you  
don't understand. Regurgitation for the case of POST is an application- 
specific thing, and I'm not sure what the spec can really say about  
that; it's more implementation advice than HTTP spec language.

Other methods that may have "regurgitation" implications -- e.g.,  
PATCH -- will have to craft their own language like POST's.

So, I'm -1 on i102 (which is what this discussion is really about).

On 29/02/2008, at 12:07 AM, Brian Smith wrote:

> Perhaps the