Subject: Resumable Uploads

From: Felix Geisendörfer <[email-masked]>
Date: Thu, 18 Apr 2013 13:37:51 +0000

Hi,

I'm interested in finding out how to perform resumable uploads over http
while being compliant with existing specifications. The result of this work
will be shared with the community to create interopable server/client
software to simplify file uploading on the web.

The first draft for this can be seen here [1]. I'm new to protocol
authorship, so please forgive any ignorance you may see in my work. This is
no attempt at competing with existing standards, I merely want to identify
the relevant subsets of http and make them easy for people to implement.

Anyway, here are the things I'm wondering about specifically:

- How should a http client transfer a partial byte range of a given
resource? Specifically:
- Should the PUT or PATCH method be used? (neither seem fully appropiate
according to my research)
- What header should be used to indicate the transfered range for the
upload? Julian Reschke pointed out that Content-Range may not be appropiate
[2].
- What header should be used to indicate the received range of an
interrupted upload?

Ideally clients should be able to transfer multiple parts of a file using
parallel requests (to fully utilize long fat pipes). This will require
servers to indicate which parts of a file have previously been received.
Another thing I'd like for servers, is to process partially received
requests, otherwise small chunk sizes would have to be used for mobile
which would lead to significant overhead.

Thanks in advance for any insights you might be able to provide!

[1]: http://www.tus.io/protocols/resumable-upload.html
[2]: https://twitter.com/jreschke/status/324872511697326080

--
Felix Geisendörfer (felixge.de)
Co-Founder, Transloadit (transloadit.com)

---

From: Daniel Stenberg <[email-masked]>
Date: Thu, 18 Apr 2013 23:18:39 +0200 (CEST)

> the relevant subsets of http and make them easy for people to implement.

My initial view on this:

"Resume" would be better replaced with "sending a part of a remote resource to 
get applied on the remote side", and in your most common case that part of 
data would modify the end of the remote resource (by adding data to it).

Said like that, isn't that what PATCH is?

-- 

  / daniel.haxx.se

---

From: Kevin Swiber <[email-masked]>
Date: Thu, 18 Apr 2013 17:57:04 -0400

On Apr 18, 2013, at 5:18 PM, Daniel Stenberg <[email-masked]> wrote:

>> The first draft for this can be seen here [1]. I'm new to protocol authorship, so please forgive any ignorance you may see in my work. This is no attempt at competing with existing standards, I merely want to identify the relevant subsets of http and make them easy for people to implement.
> 
> My initial view on this:
> 
> "Resume" would be better replaced with "sending a part of a remote resource to get applied on the remote side", and in your most common case that part of data would modify the end of the remote resource (by adding data to it).
> 

IIRC, PATCH makes no special consideration for byte ranges.  RFC 5789 defines the request body as a "patch document."  Before minting a new media type to support byte-range patch documents, it looks like the authors and interested members of the community are trying to fit this into existing HTTP functionality via byte-range headers and range-related status codes.  

A couple of questions:

1. In the case of a resource being in a known-partial state, what response should be sent when a GET request is received and the request has no Range header?
3. What potential HTTP-compatibility issues do you foresee with such a solution?

While an "application/octet-patch" media type or something like that might be a necessity, I think there's value in fleshing out the other HTTP bits at this time.

Thanks,

-- 
Kevin Swiber
@kevinswiber
https://github.com/kevinswiber

---

From: Albert Lunde <[email-masked]>
Date: Thu, 18 Apr 2013 17:09:54 -0500

>> "Resume" would be better replaced with "sending a part of a remote resource to get applied on the remote side", and in your most common case that part of data would modify the end of the remote re…

Isn't this more in the range of functionality provided by webdav servers?

-- 
     Albert Lunde  [email-masked]
                   [email-masked]  (address for personal mail)

---

From: Julian Reschke <[email-masked]>
Date: Fri, 19 Apr 2013 08:22:27 +0200

On 2013-04-19 00:09, Albert Lunde wrote:
> ...
> ...

It's certainly something a WebDAV server might want to implement, but 
right now there is no spec for this.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 19 Apr 2013 13:28:09 +0200

On 2013-04-19 12:57, Felix Geisendörfer wrote:
> Thanks for the replies!
>
> Using PATCH with a custom media type is unfortunately not a viable
> solution for browser based clients.  The XHR2 send interface [1] and
> File API [2] are not flexible enough to perform the body wrapping
> without loading the entire request body into memory. There is no way to
> stream body data other than providing a file object (or a slice of it).
>
> This makes specifying the enclosed byte-range in the request header the
> most desirable option. So far we are abusing the "Content-Range" header
> for this (and so does the YouTube API [3]).
>
> So I'm wondering what you'd consider to be the lesser evil: Using
> Content-Range for this, or creating a custom header?

See 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p2-semantics-22.html#rfc.section.4.3.4.p.11>:

"An origin server SHOULD reject any PUT request that contains a 
Content-Range header field (Section 4.2 of [Part5]), since it might be 
misinterpreted as partial content (or might be partial content that is 
being mistakenly PUT as a full representation). Partial content updates 
are possible by targeting a separately identified resource with state 
that overlaps a portion of the larger resource, or by using a different 
method that has been specifically defined for partial updates (for 
example, the PATCH method defined in [RFC5789])."

If you believe that this is unreasonable, now and here are the right 
place to discuss it.

With respect to browser constraints: can you generate a multipart body?

Best regards, Julian

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 19 Apr 2013 13:42:53 +0200 (CEST)

On Fri, 19 Apr 2013, Julian Reschke wrote:

> misinterpreted as partial content (or might be partial content that is being 
> mistakenly PUT as a full representation).

This explanation basically rules out PUT completely for upload resume, as even 
if this would instead be done with an imaginary new header called 
Partial-update-of-remote-thing-please:, it could also become subject of 
getting handled as a full representation by mistake.

And if PATCH is ruled out because how browser APIs, I guess only POST is left.

(I personally don't think limitations in existing APIs are a very good 
arguments though, as we're surely forced to update things - including APIs - 
to take advantage of this once we agree on how things should work.)

> If you believe that this is unreasonable, now and here are the right place 
> to discuss it.

My personal belief used to be that Content-Range was a suitable header for 
this purpose, back with the original RFC2616 wording. With the updated httpbis 
wording it is clear that Content-Range doesn't work for this.

-- 

  / daniel.haxx.se

---

From: Martin Thomson <[email-masked]>
Date: Fri, 19 Apr 2013 11:09:33 -0700

This is certainly interesting work.  I find that HTTP uploads are too brittle.

One thing that I'd like to get a better understanding of is the
situation that lead to the need for resumption.

Is it the case that you have a PUT (or some other request) that was
interrupted in some way?  Establishing expectations on server behavior
with respect to that request is probably more important than
concentrating on the subsequent repair request.

--Martin

p.s., Content-Range and PUT don't appear to be the right choice,
though it might be possible if the server was able to indicate support
prior to the interruption.  A 1xx response might work; 100-continue
seems like a good place to indicate this, though it might be crazy to
tie the fate of this to something that is endangered as it is.

On 19 April 2013 04:42, Daniel Stenberg <[email-masked]> wrote:
> This explanation basically rules out PUT completely for upload resume, as
> even if this would instead be done with an imaginary new header called
> this purpose, back with the original RFC2616 wording. With the updated
> httpbis wording it is clear that Content-Range doesn't work for this.
>
> --
>
>  / daniel.haxx.se
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 19 Apr 2013 22:08:59 +0200

On Apr 19, 2013, at 20:09, Martin Thomson <[email-masked]> wrote:

> I find that HTTP uploads are too brittle.

I'd like to understand the failure model a bit better.

Your assumption seems to be that if an upload breaks, it breaks cleanly, i.e. all data that have been received by the server are fine up to the last byte.
That hasn't always been my experience, in particular with broken hotel NATs etc., but it may be good enough if there is some post-resumption check.

The other extreme is doing an HTTP version of rsync, which I think would solve a lot of problems at the same time.

Grüße, Carsten

---

From: Daniel Stenberg <[email-masked]>
Date: Fri, 19 Apr 2013 22:56:54 +0200 (CEST)

> I'd like to understand the failure model a bit better.

I'd just add that a resumed upload does not necessarily imply a previous 
upload failure. It could also be that you're appending parts over time as they 
"come in" or similar.

-- 

  / daniel.haxx.se

---

From: Martin Thomson <[email-masked]>
Date: Fri, 19 Apr 2013 14:12:01 -0700

On 19 April 2013 13:56, Daniel Stenberg <[email-masked]> wrote:
> upload failure. It could also be that you're appending parts over time as
> they "come in" or similar.

I'd have thought that this is a perfect fit for PATCH.  Nothing new needed.

---

From: Nico Williams <[email-masked]>
Date: Sat, 20 Apr 2013 00:02:49 -0500

> I'd have thought that this is a perfect fit for PATCH.  Nothing new
needed.

Sure, but how do you indicate that a transfer is complete?

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Sat, 20 Apr 2013 00:07:47 -0500

Oh, maybe you'd like O_APPEND semantics?  (Not that those are tightly
defined, but you know what I mean.)

Should HTTP have random byte range I/O and byte range locking?  Should it
be a filesystem or object storage protocol?  No, well, maybe; HTTP is every
protocol :)

Nico
--

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 20 Apr 2013 15:59:27 +1000

On 18/04/2013, at 11:37 PM, Felix Geisendörfer <[email-masked]> wrote:

> Hi,
> 
> I'm interested in finding out how to perform resumable uploads over http while being compliant with existing specifications. The result of this work will be shared with the community to create interopable server/client software to simplify file uploading on the web.

Thanks!

Daniel wrote:

>> "An origin server SHOULD reject any PUT request that contains a Content-Range header field (Section 4.2 of [Part5]), since it might be misinterpreted as partial content (or might be partial content that is being mistakenly PUT as a full representation).
> 
> This explanation basically rules out PUT completely for upload resume, as even if this would instead be done with an imaginary new header called Partial-update-of-remote-thing-please:, it could also become subject of getting handled as a full representation by mistake.

I agree. You can't guarantee that a new header is understood by the recipient (M-PUT died long, long ago), and this isn't a backwards-compatible change in the semantics of PUT.

> I'd have thought that this is a perfect fit for PATCH.  Nothing new needed.

Agreed, except a new PATCH format that's range-friendly would be necessary. That's not a huge undertaking, because it could reuse at least some of the existing syntax.

I'd be willing to help work on this, or just provide input / reviews.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: Felix Geisendörfer <[email-masked]>
Date: Sat, 20 Apr 2013 10:30:33 +0200

On Sat, Apr 20, 2013 at 7:59 AM, Mark Nottingham <[email-masked]> wrote:

> Agreed, except a new PATCH format that's range-friendly would be
> necessary. That's not a huge undertaking, because it could reuse at least
> some of the existing syntax.
>

IMO the simplest solution would be an "Offset" header that simply gives the
start offset where the data should be applied. The end offset is implicit
through the message length.

> I'd be willing to help work on this, or just provide input / reviews.

That's fantastic, thank you so much! I'm also very interested in helping in
whatever way possible.

On Fri, Apr 19, 2013 at 10:08 PM, Carsten Bormann <[email-masked]> wrote:

Your assumption seems to be that if an upload breaks, it breaks cleanly,
> i.e. all data that have been received by the server are fine up to the last
> byte.
>

Yes, but IMO we should assume that the underlaying transport layer is not
corrupting data under normal circumstances.

Applications that require stronger guarantess should split their uploads
into small chunk requests with individual checksums and instruct their
servers to not process anything that doesn't match the checksum. This
carries a higher overhead, but such is life : ).

Our goal for tus.io is to describe the simplest interoperable resumable
upload strategy over http that will work in most situations (the core). The
rest of the protocol will cover extensions (chunking, checksums, uploading
chunks in parallel, etc.) which clients / servers can choose to implement
as well. Ideally a nice ecosystem of compatible libraries will then
flourish on top.

Cheers,

---

From: Yoav Nir <[email-masked]>
Date: Sat, 20 Apr 2013 19:43:34 +0000

On Apr 20, 2013, at 11:30 AM, Felix Geisendörfer <[email-masked]<mailto:[email-masked]>> wrote:

On Sat, Apr 20, 2013 at 7:59 AM, Mark Nottingham <[email-masked]<mailto:[email-masked]>> wrote:
Agreed, except a new PATCH format that's range-friendly would be necessary. That's not a huge undertaking, because it could reuse at least some of the existing syntax.

IMO the simplest solution would be an "Offset" header that simply gives the start offset where the data should be applied. The end offset is implicit through the message length.

How does the server know when the whole thing has been uploaded?  Wouldn't we need some kind of "total-length" header, perhaps in the initial PUT?

Or are we assuming that the initial PUT is trying to upload everything, and only failure leads to sending the PATCH?

Yoav

---

From: Ilari Liusvaara <[email-masked]>
Date: Sun, 21 Apr 2013 00:09:57 +0300

> Or are we assuming that the initial PUT is trying to upload everything, and only failure leads to sending the PATCH?

This can't be assumed if client is JS running on the browser (even
assuming it can send PATCH). This is because JS code can't read huge
chunks of file at once[1].

[1] Think transferring e.g. 22,682,958,652 or 26,578,892,498 byte file [2]...
[2] Yes, these sizes are from real-world cases.

-Ilari

---

From: Daniel Stenberg <[email-masked]>
Date: Sat, 20 Apr 2013 23:20:35 +0200 (CEST)

On Sat, 20 Apr 2013, Yoav Nir wrote:

> How does the server know when the whole thing has been uploaded?  Wouldn't 
> we need some kind of "total-length" header, perhaps in the initial PUT?
>

The initial upload could still send "everything" that is everything for that 
particular operation at that time, and then you'd set the expected size of the 
body like always with Content-Length: (or use chunked-encoding or ...).

If you then want to append data to the remote resource, which very well could 
happen without a failure anywhere. Let's for example imagine the case where 
host A just wants to append data to a logfile on host B. It could then append 
a chunk of data every now and then.

I agree with Fredrik's statement that a simple PUT + offset header would be 
easiest for a client (and possibly server too), but I just previously quoted 
the reasoning in the spec as to why it currently doesn't think that is a good 
idea. It would be more productive if we would instead counter that argument or 
present an alternative way.

-- 

  / daniel.haxx.se

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 21 Apr 2013 13:29:12 +1200

On 20/04/2013 5:02 p.m., Nico Williams wrote:
>
>
> On Apr 19, 2013 5:13 PM, "Martin Thomson" <[email-masked] 
> <mailto:[email-masked]>> wrote:
> >
> > On 19 April 2013 13:56, Daniel Stenberg <[email-masked] 
> <mailto:[email-masked]>> wrote:

I would think that any API using PATCH, started with a PUT to create the 
resource, used a series of PATCH to updated it and is treating the 
intermediary stages between initial PUT and followup PATCH's as:

a) time periods when GET on the resource can pull the current state in 
its entirety (no need for a completion signal, as every PATCH makes a 
new "complete" / up-to-date copy)

b) using a final POST to perform resource manipulation through some 
control script - ie copy the final solution to a proper Location, and 
run any post-processing checks.

c) using LOCK/UNLOC requests on theh resource URL explicitly to signaly 
the in-use state across teh period when the resource is being PATCH'd.

And yes, all of these assume working API no problems with a particular 
request/response message. The interesting cases for resume are the ones 
when corruption, loss, or disconnection happen while the server is 
receiving a request.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 21 Apr 2013 12:08:39 +1000

On 21/04/2013, at 5:43 AM, Yoav Nir <[email-masked]> wrote:

> 
> On Apr 20, 2013, at 11:30 AM, Felix Geisendörfer <[email-masked]> wrote:
> 

One possibility would be to have a flag in the request (header or body) that indicates that after this PATCH, it's complete. E.g.,

PUT /foo HTTP/1.1  # create the new resource

PATCH /foo HTTP/1.1  # start uploading it...
PATCH /foo HTTP/1.1  # continue...
...

PATCH /foo HTTP/1.1
Upload-Complete: 1  # we're done!

You'd need conditionals (probably, if-match) on each PATCH to make sure it hasn't changed since the last PATCH (or PUT).

The PUT give the server an opportunity to refuse the request, and its response can advertise the details of how to PATCH it.

The only difficult part I can see is figuring out what to return to a GET that occurs before the upload is complete. And maybe how long to keep these partially-uploaded resources around (although that's really a server implementation detail).

It'd be nice if the PATCH format could modify headers cleanly too.

Cheers,

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 21 Apr 2013 14:42:44 +1200

> The only difficult part I can see is figuring out what to return to a GET that occurs before the upload is complete. And maybe how long to keep these partially-uploaded resources around (although t…

Not too difficult. The processing model for these type of systems falls 
into two categories of locked or lockless. The GEt on locked must be 
responded with some form of error-because-locked response, and GET on 
lockless must always return the current representation with recipient 
system able to cope with corrupted/incomplete representations on its own 
outside of the delivery protocol.

Amos

---

From: Felix Geisendörfer <[email-masked]>
Date: Sun, 21 Apr 2013 12:12:45 +0200

On Sat, Apr 20, 2013 at 11:09 PM, Ilari Liusvaara <
[email-masked]> wrote:

> This is because JS code can't read huge chunks of file at once[1].
>

AFAIK that's not true. File API 8.1 says: "The File interface, which
inherits from Blob, is immutable, and thus represents file data that can be
read into memory at the time a read operation is initiated." [1]. In my
testing this optimization is found in all browser implementations.

On Sat, Apr 20, 2013 at 11:20 PM, Daniel Stenberg <[email-masked]> wrote:
>
> I agree with Fredrik's statement that a simple PUT + offset header would
> be easiest for a client (and possibly server too), but I just previously
> quoted the reasoning in the spec as to why it currently doesn't think that
> is a good idea. It would be more productive if we would instead counter
> that argument or present an alternative way.

My proposal is to use PATCH + Offset header. PUT could still be used for
creating the resource initially and setting expectations for final size,
checksum, etc.

Overall I think there are a few separate things that are currently lacking
http primitives. In order of priority:

1. Defining a simple way to insert data at a given offset of a resource.
(PATCH + Offset header seems nice/simple)
2. Defining a fixed file size for a resource when creating it (suggestion:
"Final-Length" header), regardless of how much data will be transmitted
with the creation request.
3. Defining a way for a server to provide the state of a partial resource
via HEAD requests. (could also be a "Offset" header)

(I've left out checksums, EOF/Upload-Complete and a few other things on
this list, just to keep it simple for now.)

These primitives could then be combined into various client/server patterns
depending on application requirements.

I don't think the http spec should define the server behavior of GET
requests on partially uploaded resources, as there are many patterns that
would make sense for different applications.

[1]: http://www.w3.org/TR/FileAPI/#file-attrs

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 21 Apr 2013 13:23:39 +0200

> Upload-Complete: 1  # we're done!

This is starting to look like draft-ietf-core-block, and is bound to have a number of the same characteristics (except that CoAP blocks are small and fixed-size).
(Of course, -block is solving a different problem...)

How to do this properly probably depends on whether the new resource is freestanding, is bound to be part of a collection, or is an update of an existing resource.

The first PUT may not really create the resource, it might create a temporary that can become the new representation of the resource.
The resource representation is only really completely uploaded with the final patch.

If access to the resource-in-creation is exclusive (e.g., a new URI that is only used by the uploader), the resource can grow in place.
If you don't want to indicate explicitly that it is done, the client can do that implicitly by just starting to use the URI.
Garbage collection (incompletely uploaded resources) is more fun, though.

If access is non-exclusive (multiple clients could be updating it in parallel), you can lock, or you can create an explicit temporary for each uploader, enabling "the last update wins".

Since the original project seems to be about uploading videos, exclusive access is probably the better paradigm.
We don't know if a PATCH (or even the initial PUT) is going to go through completely, so indicating at the start that it is the last one may be impossible.
An explicit close, conditional on the size actually received, may be the best way to do this.
This could use the LINK method or something else.

Grüße, Carsten

---

From: Felix Geisendörfer <[email-masked]>
Date: Fri, 19 Apr 2013 15:13:05 +0000

Thanks for the additional insights.

And if PATCH is ruled out because how browser APIs, I guess only POST is
> left.

Actually, PATCH could work great if there was an appropiate header to
indicate the offset. Example of resuming an upload that was interrupted
after 20 bytes:

POST /file/my.jpg
Content-Length: 100

<file data … interrupted after 20 bytes>

HEAD /file/my.jpg

HTTP/1.1 200 OK
Offset: 20

PATCH /file/my.jpg
Content-Length: 80
Offset: 20

<file data>

I realize RFC 5789 defines the request body as a "patch document", but it
creates a lot of complexity for this use case.

Cheers,

---

From: Felix Geisendörfer <[email-masked]>
Date: Fri, 19 Apr 2013 10:58:40 +0000

Thanks for the replies!

Using PATCH with a custom media type is unfortunately not a viable solution
for browser based clients.  The XHR2 send interface [1] and File API [2]
are not flexible enough to perform the body wrapping without loading the
entire request body into memory. There is no way to stream body data other
than providing a file object (or a slice of it).

This makes specifying the enclosed byte-range in the request header the
most desirable option. So far we are abusing the "Content-Range" header for
this (and so does the YouTube API [3]).

Don't get me wrong, I really want a proper solution for this in the long
run and would love to help with specifying it, if it's an interesting
problem to solve with http2. But if such a solution can't be had today, I'd
still like to create a stopgap for now.

[1] http://www.w3.org/TR/XMLHttpRequest2/#the-send-method
[2] http://www.w3.org/TR/FileAPI/
[3]
https://developers.google.com/youtube/v3/guides/using_resumable_upload_protocol

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 22 Apr 2013 10:09:04 +0200

On Apr 19, 2013, at 17:13, Felix Geisendörfer <[email-masked]> wrote:

> "patch document",

How about an application/bytes-at like this:

-- Starts with an 8-byte binary unsigned integer (network byte order) giving the position at which these data are to be written
-- the data follow

Makes you independent from intermediaries bashing headers.

(A potential application/append-bytes, without a position, would be bad, because part of your old PUT could still be in a TCP retransmission queue somewhere, creating a race condition.)

Grüße, Carsten

---

From: Felix Geisendörfer <[email-masked]>
Date: Mon, 22 Apr 2013 12:15:33 +0200

> How about an application/bytes-at like this:

What about my suggestion of simply defining an "Offset" header? Any kind of
patch body format will currently rule out HTML5 clients as discussed
earlier, and I'm more worried about that than about proxies stripping
headers.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 22 Apr 2013 12:18:29 +0200

On Apr 19, 2013, at 12:58, Felix Geisendörfer <[email-masked]> wrote:

>  XHR2 send interface

Well, that probably should be fixed to allow an array of things, and not a single thing, where the things are then concatenated on the wire.

But until then:  
Put the offset:

-- in the URI: 
   PATCH /foo?at=0020 HTTP/1.1
-- in the Content-Type: 
   Content-Type: application/byteslice; at=0020

I'm not sure any other header is likely to survive today's world of middleboxes.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 22 Apr 2013 12:34:44 +0200

On 2013-04-22 12:15, Felix Geisendörfer wrote:
> On Mon, Apr 22, 2013 at 10:09 AM, Carsten Bormann <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> What about my suggestion of simply defining an "Offset" header? Any kind
> of patch body format will currently rule out HTML5 clients as discussed

But you *can* send multipart body from Javascript, right?

Best regards, Julian

---

From: Felix Geisendörfer <[email-masked]>
Date: Mon, 22 Apr 2013 18:09:55 +0200

> But you *can* send multipart body from Javascript, right?

Yes. It's an option, but I think multipart would add a lot of complexity if
all we need is a way to indicate an offset argument.

On Mon, Apr 22, 2013 at 12:18 PM, Carsten Bormann <[email-masked]> wrote:

> >  XHR2 send interface
> Well, that probably should be fiaxed to allow an array of things, and not
> a single thing, where the things are then concatenated on the wire.
>

I agree. But that will take time.

>    Content-Type: application/byteslice; at=0020

Thanks for the suggestions. The querystring might be the best option for
something the web can use today.

That being said, I'd still like to see http provide built-in primitives for
resume operations in the future and make sure whatever userland protocol we
end up defining has these upcoming mechanisms in mind / includes a clear
upgrade path.

So if my concerns for portability were lifted, it seems that defining a
PATCH format would be the best solution? Would this list still be the right
place for carrying on this discussion, or would this be a separate RFC?

Cheers,

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 22 Apr 2013 18:16:00 +0200

> Thanks for the suggestions. The querystring might be the best option for something the web can use today.

Actually, I think the content-type parameter is the winner.
This doesn't require limiting the URI space/design.
It does require access to the content-type both at the sending and receiving end; maybe I'm not aware of environments where that would be a problem.
(It doesn't work for CoAP, but we have our own way of doing this.)

> So if my concerns for portability were lifted, it seems that defining a PATCH format would be the best solution? Would this list still be the right place for carrying on this discussion, or would this be a separate RFC?

I thought we just designed a complete PATCH format for resuming uploads.  Aren't we done?

Grüße, Carsten

---

From: Ken Murchison <[email-masked]>
Date: Mon, 22 Apr 2013 13:43:21 -0400

> software to simplify file uploading on the web.
Here are my thoughts on this, some of which have already been mentioned 
by others.

- The client should/must have a way to signal to the server that its 
supports partial uploads so that the server can respond accordingly.  
I'd suggest a "partial-upload" (or similar)  preference to be used with 
the Prefer [1] request header.

- Allow progress of the upload to be reported back to the client via 
Progress [2] (or similar) response header.

- PATCH seems to be designed explicitly for the purpose up updating 
existing resources and would make sense to use for completing/repairing 
an upload.  I'd suggest use of multipart/byteranges as the "patch" document.

- A server will acknowledge that it supports "partial-upload" by 
including an "Accept-Patch: multipart/byteranges" header in its responses.

- Allow HEAD requests to include range request semantics in the presence 
of "Prefer: partial-upload".

An example of how this might work (I've left out most headers for brevity);

Initial request:

PUT /file.pdf HTTP/1.1
Expect: 100-continue
Prefer: partial-upload
Content-Type: application/pdf
Content-Length: 1000

HTTP/1.1 100 Continue
Allow-Patch: multipart/byteranges

[ 1000 bytes of data ]
HTTP/1.1 102 Processing				<<< optional progress status
Progress: 256/1000

HTTP/1.1 102 Processing				<<< optional progress status
Progress: 512/1000

<<< connection lost >>>

Since in the case above the client already knows that the server 
processed bytes 0-511, it can try to resume the upload immediately 
(saving a round-trip). Otherwise it can check the current status with a 
HEAD range request:

HEAD /file.pdf HTTP/1.1
Prefer: partial-upload
Range: bytes=0-

HTTP/1.1 206 Partial Content
Allow-Patch: multipart/byteranges
Content-Range: bytes 0-511/1000

Resumption of upload:

PATCH /file.pdf HTTP/1.1
Prefer: partial-upload
Content-Type: multipart/byteranges; boundary=THIS_STRING_SEPARATES
Content-Length: xxx

--THIS_STRING_SEPARATES
Content-Type: application/pdf
Content-Range: bytes 512-999/1000

[ 488 bytes of data ]
--THIS_STRING_SEPARATES--

HTTP/1.1 102 Processing				<<< optional progress status
Progress: 256/488

HTTP/1.1 204 No Content				<<< completed successfully

This scheme can probably be tweaked to work with chunked uploads, but I 
haven't thought much about it yet.

In cases where a client does a HEAD/GET on a partial resource without 
"Prefer: partial-upload", I don't know what the server should do.  There 
are at least 4 options:

    * Treat the resource as complete and return 200
    * Treat the resource as partial and always return 206 (will probably
      break clients)
    * Treat the resource as non-existent and return 404
    * Fail the request with a 403 (or similar)

To append data to an existing resource we could extend the Content-Range 
ABNF a little to allow a PATCH request as follows:

PATCH /log.txt HTTP/1.1
Content-Type: multipart/byteranges; boundary=THIS_STRING_SEPARATES
Content-Length: xxx

--THIS_STRING_SEPARATES
Content-Type: text/plain
Content-Range: bytes +200/*			<<<  append 200 bytes to existing length

[ 200 bytes of data ]
--THIS_STRING_SEPARATES--

HTTP/1.1 204 No Content

[1] http://tools.ietf.org/html/draft-snell-http-prefer
[2] http://tools.ietf.org/html/draft-decroy-http-progress

-- 
Kenneth Murchison
Principal Systems Software Engineer
Carnegie Mellon University