Subject: Draft for Resumable Uploads

From: Marius Kleidl <[email-masked]>
Date: Fri, 1 Apr 2022 11:48:16 +0200

Hello HTTP working group,

we are all familiar with connectivity disruptions affecting our
internet activities. One example is when a large file download is
interrupted; say a 100 MB file download encounters a network loss
after the client receives 70 MB. Fortunately, resumable HTTP downloads
using range requests are a widely deployed standard feature that
allows clients to fetch the remaining 30 MB only, saving time and
resources for both endpoints. However, in the opposite direction,
there is not a standard convention for resuming HTTP uploads.

Across the HTTP ecosystem there are several different approaches to
providing resumable uploads. We are aware of at least one attempt to
try and standardize an approach [1], but to our knowledge none have
succeeded in being adopted and driven to conclusion.

We believe resumable uploads are a common problem and that there is
value in a standard resumable upload approach. We've been working on a
document [2] [3] that uses HTTP to solve what we believe to be the
core problem set, while also allowing for extended use cases. We are
bringing this to the list to understand if there is interest in the
working group to solve the problem, and whether our document is a good
basis for a solution.

In case you are interested in the background of this draft: The origin
is within the tus project [4], which has been developing a HTTP-based
protocol for resumable uploads [5] since 2013 (tus was also posted on
this mailing list at the time [6]). Furthermore, we also provide
various open-source implementations [7] to allow easy usage on the
web, in mobile applications, desktop application, or server
environments. tus has seen great adaption, proving that there is a
demand for an open-source solution providing resumable uploads.

We hope to bring resumable uploads to more people. For this, adopting
resumable uploads into HTTP would be a great step. There is also
interest in including support for resumable uploads natively into
platforms, like browsers and mobile SDKs, so that developers do not
have to bring their own library for resumable uploads.

We have taken the main uploading process from our tus protocol and
reworked it into a self-containing draft, which we want to present to
you! As such, this draft can be seen as an evolution of our work on
tus and as a step to increase availability of resumable uploads.

Thank you for any feedback in advance!

Best regards,
Marius Kleidl

[1] https://lists.w3.org/Archives/Public/ietf-http-wg/2019JulSep/0066.html
[2] https://datatracker.ietf.org/doc/draft-tus-httpbis-resumable-uploads-protocol/
[3] https://github.com/tus/tus-v2
[4] https://tus.io/
[5] https://tus.io/protocols/resumable-upload.html
[6] https://mailarchive.ietf.org/arch/msg/httpbisa/I__B5Kc7h-1TvRRB9rmjY8tR-T0/
[7] https://tus.io/implementations.html

---

From: "Roy T. Fielding" <[email-masked]>
Date: Fri, 1 Apr 2022 10:30:27 -0700

> On Apr 1, 2022, at 2:48 AM, Marius Kleidl <[email-masked]> wrote:
> 
> we are all familiar with connectivity disruptions affecting our internet activities. One example is when a large file download is interrupted; say a 100 MB file download encounters a network loss after the client receives 70 MB. Fortunately, resumable HTTP downloads using range requests are a widely deployed standard feature that allows clients to fetch the remaining 30 MB only, saving time and resources for both endpoints. However, in the opposite direction, there is not a standard convention for resuming HTTP uploads.
> 
> Across the HTTP ecosystem there are several different approaches to providing resumable uploads. We are aware of at least one attempt to try and standardize an approach [1], but to our knowledge none have succeeded in being adopted and driven to conclusion.
> 
> We believe resumable uploads are a common problem and that there is value in a standard resumable upload approach. We've been working on a document [2] [3] that uses HTTP to solve what we believe to be the core problem set, while also allowing for extended use cases. We are bringing this to the list to understand if there is interest in the working group to solve the problem, and whether our document is a good basis for a solution.
> 
> In case you are interested in the background of this draft: The origin is within the tus project [4], which has been developing a HTTP-based protocol for resumable uploads [5] since 2013 (tus was also posted on this mailing list at the time [6]). Furthermore, we also provide various open-source implementations [7] to allow easy usage on the web, in mobile applications, desktop application, or server environments. tus has seen great adaption, proving that there is a demand for an open-source solution providing resumable uploads.
> 
> We hope to bring resumable uploads to more people. For this, adopting resumable uploads into HTTP would be a great step. There is also interest in including support for resumable uploads natively into platforms, like browsers and mobile SDKs, so that developers do not have to bring their own library for resumable uploads.
> 
> We have taken the main uploading process from our tus protocol and reworked it into a self-containing draft, which we want to present to you! As such, this draft can be seen as an evolution of our work on tus and as a step to increase availability of resumable uploads.
> 
> [1] https://lists.w3.org/Archives/Public/ietf-http-wg/2019JulSep/0066.html <https://lists.w3.org/Archives/Public/ietf-http-wg/2019JulSep/0066.html>
> [2] https://datatracker.ietf.org/doc/draft-tus-httpbis-resumable-uploads-protocol/ <https://datatracker.ietf.org/doc/draft-tus-httpbis-resumable-uploads-protocol/>
> [3] https://github.com/tus/tus-v2 <https://github.com/tus/tus-v2>
> [4] https://tus.io/ <https://tus.io/>
> [5] https://tus.io/protocols/resumable-upload.html <https://tus.io/protocols/resumable-upload.html>
> [6] https://mailarchive.ietf.org/arch/msg/httpbisa/I__B5Kc7h-1TvRRB9rmjY8tR-T0/ <https://mailarchive.ietf.org/arch/msg/httpbisa/I__B5Kc7h-1TvRRB9rmjY8tR-T0/>
> [7] https://tus.io/implementations.html <https://tus.io/implementations.html>
> 
This is probably not a good day to discuss this, but it is clear from the
draft that this is not using HTTP correctly.

tus-v2 assumes that there is a separate resource for uploading, as opposed to
targeting a resource and letting the server decide whether it can upload into
a temporary resource for that target. It doesn't indicate what the server
is to do with the data once it is uploaded, which implies this is just part
of a private agreement instead of a standard protocol.

Subsequent requests target the same upload resource, instead of targeting
a separate temporary resource in progress. This results is some seriously
confused semantics when the client ends with a DELETE targeting the resource
for uploading.

Changing the semantics of an existing method using a header field is only
interoperable if the new field can be ignored. That is not the case here
for a DELETE on the process URI.

Likewise, not targeting by resource (URI) interferes with resource-based
access control and authorization, and fails to distinguish between uploads
where the user agent knows where to PUT the data and those where the
user agent is asking the server to choose where to POST the data.

For example, what happens when the server includes multiple
user-authenticated subtrees and this user is only authorized to upload
to some of them?

A simple fix is to send the initial upload as a PUT (to a target URI for
the completed upload) or as a POST (to clearly allow the server to select
a destination). The server can indicate that it supports continuation by
providing a temporary URI in a 1xx response. This new target is essentially
a buffer with a URI. The client can then monitor/continue requests on the
new URI, cancel by sending DELETE to that new URI, or finalize the upload
by sending some final metadata (e.g., DIgest) to that new URI. Once final
(either my completing the original request or receiving a finalized on the
temporary URI, the server can move the received data to where the client
indicated and delete the temporary URI.

The temporary URI is the token -- there is no need for a separate identifier,
unless you want to recover from missed responses (i.e., be able to repeat
the same request multiple times and let the server decide when it was
already done, for which a general request-id would be more appropriate).

Furthermore, the above can be generalized to more useful cases where
very large uploads are needed in practice. All of the ones that I have seen
deployed for real reasons have been to solve load/scale/speed problems
elsewhere in a chain of intermediaries, not just to send a very large file
to an HTTP origin server (which the vast majority of servers can handle
just fine with HTTP/1.1 over TCP).

For example, sending terabytes of data to S3 in parallel uploads to
multiple services that are then reassembled within AWS. This requires a
design where the user agent requests instruction on how/where to upload
each part in parallel and the server reconstitutes the data upon receiving
finalization of every part. IOW, the initial method with Expect and a field
indicating how large the upload will be, resulting in a 1xx/3xx list of
temporary target URIs (or URI templates) selected by the server,
potentially on different origins, where each indicated range can be
resumably-uploaded in parallel and then finalized.

Note that, if you stick with HTTP semantics and URIs as identifiers, the
complex use case is just a generalization of the smaller case.

Cheers,

....Roy

---

From: Guoye Zhang <[email-masked]>
Date: Fri, 01 Apr 2022 12:30:21 -0700

> On Apr 1, 2022, at 10:30 AM, Roy T. Fielding <[email-masked]> wrote:
> 

tus-v2 is not intended to be a protocol using HTTP, it is designed to be an extension of HTTP. In particular, any upload methods, regardless of POST, PUT, etc can be transparently upgraded to support resumable uploads. It should not prevent the server from organizing the resource in any way.
> for a DELETE on the process URI.

DELETE is only used if the client detects the server supports tus. We can change it to a header if there are existing applications using the same URI for upload and delete.
> indicated and delete the temporary URI.

Unfortunately, we’ve determined that 1xx response is too unreliable for critical information. Many middleboxes drop it, and most client frameworks do not expose it. Therefore we only use it for feature detection and none of the critical functionalities depend on it.
> resumably-uploaded in parallel and then finalized.

Parallel upload is not in scope of this proposal. We are mainly focusing on resuming an interrupted upload.
> complex use case is just a generalization of the smaller case.

Thanks for reading the proposal and responding. The goal of the protocol is to enhance the upload process of HTTP and be method-agnostic. If our use of GET/DELETE on the same URI is a blocker for your application, we’d happy to discuss options such as changing them into header fields instead, and you can already disambiguate them by the presence of the Upload-Token header.

Guoye

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 2 Apr 2022 12:30:45 +1100

> On 2 Apr 2022, at 6:30 am, Guoye Zhang <[email-masked]> wrote:
> 

That's something you should re-examine. When running over HTTPS and especially HTTP/2 (or /3), it's been found sufficiently reliable; e.g., see browsers' emerging deployments of 103 Early Hints.

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: Guoye Zhang <[email-masked]>
Date: Fri, 01 Apr 2022 19:40:38 -0700

> On Apr 1, 2022, at 6:30 PM, Mark Nottingham <[email-masked]> wrote:
> 
> 
> 
Yes, we have looked at Early Hints. It is a performance optimization and the loss of this response does not prevent any features from working.

We hope that tus-v2 can be implemented either by the HTTP libraries themselves, or applications on top of it. For example, JavaScript `fetch` API does not expose 1xx responses at all. If we put any important info in intermediate responses, it would make the protocol unimplementable in JavaScript.

Guoye

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 2 Apr 2022 13:46:41 +1100

1xx is exposed in node.js and other JS HTTP implementations. I think what you mean is that it won't be implementable using JS in a browser. However, the right thing to do there may very well be to implement natively in the browser -- assuming that it has to be implementable without changes to browsers is an artificial constraint that may not be relevant.

> On 2 Apr 2022, at 1:40 pm, Guoye Zhang <[email-masked]> wrote:
> 
> 
>

---

From: Guoye Zhang <[email-masked]>
Date: Fri, 01 Apr 2022 21:02:19 -0700

I agree that 1xx is a great fit for this on paper, and I have also advocated using 1xx to deliver upload-token and upload-offset in the beginning. However, so far we’ve found none of the commonly used HTTP client-side library exposing this to applications today.

Our goal is to unify the many incompatible resumable upload standards currently being widely deployed across the internet, and requiring a feature that’s not useable today would significantly hinder adoption. Even if every library exposes 1xx responses right now, it will take years before library dependencies, browsers, or even OS updates are widely deployed.

I believe the pragmatic approach is to use the basic mechanism of GET/POST requests and 200/204 responses so applications can adopt it today, while also allowing HTTP libraries to gradually take over the responsibility and eventually upgrade every upload to resumable upload automatically without any application change.

Guoye

> On Apr 1, 2022, at 7:46 PM, Mark Nottingham <[email-masked]> wrote:
>

---

From: Julian Reschke <[email-masked]>
Date: Sat, 2 Apr 2022 09:48:17 +0200

Am 01.04.2022 um 21:30 schrieb Guoye Zhang:
> ...
> tus-v2 is not intended to be a protocol using HTTP, it is designed to be
> an extension of HTTP. In particular, any upload methods, regardless of
> POST, PUT, etc can be transparently upgraded to support resumable
> uploads. It should not prevent the server from organizing the resource
> in any way.
> ...

What is this supposed to mean? Could you elaborate on this distinction?

Best regards, Julian

---

From: Daniel Stenberg <[email-masked]>
Date: Sat, 2 Apr 2022 16:22:34 +0200 (CEST)

On Fri, 1 Apr 2022, Guoye Zhang wrote:

> I agree that 1xx is a great fit for this on paper, and I have also advocated 
> using 1xx to deliver upload-token and upload-offset in the beginning. 
> However, so far we’ve found none of the commonly used HTTP client-side 
> library exposing this to applications today.

Maybe not the important thing to debate here, but libcurl is one of the most 
widely used software components in the world, and an HTTP client-side library 
that "exposes" this information to applications today.

Just saying.

-- 

  / daniel.haxx.se

---

From: Lucas Pardue <[email-masked]>
Date: Sat, 2 Apr 2022 15:46:17 +0100

Hey

On Sat, Apr 2, 2022 at 3:48 AM Mark Nottingham <[email-masked]> wrote:

> 1xx is exposed in node.js and other JS HTTP implementations. I think what
> you mean is that it won't be implementable using JS in a browser. However,
> the right thing to do there may very well be to implement natively in the
> browser -- assuming that it has to be implementable without changes to
> browsers is an artificial constraint that may not be relevant.
>

Native support for 103 Early Hints in browsers is fine, but when web apps
build on top Fetch, they are immediately restricted by the lack of
visibility into these non-final responses. That limits possible application
of Early Hints, and whatever 1xx code might be used for resumable upload.
This isn't the right forum to specify the definition of Fetch but I wonder
what other innovations could be unlocked if we were able to get over the
hump of it's 1xx restrictions. Maybe it's time to try and push the platform
forward.

Cheers
Lucas

---

From: Guoye Zhang <[email-masked]>
Date: Sat, 02 Apr 2022 11:14:50 -0700

> 
> On Apr 2, 2022, at 12:48 AM, Julian Reschke <[email-masked]> wrote:
> 
Sure, what I’m trying to say is that this draft is intended to be genetic mechanism to upgrade any upload into resumable upload regardless of method or URI. I could imagine a future where this is implemented by the CDNs and browsers. The website just posts a plain HTML form, and the origin server also just receives a plain POST, but during transit, browser detects resumable upload support and handles interrupted uploads gracefully.

The exact mechanism (headers and formats of control messages) isn’t set in stone, and we’re happy to make changes if it is in conflict with any existing protocols on top of HTTP. But our overall goal is for this to be easily implementable not only on top of existing HTTP libraries, but also in the HTTP libraries themselves (which is why I said it’s extending HTTP).

Guoye

---

From: Eric J Bowman <[email-masked]>
Date: Mon, 04 Apr 2022 18:27:33 -0700

Missing rest-discuss lately. :(

>
> Sure, what I’m trying to say is that this draft is intended to 

> be genetic mechanism to upgrade any upload into resumable 

> upload regardless of method or URI.

Doesn't HTTP already provide a uniform interface capable of solving this problem? Regardless of upload method...

Server knows the upload was interrupted if the received content falls short of Content-Length, or the connection timed out, or whatever.

 Reconnected client makes HEAD request...

Server responds 206 with Content-Range: {zero to Content-Length} -- indicating an interrupted upload, if the client didn't make a range request.

I don't see any prohibition against responding 206 to a non-range request?

Client now knows how to PATCH the resource. Lather-rinse-repeat until 200 OK on HEAD.

-Eric

---

From: Guoye Zhang <[email-masked]>
Date: Mon, 04 Apr 2022 18:59:50 -0700

> On Apr 4, 2022, at 6:27 PM, Eric J Bowman <[email-masked]> wrote:
> 

Yes, you can implement something that sort of works by combining various defined methods and header fields, but there are many ambiguities:

First, how does it uniquely identify a resumable upload? Does the server need to send a unique URL to the client? What about the additional roundtrip time? tus-v2 draft solves it by defining `Upload-Token` which is a cryptographically-random token generated on the client side.

We’ve also looked at the `Range` header but decided against it due to it providing too much flexibility. Features like multipart ranges are extremely difficult to support on the server. A single integer in `Upload-Offset` models the capability better and sets the correct expectation. That being said, we can revisit this decision if the work group provides a different perspective.

Overall, we believe that a tightly-scoped standard would benefit implementors and encourage wide adoption.

Guoye

---

From: Austin Wright <[email-masked]>
Date: Mon, 4 Apr 2022 20:41:38 -0700

Hello Marius & WG,

I’m glad to see some activity on this issue; I wrote the draft that you linked to in [1]. See [2] for the most relevant of the documents.

One of the reasons I wrote a new document rather than adopting TUS, is that it appears to be using HTTP as a substrate, rather than extending HTTP as an application unto itself. The draft seems to have made some improvements in this area, but it still relies on some assumptions that I tried to avoid.

Notably, it introduces many new headers that modify the semantics of the method. The  "Upload-Offset” and "Upload-Incomplete” headers must be understood by all the HTTP intermedaries the message passes through. This could confuse caches or gateways.

I avoided this by using the PATCH method whenever possible (to modify an existing resource without replacing it, exactly the semantics we’re looking for). The HTTP server, in all cases, gets to select the URI uses to identify resources.

I sort of gave up on my draft after it received no feedback. And I never liked how resumable requests work (e.g. resuming a POST upload), it seems difficult to implement.

However I came back upon it recently. I think 1xx interim responses are ideal for communicating information about an ongoing upload. And I plan on submitting a "message/byterange” media type for consideration as a PATCH body. While this by itself does not provide discoverability, I believe APIs could at least adopt this as a standard media type for segmented/partial uploads. [3]

Do you have any thoughts on this approach?

Thanks,

Austin.

[1] https://lists.w3.org/Archives/Public/ietf-http-wg/2019JulSep/0066.html <https://lists.w3.org/Archives/Public/ietf-http-wg/2019JulSep/0066.html>
[2] https://tools.ietf.org/id/draft-wright-http-partial-upload-01.html <https://tools.ietf.org/id/draft-wright-http-partial-upload-01.html>
[3] https://github.com/awwright/http-progress/issues/1#issuecomment-1061314730 <https://github.com/awwright/http-progress/issues/1#issuecomment-1061314730> for one possible way to use message/byterange with existing, standard HTTP features.

> On Apr 1, 2022, at 02:48, Marius Kleidl <[email-masked]> wrote:
>

---

From: Martin J. Dürst <[email-masked]>
Date: Tue, 5 Apr 2022 13:52:04 +0900

> We’ve also looked at the `Range` header but decided against it due to it providing too much flexibility. Features like multipart ranges are extremely difficult to support on the server.

If that's a problem on the server side, servers that can't handle that 
would just send a simplified Content-Range header (e.g. with only a 
single range).

Regards,   Martin.

> A single integer in `Upload-Offset` models the capability better and sets the correct expectation. That being said, we can revisit this decision if the work group provides a different perspective.
>

---

From: Eric J Bowman <[email-masked]>
Date: Mon, 04 Apr 2022 23:05:51 -0700

> First, how does it uniquely identify a resumable upload?

A 206 response to a non-range request uniquely, unambiguously, and elegantly identifies an incomplete resource. Identifying a resource as both incomplete *and* completeable, introduces tight coupling at the protocol layer.

The Content-Length header should suffice to inform a client where to resume the upload. Whether it's allowed to or not, seems an application-layer concern (beyond authentication), hidden behind the uniform interface.

> Does the server need to send a unique URL to the client?

Thought we were talking Web Arch, where all URIs are unique, and the basis of messaging is a client requesting a representation of the identified server resource. Clients may receive URLs from servers, but I can always type 'em into my user-agent, which is of particular note when we're including PUT in the convo. ;)

> What about the additional roundtrip time?

Wasn't the upload interrupted? Even if it was for a nanosecond, the network connection needs to be re-established... the client needs to re-authenticate with the server... seems we're way beyond one HTTP r/t of latency making any real-world difference.

> tus-v2 draft solves it by defining `Upload-Token` which is a 

> cryptographically-random token generated on the client side.

To save one HTTP round trip when restarting an interrupted download? That's where ya lost me.

> We’ve also looked at the `Range` header but decided 

> against it due to it providing too much flexibility.

That's an interesting way of putting it, thanks! I never even considered using it to solve what I call the "partial-PUT problem" we're dealing with, shoulda left it out of my last email, really only Content-Length matters here.

> Features like multipart ranges are extremely difficult to

> support on the server.

You're preaching to the choir. Granted I've been ranching for the past decade, but before that I was a web developer since 1994 and have coded many a webserver. The fundamentals to which I adhere, have not changed over time. Makes me salty, but if you know where to find any of my old work on archive.org the takeaway should be that it all still functions on the latest browsers. Except the browser-native XSLT stuff.

> That being said, we can revisit this decision if the work group provides a different perspective.

All about consensus and working code. I'd love to link you to a rest-discuss thread about my PUT/PATCH demo, I had a deliberately-broken example image that responded 206 online for years, back in the aughties; archive.org serves it 200 tho. If you ever *really* tried it you were using curl. It helped me grok/explain resource v representation. The broken representation of one resource (a Mona Lisa icon), was itself a first-class, read-only resource.

Most browsers rendered the incomplete image, some displayed their broken-image icon, none choked on the 206. The same representation was available from a PUT/PATCH-enabled, access-restricted URI (allowing it to be fixed, one way or another, without affecting the other URI dedicated to the broken variant), and the discussion was about how PATCH increases in value the larger the file. ARF vs. ARCF, C = Continue.

At some point you've transferred more of the file than you're willing to Retry and overwrite, vs. Continue appending. I guess if you want you can introduce 1xx responses into the mix there, but I'm not seeing it as necessary.

> Overall, we believe that a tightly-scoped standard would benefit

> implementors and encourage wide adoption.

All due respect, I'm seeing a tightly-coupled solution to the ages-old partial-PUT problem, which does not falsify using a 206 response to a non-range request to unambiguously communicate the state of the resource as "incomplete" where Content-Length gives the exact byte where the transfer was interrupted. IMHO, a loosely-coupled approach is better for encouraging wide adoption, under the standard as written.

-Eric

---

From: Julian Reschke <[email-masked]>
Date: Tue, 5 Apr 2022 08:09:32 +0200

Am 05.04.2022 um 08:05 schrieb Eric J Bowman:
> A 206 response to a non-range request uniquely, unambiguously, and
> elegantly identifies an incomplete resource. Identifying a resource as
> both incomplete *and* completeable, introduces tight coupling at the
> protocol layer.
> ...

It's an interesting idea; but we would need to modify the core specs for
that (right now it's only defined for responses to range requests).

Best regards, Julian

---

From: Eric J Bowman <[email-masked]>
Date: Mon, 04 Apr 2022 23:28:21 -0700

---- On Mon, 04 Apr 2022 23:09:32 -0700 Julian Reschke <[email-masked]> wrote ----

 
Best regards, Julian

 -------------------------------------

Exploring the undefined aspects of HTTP to flesh out the core specs, is preferable to me over adding to or even extending HTTP. Y'all never told me *not* to respond 206 on a non-range request. Implementations just don't care about 206 being undefined for this; they just "fall back to 200". As they should. Nothing breaks.

-Eric

---

From: Julian Reschke <[email-masked]>
Date: Tue, 5 Apr 2022 09:38:15 +0200

Am 05.04.2022 um 08:28 schrieb Eric J Bowman:
> ---- On Mon, 04 Apr 2022 23:09:32 -0700 *Julian Reschke
> <[email-masked]>* wrote ----
>
>     Exploring the undefined aspects of HTTP to flesh out the core specs,
>     is preferable to me over adding to or even extending HTTP. Y'all
>     never told me *not* to respond 206 on a non-range request.
>     Implementations just don't care about 206 being undefined for this;
>     they just "fall back to 200". As they should. Nothing breaks.
>
>     -Eric

Hm, no.

If the client gets a 206 for incomplete content and falls back to 2xx
handling, it will assume the content is indeed complete.

Or am I missing something here?

Best regards, Julian

---

From: Guoye Zhang <[email-masked]>
Date: Tue, 05 Apr 2022 00:53:44 -0700

> On Apr 4, 2022, at 11:05 PM, Eric J Bowman <[email-masked]> wrote:
> 
> >

To give you an example, let’s suppose we have an html form like this:

<form action="/upload" enctype="multipart/form-data" method="post">
  <input type="file" name="file">
  <input type="submit" value="Submit">
</form>

When a user click “Submit”, the browser sends a `POST /upload` request with the file content as the request body. However, if the upload interrupts and the client tries `HEAD /upload`, how does the server know which upload the client is referring to?

The server can of course use surrounding information (such as Cookie) to look up the incomplete upload. However, that prevents the same client to submit multiple forms concurrently. Alternatively, the server can also generate dynamic HTML pages to encode a unique identifier in the form action URL, but that prevents caching and doesn’t solve other non-HTML use cases.

`Upload-Token` is designed to resolve this issue by labeling every upload with a unique ID.
> At some point you've transferred more of the file than you're willing to Retry and overwrite, vs. Continue appending. I guess if you want you can introduce 1xx responses into the mix there, but I'm…

Thanks, I will look it up.
> All due respect, I'm seeing a tightly-coupled solution to the ages-old partial-PUT problem, which does not falsify using a 206 response to a non-range request to unambiguously communicate the state…
The recent trend has shifted to define standards with no ambiguity and with strict requirements for implementations, as people recognize that unplanned extensibility makes interop much more difficult. HTTP/3 standard is a good example of that philosophy, whereas if you give the PATCH RFC5789 to 100 people to implement, they will end up with 100 incompatible protocols.

Going back to the current topic. We want to answer these 3 questions, in this specific order:

(1) Is a resumable upload standard necessary?

	I think this is an easy argument. Current state of many incompatible protocols makes it impossible to interop.

(2) Does everyone agree with tus-v2’s main goals? To design a mechanism to upgrade any upload to resumable upload, and to be easily implementable on top of any HTTP libraries.

	Many design decisions are the direct results of these goals, and we hope to gain consensus on these goals and potentially seek out more goals others might have.

(3) What about the protocol details? 204 vs 206, Upload-Offset vs Content-Length vs Content-Range.

	I think we should answer the first two before digging into these, as it’s the least critical and we can easily make changes.

Guoye

---

From: Julian Reschke <[email-masked]>
Date: Tue, 5 Apr 2022 10:11:41 +0200

Am 05.04.2022 um 09:53 schrieb Guoye Zhang:
>> The Content-Length header should suffice to inform a client where to
>> resume the upload. Whether it's allowed to or not, seems an
>> application-layer concern (beyond authentication), hidden behind the
>> uniform interface.
>>
>> >
>> Thought we were talking Web Arch, where all URIs are unique, and the
>> basis of messaging is a client requesting a representation of the
>> identified server resource. Clients may receive URLs from servers, but
>> I can always type 'em into my user-agent, which is of particular note
>> when we're including PUT in the convo. ;)
>>
>> >
>> Wasn't the upload interrupted? Even if it was for a nanosecond, the
>> network connection needs to be re-established... the client needs to
>> re-authenticate with the server... seems we're way beyond one HTTP r/t
>> of latency making any real-world difference.
>>
>> >
> <form action="/upload"enctype="multipart/form-data"method="post">
> When a user click “Submit”, the browser sends a `POST /upload` request
> with the file content as the request body. However, if the upload
> interrupts and the client tries `HEAD /upload`, how does the server know
> which upload the client is referring to?
>
> The server can of course use surrounding information (such as Cookie) to
> look up the incomplete upload. However, that prevents the same client to
> submit multiple forms concurrently. Alternatively, the server can also
> generate dynamic HTML pages to encode a unique identifier in the form
> action URL, but that prevents caching and doesn’t solve other non-HTML
> use cases.
>
>> That's an interesting way of putting it, thanks! I never even
>> considered using it to solve what I call the "partial-PUT problem"
>> we're dealing with, shoulda left it out of my last email, really only
>> Content-Length matters here.
>>
>> >
>> You're preaching to the choir. Granted I've been ranching for the past
>> decade, but before that I was a web developer since 1994 and have
>> coded many a webserver. The fundamentals to which I adhere, have not
>> changed over time. Makes me salty, but if you know where to find any
>> of my old work on archive.org the takeaway should be that it all still
>> functions on the latest browsers. Except the browser-native XSLT stuff.
>>
>> >
>> All about consensus and working code. I'd love to link you to a
>> rest-discuss thread about my PUT/PATCH demo, I had a
>> deliberately-broken example image that responded 206 online for years,
>> back in the aughties; archive.org serves it 200 tho. If you ever
>> *really* tried it you were using curl. It helped me grok/explain
>> resource v representation. The broken representation of one resource
>> (a Mona Lisa icon), was itself a first-class, read-only resource.
>>
>> Most browsers rendered the incomplete image, some displayed their
>> broken-image icon, none choked on the 206. The same representation was
>> available from a PUT/PATCH-enabled, access-restricted URI (allowing it
>> to be fixed, one way or another, without affecting the other URI
>> dedicated to the broken variant), and the discussion was about how
>> PATCH increases in value the larger the file. ARF vs. ARCF, C = Continue.
>>
>> At some point you've transferred more of the file than you're willing
>> to Retry and overwrite, vs. Continue appending. I guess if you want
>> you can introduce 1xx responses into the mix there, but I'm not seeing
>> it as necessary.
>
>> All due respect, I'm seeing a tightly-coupled solution to the ages-old
>> partial-PUT problem, which does not falsify using a 206 response to a
>> non-range request to unambiguously communicate the state of the
>> resource as "incomplete" where Content-Length gives the exact byte
>> where the transfer was interrupted. IMHO, a loosely-coupled approach
>> is better for encouraging wide adoption, under the standard as written.
>>
>> -Eric
>>
> The recent trend has shifted to define standards with no ambiguity and
> with strict requirements for implementations, as people recognize that
> unplanned extensibility makes interop much more difficult. HTTP/3
> standard is a good example of that philosophy, whereas if you give the
> PATCH RFC5789 to 100 people to implement, they will end up with 100
> incompatible protocols.
> ...

Could you elaborate a bit?

Best regards, Julian

---

From: Guoye Zhang <[email-masked]>
Date: Tue, 05 Apr 2022 12:22:46 -0700

> On Apr 5, 2022, at 1:11 AM, Julian Reschke <[email-masked]> wrote:
> 

Oh, it was just an offhand comment on how vague the PATCH standard is, as a response to an earlier comment in the thread:

>>> Client now knows how to PATCH the resource.

I was exaggerating a bit, but I have no idea how to PATCH a resource after reading RFC5789, it just introduces a general concept and leaves nearly everything open to implementations. That’s why I believe we need further standards for any concrete use.

Guoye

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 6 Apr 2022 08:17:04 +1000

> On 6 Apr 2022, at 5:22 am, Guoye Zhang <[email-masked]> wrote:
> 

PATCH intentionally leaves everything up to the media type of the PATCH request, not the implementation. With hindsight, at least one or two well-defined PATCH media types should have been defined at the same time as 5789 - their absence (especially JSON's) created a lot of confusion.

Cheers,

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 6 Apr 2022 08:04:47 +0900

> PATCH intentionally leaves everything up to the media type of the PATCH request, not the implementation. With hindsight, at least one or two well-defined PATCH media types should have been defined…

Has that confusion now cleared, or is there a way to clear it?

Regards,   Martin.

---

From: Austin Wright <[email-masked]>
Date: Tue, 5 Apr 2022 17:45:50 -0700

> On Apr 5, 2022, at 00:38, Julian Reschke <[email-masked]> wrote:
> 

This is also my understanding.

First, it seems the semantics are different enough that we should mint a new status code for this. 206 implies that the response is not the literal content of the resource; the actual content of the resource is found by parsing the multipart/byteranges response.

Second, the client would have to make the request advertising that it understands such a “sparse document”. Maybe use the “Prefer” header for this:

GET /upload-target HTTP/1.1
Prefer: sparse

Finally, I’m not sure I fully understand what the response would indicate exactly. Shall the server actually support “sparse documents” (where some bytes are undefined)? This would allow users to upload segments out-of-order, and in parallel from different uplinks. If the client does not support sparse documents, how should the server respond? (Fill in the undefined regions with zeros?)

Thanks,

Austin.

---

From: Eric J Bowman <[email-masked]>
Date: Tue, 05 Apr 2022 19:58:54 -0700

> Or am I missing something here? 

Well, Julian, I think the likelihood that I've missed something here is way higher (I think my value to the WG at this point is fresh set of eyes, as HTTP no longer rules my daily existence as it once did, hope my experience means something when it comes to my limited participation despite reading 95% of posts), but yes I think you have. If the resource is "broken image file" then it's right and proper for the client to assume 200. Clients "in the know" about the uploading of that image will know better than to make that assumption, and aren't they the only clients that matter as far as interpreting the 206 differently than a 200? Because they're the ones which initiated the upload?

-Eric

---

From: Eric J Bowman <[email-masked]>
Date: Tue, 05 Apr 2022 21:31:00 -0700

>----------------------------

</form>

>------------------------------

Indeed! In your example /upload is a tightly-coupled RPC endpoint; if the request body is the file content why are you using POST instead of PUT? 'HEAD/resource' lets the server know exactly which "upload" is referenced: if it was interrupted, the server knows it, and responds 206. Because REST.

>
>------------------------------

>------------------------------

Or, the client and server could use a uniform interface based on the URI of the actual resource being manipulated via the transfer of representations which precisely indicate application state? So much easier in my experience from a coding and caching perspective, regardless of how browser behavior changes over time. Can we agree to assume the UA is curl, to continue this discussion? Instead of limiting it to the protocol shortcomings of HTML forms as implemented in browsers, which are easily worked around?

>
>------------------------------

>------------------------------

No, I got that. Once again, no offense intended, but this is where you lose me as a REST acolyte. The uniform interface allows well-defined actions against well-defined resources, without all the overhead and complexity of assigning per-action UIDs -- that's some other (SOAPy) architectural style.

>
>------------------------------

>------------------------------

I wish you could, but I know of no archive of the old Yahoo rest-discuss list, or it'd be easy to search "partial PUT and 206" because I think that was my thread title bitd...???

>
>------------------------------

>------------------------------

I prefer to stick with fundamentals which have stood me in good stead for decades (I pick up occasional hired-gun work for HTML, CSS, JS, and even XSLT because the blue-collar IT folks I know deal with XML daily; the first CMS I ever coded was SSJS when it was called Netscape LiveWire and everyone was like, JS is a client-side language). I can't follow trends. Especially not in calving season. No offense taken if you choose to disregard my review of your concept based on I'm out-of-touch, old-fashioned, whatevs; I am what I am, and that's always in favor of loose coupling when it comes to HTTP.

No. 20 years ago, the question was "are resumable uploads necessary" and the answer was including PATCH in the uniform interface. Not that it made me happy, as I was arguing for an APPEND method lol. One of the absolute joys of putting ideas to the rogue's gallery of usual suspects around here, is having most of 'em shot down, and learning from it. Me replying to two active threads simultaneously on this list, is extra-unusual and does not mean I won't r-lurk and say nothing for years, but you've got my goat with this one.

> I think this is an easy argument. Current state of many 

> incompatible protocols makes it impossible to interop.

Heh! :) I think this is amongst the hardest of HTTP arguments or it woulda been settled long ago.

> (2) Does everyone agree with tus-v2’s main goals? To design a

> mechanism to upgrade any upload to resumable upload...

-1

HTTP protocol is agnostic as far as how the representations are created... for any given resource, they may be uploaded, calculated, read from disk, result from a UA dialog, etc. I maintain that the starting point ought to be identifying a resource as incomplete and putting PUT/PATCH in an "Allow" header, problem solved because uniform interface.

>
>------------------------------

>------------------------------

I've never found a use for 203 or 205, but the rest of 200-206... I'm a big fan of 201/202. Because sometimes content needs to wait on moderation, transformation, etc. before a subsequent HEAD reports 200 i.e. successful upload. 204's a cache hit. 206 just needs a bit of fleshing out. Upload-Offset, also useless to me. Content-Range, nice feature if implemented, not relevant here. The only client gonna know what the Content-Length should be, is the one that failed to upload, hence interpreting C-L as "last byte received" on a 206 is obvious to said client, and the resulting action (retry with PUT or append with PATCH) is transparent to all, and really easy to spot in the httpd's logfile.

-Eric

---

From: Eric J Bowman <[email-masked]>
Date: Tue, 05 Apr 2022 22:16:22 -0700

Hi Austin, you ask good questions!

> Finally, I’m not sure I fully understand what the response 

> would indicate exactly. Shall the server actually support 

> “sparse documents” (where some bytes are undefined)? 

It's all up to how the client supports rendering the media type at hand. Regardless of serving my deliberately-broken image files as 200 or 206 for several years, some browsers "filled it in" with blocks of grey, others used their broken-image icon; some browsers went from broken-icon to filled-in image, while others regressed from filled-in image to broken-image icon. Some browsers behaved differently depending on platform, i.e. comes down to image-rendering libraries. All the server can really do, is somehow indicate to the client that the requested representation is incomplete, if the server knows that for a fact.

> This would allow users to upload segments out-of-order,

> and in parallel from different uplinks. If the client does 

> not support sparse documents, how should the server 

> respond? (Fill in the undefined regions with zeros?)

The server only cares about media-type, really. Oh, you need PNG? I have that. Just wanna let you know it's broken! Do with it what you will. I'll allow PUT/PATCH if you no likey and are authorized for those methods.

Distributed out-of-order uploads... Hmmm. I had to give that some thought, but it doesn't change my position. Any number of participating clients contributing to a file upload, just have to agree how to "fill in the blanks" on the returned representation until it's 200 OK.

I think you and mnot are correct that we need better-defined PATCH media types, I believe that's where to solve this problem, but how any media type is rendered has traditionally and properly been a client-side concern in HTTP.

-Eric

---

From: Guoye Zhang <[email-masked]>
Date: Tue, 05 Apr 2022 23:08:03 -0700

> On Apr 5, 2022, at 9:31 PM, Eric J Bowman <[email-masked]> wrote:
> 
> >
I hope we can improve things today, and many upload endpoints are modeled after an HTML form submission today because of its native support in browsers. While I cannot judge the REST-fulness of it, my preference would not be to ask everybody to redesign their applications.

> No. 20 years ago, the question was "are resumable uploads necessary" and the answer was including PATCH in the uniform interface. Not that it made me happy, as I was arguing for an APPEND method lo…
Well, I was still in elementary school 20 years ago so I’m not familiar with the history. APPEND would have been nice.

> Heh! :) I think this is amongst the hardest of HTTP arguments or it woulda been settled long ago.
The state of things is that nearly no upload is resumable, and anyone who needs it invents their own incompatible protocols. Someone else should chime in, as I do not know how to proceed if we cannot even agree on whether this is a problem worth solving.

> HTTP protocol is agnostic as far as how the representations are created... for any given resource, they may be uploaded, calculated, read from disk, result from a UA dialog, etc. I maintain that th…
How do you deliver the “Allow” header? If the upload is interrupted, the server likely hasn’t had a chance to send the response yet. Do you require clients to send an `OPTIONS *` request before uploading?

As an implementor, I don’t think any of these details is so obvious such that it’s not worth documenting. Happy to explain what consideration went into our decisions in the draft, and would be glad to revisit them if you have a concrete alternative.

Guoye

---

From: Austin Wright <[email-masked]>
Date: Wed, 6 Apr 2022 00:53:08 -0700

> On Apr 5, 2022, at 22:16, Eric J Bowman <[email-masked]> wrote:
> 

So then I think a simple modification of my “Partial Uploads” document (draft-wright-http-partial-upload-01 <https://tools.ietf.org/id/draft-wright-http-partial-upload-01.html>) would work well.

First, it defines the message/byterange media type, for making changes to a specific byte range. This is the bulk of the desired functionality, I think.

Second, 2__ Sparse Resource would indicate that the resource has some regions filled in by the server, and might not be valid according to the media type definition. But I’m not confident that all user-agents would safely handle a 2__ Sparse Resource. If the resource represents executable code, the result could be very bad. Maybe I remove this? It seems wrong to me that a server could send back a document it knows is invalid according to the media type. Maybe there should be an error for clients that request a sparse resource without indicating they can support the response.

Finally, to that document, I would add a Prefer header that indicates the client supports sparse resources, and can accept a 206 Partial Content response that excludes any undefined byte ranges (unlike 2__, which would zero out these regions). If the request is a Range request (with a Range header), the server would exclude undefined ranges in its response. The client could specify in the Prefer header if it prefers multipart responses, or only single-part responses with a Content-Range header.

Synchronizing an interrupted upload might look something like this:

> HEAD /upload-target HTTP/1.1
> Prefer: sparse=single

< HTTP/1.1 206 Partial Content
< Content-Range: 400-499/1000
< Content-Length: 100

Note how the sparse=single forces the response to include a Content-Range header (instead of being embedded in a multipart response); and how the presence of this header indicates the server supports sparse resources.

By the way, I’m calling these “sparse resources” in the same way there’s sparse files <https://en.wikipedia.org/wiki/Sparse_file>. Servers could potentially implement sparse resources by directly mapping onto sparse files.

Thanks,

Austin.

---

From: Guoye Zhang <[email-masked]>
Date: Wed, 06 Apr 2022 01:29:09 -0700

> On Apr 4, 2022, at 8:41 PM, Austin Wright <[email-masked]> wrote:
> 
> Hello Marius & WG,
> 

Originally, an earlier internal draft version had “Upload Creation Procedure” and “Upload Appending Procedure” with PATCH. Then we recognized that they are nearly the same thing with all the same requirements, just different offsets, so we merged them into a single “Upload Transfer Procedure”.

We can definitely revisit this decision if the consensus is to adopt and improve PATCH.

Guoye
> [2] https://tools.ietf.org/id/draft-wright-http-partial-upload-01.html
> [3] https://github.com/awwright/http-progress/issues/1#issuecomment-1061314730 for one possible way to use message/byterange with existing, standard HTTP features.
> 
>

---

From: Eric J Bowman <[email-masked]>
Date: Wed, 06 Apr 2022 02:20:08 -0700

>---------------------

>---------------------

My preference would be not asking everybody to restrict their applications to POST based on the limitations of HTML browsers, in an age when any HTTP method is available in some JS library or another. If I want to revive my old browser-native XSLT code, it would be a piece of cake to just "wrap it" a bit different and have it still work -- the XSLT of it hasn't bit-rotted. Given that, and my having changed careers a decade ago, if we're still stuck with just GET and POST when discussing HTTP, I guess I may as well just mosey on back to the herd.

As to RESTfulness, using HTML forms certainly qualifies if the implementations are /resource not /endpoint oriented -- to oversimplify, no big thing to POST my PATCH given proper application architecture.

>---------------------

>---------------------

My hobby protocol, MONTage, eschews the UA in favor of the AI and has SUBMIT/RETRACT methods and so on, but still no APPEND for the same reason PATCH was preferred bitd. The semantics of the operation are the same, regardless of what byte you're starting at, or if the last byte happens to be EOF. So Append doesn't warrant its own method. One of many things I've been legitimately wrong about, architecture-wise.

>---------------------

>---------------------

I don't think anyone has even meant to imply that this isn't a problem worth solving. That being said... 20 years ago we figured _every_ upload was *replaceable* on failure, while realizing PATCH would increase in value hand-in-hand with filesize over time. Reckoning day has arrived! ;) Thanks for your contribution, and I mean that, otherwise I wouldn't bother. If I intended to stick around and argue this I'd go back to using Sylpheed and proper email quoting. Just wanted to point out "prior art" exists re: 206, even if my example code has bit-rotted beyond recovery it's still quite simple and straightforward.

>---------------------

-1

>-----------------

> How do you deliver the “Allow” header? If the upload is

> interrupted, the server likely hasn’t had a chance to send the 

> response yet. Do you require clients to send an `OPTIONS *`

> request before uploading?

Ha! I'm a huge aficionado of OPTIONS, especially with a wildcard, but I don't see how it applies here? An OPTIONS request is not prerequisite to an Allow header in response. I'm also not aware of any restriction to including the Allow header on *any* 2xx response, or on a PUT request, or even a 404/410. So let's work this out:

client1: PUT /resource C-L = 256

server: only receives 128 before connection closes

client2: GET /resource before that connection closes

server: 404 if the PUT was meant to create; otherwise 200/204 with Etag, expected behavior either way

client 1: connection failed 128 bytes in, no server response

server: decides how to handle the 128 bytes received, accepts instead of tossing

client2: GET /resource

server: 206 C-L = 128

client1: is back, HEAD /resource

server: 206 C-L = 128

client1: PATCH /resource 128-256 (or, re-PUT)

server: 2xx

--later--
client2: GET /resource

server: 200 OK

I'm not seeing any problem with that, which in any way shape or form, needs any sort of convoluted approach using 1xx or headers to clarify a fundamental HTTP transaction.

>---------------------

>---------------------

As an implemeter, I have reached a Zen-like acceptance of the fact that I never have been nor will I ever be on the "winning side" in any debate with a browser vendor. I swear to you though, I am not opposed to your idea on the basis of who you work for. I am opposed to it because your considerations have led to an RPC-grafted-onto-HTTP solution. Which may very well see some adoption short-term, but I doubt it'll be there five years down the road, and I believe I have presented a concrete alternative.

-Eric

---

From: Eric J Bowman <[email-masked]>
Date: Wed, 06 Apr 2022 04:06:24 -0700

> So then I think a simple modification of my “Partial Uploads”

> document (https://tools.ietf.org/id/draft-wright-http-partial-upload-01.html) would work 

> well.

Kudos for how well you've written your document. But please don't suggest using PATCH to create a resource? Best decision I ever made regarding httpd coding was only 3 years or so ago, I eschewed Windows compatibility in favor of tight coupling to UNIX filesystem attributes. Speaking in terms of static files for simplicity...

DELETE on my httpd doesn't remove the existing resource, it sets it to zero bytes, such that subsequent GET/HEAD requests respond 410, and may or may not Allow: PATCH. No file? 404, which may Allow: PUT. I can't help but think that a PATCH request to a nonexistent resource should return 404.

> First, it defines the message/byterange media type, 

> for making changes to a specific byte range. This is the 

> bulk of the desired functionality, I think.

I would encourage you to document that media type independently from the rest of your draft.

>---------------

>---------------

Hmmm... I'm a bit rusty at explaining this, but here goes. Media-Type in HTTP has nothing to do with file format. My server intent may be to "view source" of an HTML file, so I use text/plain to prevent it from being parsed and rendered as HTML. My server intent may be to present a broken PNG, so I use image/png to have the client attempt to render it as such. Being a valid representation of said media type is not implied, and I assure you there's nothing wrong with that. If a UA fails to render, it reports the error to the User i.e. "broken image icon". Application-layer error, not protocol-layer error.

I think you've gone astray by essentially modeling "sparse" as a subtype of any given media type. Or qualifying a resource ('s representation) as being the result of executable code -- excluding static files of expected types, 99% of my resources map to server-side processes. Receiving a borked representation has no effect on the executable, because the server-side resource coding is decoupled from its representations, and hidden behind a Uniform Interface.

-Eric

---

From: Austin William Wright <[email-masked]>
Date: Wed, 6 Apr 2022 13:08:16 -0700

On Apr 6, 2022, at 04:06, Eric J Bowman <[email-masked]> wrote:
> > document (draft-wright-http-partial-upload-01 <https://tools.ietf.org/id/draft-wright-http-partial-upload-01.html>) would work 
> > well.
> >
> 

Explain for me how a Unix filesystem makes PATCH impractical?

> DELETE on my httpd doesn't remove the existing resource, it sets it to zero bytes, such that subsequent GET/HEAD requests respond 410, and may or may not Allow: PATCH. No file? 404, which may Allow…

I think it will make more sense if you think of PATCH as a superset of PUT.

PATCH, just like PUT, creates the resource if it doesn’t exist. Consider how these two requests are similar:

PUT /upload-target HTTP/1.1
Content-Length: 400
Content-Type: text/plain

[400 bytes...]

And

PATCH /upload-target
Content-Type: message/byterange

Content-Range: 0-399/400
Content-Type: text/plain

[400 bytes...]

The effects will be identical.

The advantage of the PATCH form is that you can start writing at a non-zero offset, or create the file while indicating it’s larger than the enclosed body. PUT cannot do these.

> Hmmm... I'm a bit rusty at explaining this, but here goes. Media-Type in HTTP has nothing to do with file format. My server intent may be to "view source" of an HTML file, so I use text/plain to pr…

Well, the media type tells you how to decode the response. Yes, if you receive an image/png that is half zeros, it might render (but appear corrupt to a human). But how would the user-agent know this isn’t intentional? How would it know the file is actually incomplete?

It sounded like we would avoid this by responding with a 206 Partial Content response that excludes undefined bytes. I like this solution.

> I think you've gone astray by essentially modeling "sparse" as a subtype of any given media type. Or qualifying a resource ('s representation) as being the result of executable code -- excluding st…

I didn’t intend to suggest this.

Suppose I am uploading an executable binary, how would I ensure that users cannot download it while it’s still incomplete? (Verifying a checksum would prevent major problems, but how would a Web browser know that the resource is incomplete?)

Thanks,

Austin.

---

From: Guoye Zhang <[email-masked]>
Date: Wed, 06 Apr 2022 19:13:19 -0700

> On Apr 6, 2022, at 2:20 AM, Eric J Bowman <[email-masked]> wrote:
> 
> >
> >---------------------
tus-v2 is agnostic to HTTP methods. POST was just picked for the examples because it’s the most commonly-used upload method. You can upgrade any upload operation with any method to resumable upload using the procedures described in the draft.

> I'm not seeing any problem with that, which in any way shape or form, needs any sort of convoluted approach using 1xx or headers to clarify a fundamental HTTP transaction.
There isn’t a huge delta between what you’ve described and what’s currently in the draft. You’ve also identified the same core procedures for resumable upload: creation, offset retrieval, and resumption. The only extra thing we have is cancellation since we don’t want servers to hold on to resources any longer than necessary if clients already decided to give up.

One thing that drove many aspects of the design is our desire to seamlessly upgrade any upload to resumable upload. To us, it’s hugely valuable that a browser or an OS can implement this mechanism under the hood, and all websites and applications would benefit from it without any code change on the client side. This is the reason we have tokens and other newly defined headers, since we don’t want to misinterpret the server supporting something different but happens to use the same header fields.

The reason we are bringing this to workgroup is to look for ways to improve the draft. We like how the draft turned out and how easy it is to implement, as there are already multiple experimental implementations, but we are not experts in HTTP semantics. Changing 204 to 206, adopting Content-Length, or adopting PATCH are the feedbacks we are looking for and I’m happy to discuss them further. However, I would keep strongly advocating for some way to support feature detection and automatic upgrade since it’s one of the core motivations of the draft.

Guoye

---

From: Eric J Bowman <[email-masked]>
Date: Thu, 07 Apr 2022 01:00:11 -0700

Wrong takeaway entirely. I was trying to bring the discussion around to a filesystem paradigm that's been agreed upon since before I was born, because sometimes it helps to do so when discussing HTTP in theory... this is one of those times, I think, unless I've introduced confusion -- in which case I apologize, pragmatism is always my goal.

A resource has to exist first, before it can support PATCH. Leaving it possible to PATCH a DELETEd resource back into existence -- 410 implies a defined resource in a way 404 does not (however, no consensus on my preference there). It just isn't pragmatic to expect consensus around using PATCH to create a new resource, in the face of both POST and PUT having creation semantics which have always been excluded from PATCH... and GET, for that matter.

If pigs had wings they could fly.

>----------------

Content-Length: 400

Content-Type: text/plain

[400 bytes...]

>----------------

200 OK, 201 Created, 202 Accepted...

>----------------

PATCH /upload-target

Content-Range: 0-399/400

Content-Type: text/plain

[400 bytes...]

>----------------

404 Not Found.

Unless you're coding an endpoint instead of a resource, in which case the only help I can offer you, is to think in terms of resources not endpoints. My first AI-coding effort leading to Data and Lore chucking endless rocks, reminded me of the value of Roy's thesis -- sometimes we create things that work in unexpected ways, requiring further study to understand. Which came first, WWW or REST? Now that REST explains the hows and whys of WWW, it makes sense to me to work within that paradigm...

Because of the nature of a globally-distributed IP-based network.

 It's a choice between hacking around reality (Data and Lore throwing endless rocks at one another, if you read my other recent posts) vs. playing to its strengths -- REST explains why the WWW succeeded beyond anticipation, despite being created for a distributed, unreliable network. So much easier to code Data and Lore by starting with a 1-1, reliable network.

Only if you can get consensus on the magical thinking that PATCH has "create" semantics, sorry. That's like saying Data and Lore are identical by virtue of being twins.

> The advantage of the PATCH form is that you can start writing 

> at a non-zero offset, or create the file while indicating it’s larger

> than the enclosed body. PUT cannot do these.

Nor can PATCH create a resource. Nothing wrong with using a zero-byte PUT to create a PATCHable resource, btw, fwiw.

>> Hmmm... I'm a bit rusty at explaining this, but here goes. 

>> Media-Type in HTTP has nothing to do with file format.

> Well, the media type tells you how to decode the response. 

Correct. Isn't that amazing? A jumble of bytes can be successfully interpreted in more than one way! Gopher never grok'd that. TBL certainly earned his knighthood there. Accepting that, has led to intelligent media-type design e.g. JPEG XL still works as image/jpeg.

> But how would the user-agent know this isn’t intentional?

> How would it know the file is actually incomplete?

Because 206 response code.

> Suppose I am uploading an executable binary, how would

> I ensure that users cannot download it while it’s still incomplete?

By coding your server properly? Sorry. Until the server has accepted an upload, it will continue to respond as it did before; maybe 404, maybe 200 with Etag... the Etag changes when the upload completes... If the resource is meant to be the executable file itself, what does it matter if it's incomplete? I have enough trouble dealing with non-corrupted executables to worry what might happen to those I can't compile let alone execute. ;)

But seriously, if you have /resource generating representations, but want to change how those representations are generated by changing /resource and replacing the executable outright... makes sense to go out-of-band from HTTP... which is why webhosting providers still offer FTP.

-Eric

---

From: Eric J Bowman <[email-masked]>
Date: Thu, 07 Apr 2022 01:28:53 -0700

...You can upgrade any upload operation with any method to resumable upload using the procedures described in the draft.

You can also make pigs fly by sewing wings on them, but they'll prolly fly further if you shoot 'em out of a cannon.

> There isn’t a huge delta between what you’ve described 

> and what’s currently in the draft.

We couldn't be further apart, paradigmatically speaking.

>------------------

You’ve also identified the same core procedures for resumable upload: creation, offset retrieval, and resumption. The only extra thing we have is cancellation since we don’t want servers to hold on to resources any longer than necessary if clients already decided to give up.

>------------------

Very frustrating that we mean something different by "resource." Like, Oh, you mean server resources not URIs? Best way to solve that problem is to avoid creating it in the first place, by going against the architectural style by assigning UIDs per transaction.

>-------------------------

>-------------------------

Ugh. See WS-* and tell me how well that all worked out? Sorry to be glib, but tiring of this argument a decade ago literally drove me to droving.

> ...we are not experts in HTTP semantics...

But there's no reason for Apple not to be, in 2022. Same discussions, different names, zero institutional knowledge retained over time... same attitude of we're so-and-so and you're not, so let us have our way because surely we couldn't be wrong about something as simple as web architecture. Sigh.

> ...feature detection and automatic upgrade [are] core motivations of the draft.

Why? You're going down the versioning rabbit-hole. There's no need for /v1/resource, /resource/v1, per-transaction UIDs etc. in your APIs if you're truly embracing Web architecture. If you think I'm pushing back hard, you should meet my goat. 

"Feature negotiation" doesn't chap my hide, but you implementors are rejecting such solutions out-of-hand to save a round-trip. On an unreliable distributed network! Eric.exe has crashed.

-Eric

---

From: Austin William Wright <[email-masked]>
Date: Thu, 7 Apr 2022 10:29:15 -0700

> On Apr 7, 2022, at 01:00, Eric J Bowman <[email-masked]> wrote:
> 
> in the face of both POST and PUT having creation semantics which have always been excluded from PATCH... and GET, for that matter.

PATCH (RFC 5789) specifically permits resource creation <https://httpwg.org/specs/rfc5789.html#patch>:

If the Request-URI does not point to an existing resource, the server MAY create a new resource, depending on the patch document type (whether it can logically modify a null resource) and permissions, etc.

So as long as the PATCH media type can describe creating a resource (which this does), PATCH can create resources.

This is also my understanding of how filesystems work. You don’t have a separate create operation, any write to a file creates it (depending on the flags that are set). You can even write to a nonzero offset and it will create a sparse file.

> Because 206 response code.

Ok, so undefined byte ranges would not be returned by the server, is this your understanding?

> By coding your server properly? Sorry. Until the server has accepted an upload, it will continue to respond as it did before; maybe 404, maybe 200 with Etag... the Etag changes when the upload comp…

All clients see the same resource, correct? Suppose I’m doing a segmented upload, and I finish the first half of the upload. If I make a GET request, I would expect to see 206 showing the defined bytes. But what about other users that don’t support this response? How do I ensure they’re not downloading a mangled file? Should this return a 4xx error if the client doesn’t support partial content responses?

Thanks,

Austin.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 10 Apr 2022 09:58:55 +0200

Am 07.04.2022 um 10:00 schrieb Eric J Bowman:
> Wrong takeaway entirely. I was trying to bring the discussion around to
> a filesystem paradigm that's been agreed upon since before I was born,
> because sometimes it helps to do so when discussing HTTP in theory...
> this is one of those times, I think, unless I've introduced confusion --
> in which case I apologize, pragmatism is always my goal.
>
> A resource has to exist first, before it can support PATCH. Leaving it
> possible to PATCH a DELETEd resource back into existence -- 410 implies
> ...

Says who?

Best regards, Julian

---

From: Austin Wright <[email-masked]>
Date: Sun, 10 Apr 2022 15:31:17 -0700

> On Apr 6, 2022, at 01:29, Guoye Zhang <[email-masked]> wrote:
> 

I understand them to be different things based on how they’re using HTTP semantics. "Upload Appending Procedure” seemed to follow the method semantics. In contrast, "Upload Transfer Procedure" expands each HTTP method to do something it could not previously do: Ignore the head of this request, and instead combine its body with a previous request (violating the understanding that HTTP messages are stateless).

You work around this by requiring that clients not send requests if the origin server would misunderstand the feature. This is never the client’s responsibility in HTTP, because a message may pass through multiple programs and servers, not all of which may understand the expanded semantics. And even with respect to the origin server, signals like a DNS record, or even something as specific as an OPTIONS response on the same URI, cannot technically guarantee that the origin will honor the expanded semantics. I think the difference between using HTTP as a substrate versus using it as an application, is: an HTTP substrate merely re-uses existing libraries (e.g. message & protocol parsers) but not necessarily the semantics, and does not support the entire ecosystem of caches, gateways, and proxies (which collectively form HTTP The Application).

However, I see some somewhat straightforward changes could fix this.

In general, there’s three options to adding a new feature to HTTP:

1. The server has to transparently fall back to acceptable behavior (e.g. 200 OK instead of 206 Partial Content)
2. The server has to produce an error (e.g. unknown method, or unknown media type)
3. It has to be implemented a different layer (e.g. as a feature of TCP, TLS, SCTP, HTTP/2 framing, or QUIC)

I see a combination of these being necessary:

(1) The client should assume no support, and make its first request as normal (requesting support for resumable uploads). If the server supports the feature, it can communicate this back, with a URI representing the attempted operation. This satisfies option (1) transparently fall back.

(2) If the connection needs to be resumed, the client can use a new method, or PATCH with a new media type, on the server’s selected operation URI. This satisfies option (2) produce an error.

(3) Do resumable uploads really need to be specific to HTTP? Suppose there were a feature of HTTP/2 framing, where a client could ask the server “Please generate a UUID for this stream and keep it in the background in the event of a disconnection” or “Please tell me where that stream left off, and resume this stream from there”. This satisfies option (3) different layer.

Finally, your proposal is general and low-level (to the point where the same effect could be achieved in TLS instead). Since virtually all Web applications are implemented above the transport layer, it may be technically difficult to implement, at very little benefit. It follows general stream semantics (ordered bytes), which prohibits features like multiple parallel uploads, which is important as server workloads become increasingly parallel. There would be only a handful of ways this could be implemented in Web applications:

1. The OS or HTTP server implements resumable uploads, and combines multiple HTTP messages in a manner transparent to the written application. If the subsequent HTTP requests hit a different origin server than the first one (a different node in the cluster), it would have to hand-off the request body somehow.

2. The application is written as some sort of state machine that can be handed off or shared between multiple nodes in a cluster. I’m not aware of any development frameworks that do anything remotely like this.

3. Developers choose the specific resources & methods that are likely to benefit from resumable uploads, and describe how the database stores the intermediate progress of each class of operation. This seems like a huge potential for errors due to how infrequently many code paths would be run, and how difficult it is to trigger in tests.

Resumable requests are still reasonable, but I think where we stand to benefit the most are application-layer approaches. In particular, segmented file uploading, which would enable “parallel PUT”.

Thanks,

Austin.

---

From: Glenn Strauss <[email-masked]>
Date: Mon, 11 Apr 2022 06:41:09 -0400

The tus-v2 specification reads to me as an application.  Much thought
and effort has been put into this application, but it is an application
with a protocol for client and server.

Guoye Zhang seems to be proposing tus-v2 to implement
"upload (resumable) *transactions*" and define new behavior that the
server must track and handle partial uploads and be able to report
the sparse areas back to the client.  For that, the rsync protocol,
among others, already exists, but is this necessary?  (For parallel
uploads, it might be, though other protocols like zchunk may help
client discovery of server-side state.)

Large incremental uploads are already achievable using servers which
support partial-PUT, including SabreDAV and lighttpd mod_webdav:
upload file serially in chunks, extending the file with each chunk.
Cancellation is achievable with WebDAV DELETE.  The client can choose
an alternate filename while uploading, and then use WebDAV MOVE to
rename the file into place once the upload is complete.

Support for partial-PUT is a non-standard extension to WebDAV,
at least in part due to implementation tradeoffs (resource usage,
lock timeouts, impacting UX).  tus-v2 may be trying to address this.

A robust PUT implementation (whole file replacement) allows downloads
to proceed while a new version is uploaded.  This may be mapped onto
typical filesystems by uploading to a temporary file and atomically
renaming into place when the upload is complete.  WebDAV LOCK can be
used to ensure only one uploader at a time.

When making changes to a file, whole file replacement works well for
small files and often well-enough for medium-sized files.  It is the
case of large files that network bandwidth, reconnection and re-upload
costs, server disk space, and other resource usage might have a larger
impact.

On modern filesystems with support for cloning, a server might be able
to clone the file extents into a temporary file, PATCH a portion of the
file, and then atomically rename the new file into place.  This could be
done with PATCH or with partial-PUT.  For large files, enforcing use of
WebDAV LOCK is recommended to avoid excessive numbers of large
temporary files, especially if copying a large file to a temporary
file instead of cloning.

Another solution for PATCH-like behavior is to use DVCS protocols,
e.g. git, and serve the completed files from a repository working copy.

>From my reading of the tus-v2 spec, only parallel upload is not
addressed by the solutions above.  Parallel upload and file
reconstruction is something that is currently achievable by
application-specific implementations, including tus-v2, and
potentially by some DVCS.

Eric Bowman makes numerous excellent points in prior messages, and I
would like to repeat one:

Eric J Bowman wrote:
> Unless you're coding an endpoint instead of a resource, in which
> case the only help I can offer you, is to think in terms of resources
> not endpoints.

> Indeed! In your example /upload is a tightly-coupled RPC endpoint;
> if the request body is the file content why are you using POST instead
> of PUT? 'HEAD/resource' lets the server know exactly which "upload" is
> referenced: if it was interrupted, the server knows it, and responds
> 206. Because REST.

If tus is aimed at /upload, an endpoint, then in my mind tus is an
application handling that endpoint.  Given the alternatives mentioned
above for handling resources, I do not see why a web server would
implement tus as an HTTP standard when end-users can configure tus as an
application running behind a web server to handle configured endpoints
such as /upload.

As others have pointed out, there is room for improvement in PATCH,
e.g. defining a new media type and associated behavior for PATCH.

Mark Nottingham wrote:
> PATCH intentionally leaves everything up to the media type of the
> PATCH request, not the implementation. With hindsight, at least one
> or two well-defined PATCH media types should have been defined at the
> same time as 5789 - their absence (especially JSON's) created a lot
> of confusion.

Eric J Bowman wrote:
> I think you and mnot are correct that we need better-defined PATCH
> media types, I believe that's where to solve this problem, but how
> any media type is rendered has traditionally and properly been a
> client-side concern in HTTP.

> I don't think anyone has even meant to imply that this isn't a
> problem worth solving. That being said... 20 years ago we figured
> _every_ upload was *replaceable* on failure, while realizing PATCH
> would increase in value hand-in-hand with filesize over time.
> Reckoning day has arrived! ;) Thanks for your contribution, and I
> mean that, otherwise I wouldn't bother.

Cheers, Glenn

---

From: Guoye Zhang <[email-masked]>
Date: Mon, 11 Apr 2022 12:25:45 -0700

> On Apr 10, 2022, at 3:31 PM, Austin Wright <[email-masked]> wrote:
> 
> 

Yes, this is a very good point. If a middlebox sees “Content-Type: application/json” but does not find a valid JSON in the request body, it might freak out.

It is somewhat addressed by “Clients MUST NOT send representation metadata in subsequent Upload Transfer Procedure requests”, but yes, we definitely need more considerations in this area.
> (2) If the connection needs to be resumed, the client can use a new method, or PATCH with a new media type, on the server’s selected operation URI. This satisfies option (2) produce an error.

These are great suggestions! We will discuss it internally.
> Resumable requests are still reasonable, but I think where we stand to benefit the most are application-layer approaches. In particular, segmented file uploading, which would enable “parallel PUT”.

Parallel segmented uploads were discussed, and we intentionally disallow it in the draft due to the added complexity on the server. We are matching the capability of the current tus protocol with improvements in a few areas.

Let’s suppose in the client’s perspective the connection is dead, but the server is still receiving some bytes trickling in. At this point, the client queries the offset and tries to resume. In many implementations of the current resumable upload protocol, the resumption will fail since the offset is changing and wouldn’t match. Therefore we are clarifying that querying offset should freeze the progress and terminate previous uploads. Unfortunately, this prevents the protocol from supporting parallel uploads, but we think it’s a more concrete problem to solve.

Guoye

---

From: Eric J Bowman <[email-masked]>
Date: Mon, 11 Apr 2022 16:50:08 -0700

All snark aside (sorry about that, Austin, all)...

https://httpwg.org/specs/rfc5789.html#patch:

The reason I think this is bad protocol design to the point it makes my brain explode, is it introduces the notion that method semantics vary by media type. The interface is now resource-specific, rather than generic. Transparency is lost because the message is not self-descriptive, e.g. intermediaries would require a lookup table of media types in order to determine method semantics.

My bad for bringing up OS / FS. A uniform network interface to a collection of resources is its own problem space. An OS doesn't need to worry about an unreliable connection, let alone intermediaries, for filesystem ops.

My understanding is that the server only knows it never received the entire upload. It isn't attempting to render the payload, in order to determine anything about *how* the partial upload breaks due to not being complete, let alone convey that information to the client -- because surely the client can figure that out by diff'ing the 206 response payload.

Mangled representations are a fact of life on the Web. So are 200 OK responses which say "not found" in the message body. If you're concerned that a failed upload shouldn't result in a borked retrieval, don't update the resource. If you want to give user-agents a hint about where the upload failed, I suggest 206; and acceptance of the fact that a 200 OK response doesn't imply a valid representation, which is a feature of Web architecture, not a bug -- tightly-coupled architectures fail to scale, while the Web has scaled exponentially for 30 years by allowing this sort of thing.

-Eric

---

From: Eric J Bowman <[email-masked]>
Date: Mon, 11 Apr 2022 17:39:32 -0700

>>
>> A resource has to exist first, before it can support PATCH. 
>>
 >
> Says who?
>

Common sense? Clearly-defined method semantics is part-and-parcel of a uniform interface. If we're going to muddy the waters by allowing partial PUT (or PUT no content to DELETE), and PATCH to create a primary resource (not sayin' PATCH can't result in a /previous-version resource being minted), then I guess HTML was right all along to only bother defining GET and POST in forms.

> Some origin servers support use of the Content-Range header field

> (Section 14.4) as a request modifier to perform a partial PUT, as

> described in Section 14.5.

Ugh. There's some wording above that, about how a service SHOULD use POST instead of PUT; I'd suggest (sometime in the future if PATCH is ever officially added) similar wording about SHOULD use PATCH. Some origin servers treat a zero-payload PUT as a removal request; similarly, some wording about SHOULD use DELETE could clarify.

I managed to watch a lot of golf this weekend while reviewing the latest HTTP drafts. Aside from the above, the only other editorial suggestion I have would be to add "8.4.1.4 Brotli Coding".

-Eric

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Apr 2022 07:45:36 +0200

Am 12.04.2022 um 02:39 schrieb Eric J Bowman:
> Common sense? Clearly-defined method semantics is part-and-parcel of a
> uniform interface. If we're going to muddy the waters by allowing
> partial PUT (or PUT no content to DELETE), and PATCH to create a primary
> resource (not sayin' PATCH can't result in a /previous-version resource
> being minted), then I guess HTML was right all along to only bother
> defining GET and POST in forms.
 > ...

Well. You are in disagreement with the spec.

The issue here being that "existence" of a resource is somewhat hard to
define.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Apr 2022 07:48:36 +0200

Am 12.04.2022 um 01:50 schrieb Eric J Bowman:
> PATCH (RFC 5789) specifically permits resource creation
> <https://httpwg.org/specs/rfc5789.html#patch>:
>
>
>         If the Request-URI does not point to an existing resource, the
>         server MAY create a new resource, depending on the patch
>         document type (whether it can logically modify a null resource)
>         and permissions, etc.
>
>
>     So as long as the PATCH media type can describe creating a resource
>     (which this does), PATCH can create resources.
>      >-----------------
>      >
>
>
> The reason I think this is bad protocol design to the point it makes my
> brain explode, is it introduces the notion that method semantics vary by
> media type. The interface is now resource-specific, rather than generic.
> Transparency is lost because the message is not self-descriptive, e.g.
> intermediaries would require a lookup table of media types in order to
> determine method semantics.

Not really. The semantics of PATCH is "apply the payload as patch to
whatever the current representation of the resource is". That includes
resources with no current representation.

Best regards, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 12 Apr 2022 11:10:22 -0700

> On Apr 11, 2022, at 10:45 PM, Julian Reschke <[email-masked]> wrote:
> 

I've defined this before. A resource is a mapping of a URI to value over time,
and thus always exists as a function because there is no distinction between
an origin server that doesn't exist, a resource that is not yet mapped by the
origin server (but could be), or the network being down. For example,
OPTIONS can target a resource that has no representation.

A value (the selected representation) is what might or might not exist at
any single point in time for any given request.

The question for PATCH (when I originally proposed it in HTTP/1.1)
was whether a non-existent representation would be treated equivalent
to a zero-length representation for the semantics of PATCH.
I chose to define it as such because that would match the semantics
of the Unix patch command.  Hence, that's the definition.

However, I don't think this discussion is relevant to the proposed upload
mechanism. HTTP's methods are defined by the method spec, not by
opinions on what the name allows, and Julian pointed to the right spec.

The problem with extending old methods to do new tricks is that
such extensions must be defined with respect to what happens when
the extension is ignored. A partially ignored extension can wreak havoc,
both within protocol chains and within single server implementations
that are internally layered with method-specific assumptions.

There are only two options for implementing this while staying
within HTTP. The first is to immediately respond with 202 Accepted
and the instructions for resuming/finalizing the upload if failed.
The second is to immediately send a 1xx response to the client that
directs them to a separate temporary resource corresponding to
*this* request content being uploaded, upon which the client can do
resumable tricks if this request fails.

Both achieve the goal, are method-independent, and don't change
the semantics for deployed practice.

A 202 response changes the normal response flow, which
loses whether the initial request succeeded. This is probably
not desired most of the time, since resuming an upload is rare.

A 1xx response does not change the normal response flow,
but requires that the *client* that indicated support for doing
a resumable upload support also support a 1xx response.
IMO, that trade-off is obvious.

Yes, that requires a protocol chain that supports 1xx responses.
It is completely and utterly irresponsible at this point to allow
NEW FUNCTIONALITY to be supported by stacks that are
somehow incapable of understanding the existing protocol defined
over 25 years ago. They were required to implement support for 1xx
and 100. They will want to implement support for 103. They will have to
implement this new 1xx support if they want a resumable upload
without losing the initial request status, because 1xx is the only
way to communicate method-specific, resource-based options
before a final status code is sent.

If the 1xx is lost, the client will only see the result of the initial
request in a 2xx or 4xx response, which could also include
extensions to indicate where the partial upload is kept and how
to resume/cancel it with further requests. The difference
is that the client wouldn't receive that information if the
connection fails before receiving the final response headers.

....Roy

---

From: Stefan Eissing <[email-masked]>
Date: Wed, 13 Apr 2022 09:13:37 +0200

> Am 12.04.2022 um 20:10 schrieb Roy T. Fielding <[email-masked]>:
> 

+1

---

From: Julian Reschke <[email-masked]>
Date: Wed, 13 Apr 2022 09:16:34 +0200

Am 12.04.2022 um 20:10 schrieb Roy T. Fielding:
>> On Apr 11, 2022, at 10:45 PM, Julian Reschke <[email-masked]
>> <mailto:[email-masked]>> wrote:
>>
> ....

Thanks for the explanation, Roy and +1 on the details.

(I was actually aware of the resource definition, but my experience is
that when I say "there is no such thing as an non-existing resource",
they tune out and walk away).

Best regards, Julian

---

From: Eric J Bowman <[email-masked]>
Date: Tue, 19 Apr 2022 23:39:39 -0700

Roy T. Fielding wrote:

Yes, you have, and I'm glad you used the word "defined" and good luck with OPTIONS on this URI:

http://foo.bar/xyzzy

Theoretically is, and always has been even before I just conjured it, a resource *by definition* because maybe one day it will be defined (and damn if that isn't why the Web works where Xanadu etc. failed, conceptually) and have a representation (I posit that it does not, at the time of this writing). But, it is undefinable by today's standards. Colloquially, it's really difficult to explain when HTTP defines resources as "created" if I also insist that said resource always existed in the first place.

And the Periodic Table is a horrifically outdated representation / model which remains the basis of getting work done IRL, regardless of where an "electron" may "be". Short of rewriting HTTP to use "defined" instead of "created" how can we have this resumable-upload discussion without tripping over our own pedantry, I wonder?

The semantics of my hobbyist MONTage protocol's PATCH method (and media types) are defined by DARCS patch theory. Whilst taking your (indirect) advice and separating non-existent (404 in my book) from zero-length (410). Maybe I should call it PILL, or PITA? lol ;)

I'm not interpreting method semantics based on their names, Roy. My opinion is that the right spec is wrong. I always tried to transcend the limitations of HTML's reliance on GET/POST by adopting other methods, but if those methods all have overlapping semantics I don't see the point? The bug up my butt remains partial PUT becoming acceptable due to *changes* in the spec. My opinion that create/replace semantics deserves its own unsullied method could go by any number of names, and I would hope that doesn't change over time.

My opinion that method semantics shouldn't vary by media type, transcends any given method regardless of its name or intent. I did not see that 10 years ago when arguing against RFC 5789. At the remove I have now, it's so obviously wrong I feel I shouldn't even need to explain it, but yeah, that's just my opinion. I hate that my pragmatic advice to the youngsters is stick with GET/POST.

Exactly why I'm taking a purist approach in my opposition to PATCH creating resources? What has been discussed in this thread, is a new media type for that, over a decade on since PATCH started getting used for... well... patching. I don't see a problem with that "method-specific assumption" at all, it's been de facto for a long while.

> within HTTP...

I'm done ranching, my herd is sold as of October and I'm on to my next opportunity, which still leaves me a strongly-opinionated HTTP hobbyist. Playing Devil's Advocate,  in this thread, Austin and Guoye are discussing multiple clients completing a failed upload. Hence my 206 suggestion, rather than stating there is no HTTP solution to the problem at hand?

> The first is to immediately respond with 202 Accepted

Both assume only one client attempting to upload. In which case I agree with you entirely. In other cases I'm fine with deferring to your wisdom, but not without question, no disrespect intended.

I've always considered 202 as "comment awaiting moderation" which may take anywhere from a few moments, to never, whether the upload succeeded or not. Out here in the sticks, taking a pic and texting it to someone rarely succeeds. If I have to re-try, I'd rather my UA picks up where it left off. Estonia has better connectivity than rural-mountain Colorado (optical fiber getting close lately tho).

With you there. May not seem like it given my previous responses to this topic, but I guess I was assuming some generic client like libcurl that reports the 1xx and quits instead of continuing, my bad.

So, you're -1 on distributed uploads from multiple clients? Granted I don't have any use case for this, but apparently others do. Not sure what you mean by "final" status code. My years-long example 206 broken-gif file could at any time have been fixed or replaced to achieve a "final" status of 200 OK. Because I thought I understood the temporal nature of a resource (or its definition I guess).

And how are other clients informed of the partial resource, if at all, seems to be another problem folks are trying to solve. I have another suggestion using weak validators like I used to, but I'm too damn tired. Calving season went off without a hitch, but the past week has been nothing but blizzards and single-digit temps, now is when folks around my area are busy sending hay by the truckload to Churchill Downs, busy busy busy.

-Eric

---

From: Eric J Bowman <[email-masked]>
Date: Fri, 22 Apr 2022 04:46:07 -0700

Here's a better example/questions. Given the following existing-but-undefined resource (please click through):

http://example.com/xyzzy

Wouldn't sending PATCH "404 Not Found" be reasonably expected to change the 404 response body accordingly?

(no context switching)

As opposed to "creating" a resource which eventually responds 200 OK with "404 Not Found" as its message body?

(context switching)

A DARCS patch file may contain instructions for line changes, find/replace, or filesystem ops. I've created two media types, text/vnd.darcs and text/vnd.darcs-inverse. I played around with just the first using AMEND/REVERT methods, but chose to stick with PATCH and create the -inverse media type.

If sent as a PATCH request, line-change/find-replace are interpreted as amend/revert based on media type. Filesystem ops are ignored.

(no context switching)

If sent as a POST request, line-change/find-replace are ignored. Filesystem-op instructions are followed or undone, based on media type.

(context switching)

If sent as a PUT request, I don't care which vnd.darcs media type, as the patch itself is a first-class resource in DARCS. Which either overwrites (no context switching) or creates (context switching), based on method definition -- the patch is not applied, no partial PUT.

I consider that RESTful implementation of HTTP protocol, iff the PUT target comes from the server.

I would consider it bad practice (and hard to maintain) to create text/vnd.darcs-execute to allow partial PUT (overriding the method definition) by media type, or to allow PATCH to engage in context-switching (overriding separation of concerns) by media type. A patch file shouldn't be executable, just an instruction set for an executable.

I consider those REST mismatches, for subverting the Uniform Interface constraint.

My opinion is informed by DARCS being patch-centric, as opposed to the typical snapshot-centric revision control systems or UNIX diff/patch. One or more "context" resources must first be defined, in order for any patch to have meaning. A find/replace foo/bar patch may be applied to any context. A client can derive any snapshot from a context plus one or more patches; no server-derived snapshot URI needed. Roll your own repo and define that snapshot as a context, perhaps assigning it a URI. Just a stored procedure.

DARCS is just recursive lambda calculus expressed as Haskell functions, grok that at the risk of hating git. ;) It almost solves for distributed resumable uploads: conflict management is the cutting edge of DARCS development.

Restricting PATCH to just mean "patch" and disallowing partial PUT simplifies the security scheme to "user/group by protocol method, per naming authority" copacetic with peer-to-peer distributed-repository operations. Anyone can GET/HEAD/OPTIONS/PATCH. Operators can approve (apply) a PATCH. Creators may define contexts. Etc. This is significantly harder to implement if PUT or PATCH semantics vary by media type!

Wouldn't application-layer concerns bleeding through to the protocol layer like that, indicate a layered-system mismatch? Abiding by the uniform interface constraint would preclude the layering violation. I see value in acknowledging REST mismatches even if it changes nothing (cookies still won't die even with the UI pollution legislatively mandated for using them). Serves to inform implementers who care about getting the most REST benefits HTTP has to offer, of BCP expressed as what not to do.

-Eric