Subject: Digest: use in requests

From: Roberto Polli <[email-masked]>
Date: Mon, 28 Dec 2020 18:57:59 +0100

Dear all / chairs,

I think this is the last relevant issue to face in Digest, and
we really need your support. To reduce noise, I tried to tackle it
directly with @Julian Reschke - which I have to thank for all the support.

This discussion has been lingering around since IETF-106 so it would be great
to tackle it.

The point is related to whether Digest on requests containing
partial representations should be computed on the payload-data
or on the representation data. The actual Digest formulation derives
from RFC3230 and always uses representation-data.

 The gh-issue is:
-  https://github.com/httpwg/http-extensions/issues/970

Besides this point - I would be even fine to close it with no-action
to avoid introducing inconsistencies with RFC3230 - and a minor
algorithm issue, this spec is imho ready for a WGLC.

Thanks for your support and have a nice day,
R.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 28 Dec 2020 19:28:50 +0100

Am 28.12.2020 um 18:57 schrieb Roberto Polli:
 > ...

And this is where the problem starts.

Could you please *define* what a request "containing a partial
representation" is, and how a component can detect that case?

Furthermore, a real-world example would be useful.

Best regards, Julian

---

From: Roberto Polli <[email-masked]>
Date: Tue, 29 Dec 2020 10:56:54 +0100

Hi @Julian Reschke,

replies in-line.

TL;DR: if http forbids partial representations in requests, this discussion
will be solved without changing the spec.

Il lun 28 dic 2020, 19:33 Julian Reschke <[email-masked]> ha scritto:
> representation" is,

This is the first question: can a request payload-data convey a
partial representation?
My understanding of semantic-latest §6.4 is: yes, but if we think it
is not appropriate
we could clarify that. See:

- https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#rfc.section.6.4.p.1
- https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#rfc.section.6.4.2.p.1

> and how a component can detect that case?

As Digest is an end-to-end integrity mechanism, the intended components
are not supposed to be agnostic intermediaries. Instead, they are expected
to have some knowledge about the purpose of the payload (eg. in case
of id-* algorithms they
need to content-decode the payload to validate it, or in case of Range
Requests).

Reading https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#rfc.section.6.4.1.p.1
```The purpose of a payload in a request is defined by the method semantics```
iiuc the receiver, aware of the request semantic, knows its purpose
and how to process it, including whether it conveys a partial
representation or not.

> Furthermore, a real-world example would be useful.

I cannot mention existing standards conveying partial representations
in requests,
but I think this is not a reason for changing the representation-data-digest
mechanism defined in continuity with RFC3230 and specifying here a behavior
(no partial representations in request) that is in the domain of
httpbis-semantics.

Besides coherence with RFC3230, imho if a future method/header allows
to convey a partial representation
in a request, it should be possible to use Digest to convey the
checksum of the complete representation.

Thanks to everybody for your feedback,
R.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 29 Dec 2020 11:24:16 +0100

> need to content-decode the payload to validate it, or in case of Range

Hm, that seems like an odd choice for a protocol spec. If the spec
doesn't say what the Digest means for any request, it's not really
defining a protocol.

I would *hope* that we can define things so that Digests can
automatically produced and checked by user agents (browsers) and servers
(such as a servlet container).

> and how to process it, including whether it conveys a partial

But "partial repesentation" is a term defined by HTTP; there is (or
should be) an algorithm that - when inspecting *any* HTTP message -
tells you whether it's "partial" or not. In HTTP, this is defined by the
appearance of "Content-Range" for some specific response status codes.

> mechanism defined in continuity with RFC3230 and specifying here a behavior

I'm not convinced that continuity with RFC 3230 is important, but could
you be a bit more specific here?

> checksum of the complete representation.

It *really* would be good to discuss something *concrete* here.

Let's consider an upload protocol that sends multiple chunks, and then
lets the server combine these into the final resource.

In that protocol, Digest on each chunk would be use to check the
integrity of each chunk.

For the final step of creating the final full resource, the client could
send the expectec digest of the final resource in a *custom* field
defined for the upload protocol (it would use the same algorithms etc,
but use a different way to convey it to the server).

With that, generic libraries could at least verify Digests on each of
the chunks.

Best regards, Julian

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 29 Dec 2020 12:17:34 +0000

Hi Julian,

Just adding my 2c as responses in-line:

> (such as a servlet container).

FWIW, subresource integrity (SRI) is implemented in browsers. The specifics
are different, the hash applies to the identity encoding, so UAs need to
reverse any content encoding before validation. The fundamentals carry over
so it should be possible but I've not seen any signals that browsers are
interested in automatic Digest validation (yet?).

> > Reading
> https://httpwg.org/http-core/draft-ietf-httpbis-semantics-latest.html#rfc.section.6.4.1.p.1

> <snip>

This is indeed the most likely use case. A very quick survey indicates that
there seem to be some examples of PUT requests with Content-Range in the
wild. I have no experience with these, nor knowledge of how popular they
actually are.

* Amazon S3 Glacier [1]

"This multipart upload operation uploads a part of an archive. You can
upload archive parts in any order because in your Upload Part request you
specify the range of bytes in the assembled archive that will be uploaded
in this part."

Example:
PUT /AccountId/vaults/VaultName/multipart-uploads/uploadID HTTP/1.1
Host: glacier.Region.amazonaws.com
Date: Date
Authorization: SignatureValue
Content-Range: ContentRange
Content-Length: PayloadSize
Content-Type: application/octet-stream
x-amz-sha256-tree-hash: Checksum of the part
x-amz-content-sha256: Checksum of the entire payload
x-amz-glacier-version: 2012-06-01

* Google Drive [2]

"Upload the content in multiple chunks. Use this approach if you need to
reduce the amount of data transferred in any single request. You might need
to reduce data transferred when there is a fixed time limit for individual
requests, as can be the case for certain classes of Google App Engine
requests."

"Add these HTTP headers:
    Content-Length. Set to the number of bytes in the current chunk.
    Content-Range. Set to show which bytes in the file you upload. For
example, Content-Range: bytes 0-524287/2000000 shows that you upload the
first 524,288 bytes (256 x 1024 x 2) in a 2,000,000 byte file."

* Google Cloud Storage [3]

"This page describes how to make a resumable upload request in the Cloud
Storage JSON and XML APIs. This protocol allows you to resume an upload
operation after a communication failure interrupts the flow of data."

Example:
curl -i -X PUT --data-binary @CHUNK_LOCATION \
    -H "Content-Length: CHUNK_SIZE" \
    -H "Content-Range: bytes
CHUNK_FIRST_BYTE-CHUNK_LAST_BYTE/TOTAL_OBJECT_SIZE" \
    "SESSION_URI"

* draft-wright-http-partial-upload-01 (expired) [4]

"This document specifies a new media type intended for use in PATCH
   payloads that allows a resource to be uploaded in several segments,
   instead of a single large request."

Example:
PATCH /uploads/foo HTTP/1.1
   Content-Type: message/byterange
   Content-Length: 283
   If-Match: "xyzzy"
   If-Unmodified-Since: Sat, 29 Oct 1994 19:43:31 GMT

   Content-Range: bytes 100-299/600
   Content-Type: text/plain
   Content-Length: 200

Finally, Dropbox [5] does things a little differently and uses the
Dropbox-API-Arg JSON header field to communicate a cursor containing an
offset of the bytes uploaded so far (which I guess means that parallel
transfers aren't supported).

Example:
curl -X POST https://content.dropboxapi.com/2/files/upload_session/append_v2
\
    --header "Authorization: Bearer"
    --header "Dropbox-API-Arg: {\"cursor\": {\"session_id\":
\"1234faaf0678bcde\",\"offset\": 0},\"close\": false}"
    --header "Content-Type: application/octet-stream"
    --data-binary @local_file.txt

To conclude, I'm not exactly sure how these examples influence the
discussion. It seems that there are actually concrete cases of "partial
requests" but it's unclear to me if these break HTTP semantic rules and/or
if it should be documented for formally. The examples I've seen are for
APIs that also have their own custom means for integrity checks, or still
use Content-MD5. It would be nice if something like Digest covered all
avenues and we could get folks to switch to it, but I've not seen any
signals that such APIs are interested in Digest. Therefore, I'm wary of
Digest taking on too much work to describe something without any
implementer interest. In the interest of progress, if partial requests for
uploads is something people think needs standardising, I think that could
be done as an independent follow-on work item e.g. a document that updates
Digest.

Cheers
Lucas

[1] -
https://docs.aws.amazon.com/amazonglacier/latest/dev/api-upload-part.html
[2] -
https://developers.google.com/drive/api/v3/manage-uploads#http---multiple-requests
[3] - https://cloud.google.com/storage/docs/performing-resumable-uploads
[4] - https://tools.ietf.org/html/draft-wright-http-partial-upload-01
[5] -
https://www.dropbox.com/developers/documentation/http/documentation#files-upload_session-append:2

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 29 Dec 2020 12:35:40 +0000

Missed one, would not be surprised if there are more:

* Microsoft OneDrive [1]

"To upload the file, or a portion of the file, your app makes a PUT request
to the *uploadUrl* value received in the *createUploadSession* response.
You can upload the entire file, or split the file into multiple byte
ranges, as long as the maximum bytes in any given request is less than 60
MiB.

The fragments of the file must be uploaded sequentially in order. Uploading
fragments out of order will result in an error."
Example
PUT https://sn3302.up.1drv.com/up/fe6987415ace7X4e1eF866337
Content-Length: 26
Content-Range: bytes 0-25/128

[1] -
https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_createuploadsession?view=odsp-graph-online#example

On Tue, Dec 29, 2020 at 12:17 PM Lucas Pardue <[email-masked]>
wrote:

> FWIW, subresource integrity (SRI) is implemented in browsers. The
> specifics are different, the hash applies to the identity encoding, so UAs
> need to reverse any content encoding before validation. The fundamentals
> carry over so it should be possible but I've not seen any signals that
> browsers are interested in automatic Digest validation (yet?).
>
>
>> > Reading
> This is indeed the most likely use case. A very quick survey indicates
> that there seem to be some examples of PUT requests with Content-Range in
> the wild. I have no experience with these, nor knowledge of how popular
> they actually are.
>
> * Amazon S3 Glacier [1]
>
> curl -X POST
> https://content.dropboxapi.com/2/files/upload_session/append_v2 \

---

From: Julian Reschke <[email-masked]>
Date: Tue, 29 Dec 2020 14:38:52 +0100

Am 29.12.2020 um 13:17 schrieb Lucas Pardue:
> that there seem to be some examples of PUT requests with Content-Range
> in the wild. I have no experience with these, nor knowledge of how
 > ...

Note:

"An origin server that allows PUT on a given target resource MUST send a
400 (Bad Request) response to a PUT request that contains a
Content-Range header field (Section 4.2 of [RFC7233]), since the payload
is likely to be partial content that has been mistakenly PUT as a full
representation. ..." --
<https://greenbytes.de/tech/webdav/rfc7231.html#rfc.section.4.3.4.p.11>

>     Content-Range: bytes 100-299/600

FWIW, use of Content-Range in PATCH is unspecified.

> https://content.dropboxapi.com/2/files/upload_session/append_v2
> <https://content.dropboxapi.com/2/files/upload_session/append_v2> \
> requests" but it's unclear to me if these break HTTP semantic rules
> and/or if it should be documented for formally. The examples I've seen
 > ...

The only concept that HTTP itself for "partial" is defined in RFC 7233.
This is for responses. It *could* be extended to requests, but the spec
is very very clear that it can not be used for PUT.

If there's interest in changing the above requirement, now would be the
time to raise this.

> are for APIs that also have their own custom means for integrity checks,
> or still use Content-MD5. It would be nice if something like Digest
> covered all avenues and we could get folks to switch to it, but I've not
> seen any signals that such APIs are interested in Digest. Therefore, I'm
> wary of Digest taking on too much work to describe something without any
> implementer interest. In the interest of progress, if partial requests
> for uploads is something people think needs standardising, I think that
> could be done as an independent follow-on work item e.g. a document that
> updates Digest.
> ...

Yes.

In any case, the protocol needs to state what a recipient ought to do
when a message has a "Digest" field on it. We can't just say "it depends".

Best regards, Julian