Subject: Re: Updating Digest header RFC3230 using "selected representation"

From: Rob Sayre <[email-masked]>
Date: Tue, 23 Jul 2019 13:12:30 -0700

This draft looks good to me.

One issue that prevented the adoption of this header and related ones (e.g.
Content-MD5) was that HTTP/1.1 trailers had a lot of interoperability
problems. This made it expensive for large files, or impossible to use for
message bodies that couldn't be pre-calculated.

I think using trailers should work these days, right? From what I
understand, this might be implied in the header model of HTTP2+, but I
think addressing this issue in the draft might be helpful.

At first glance, this header looks redundant given that HTTP is now done
over TLS, but I've seen this kind of feature catch bugs behind TLS
termination many times.

thanks,
Rob

On Tue, Apr 16, 2019 at 2:57 AM Lucas Pardue <[email-masked]>
wrote:

> I think you're on the right track.
>
> Present a salient problem statement, seek others also interested, prepare
> some text that address the problem (Google doc is ok to start with,
> focusing on content not formatting). At some point we can get an individual
> Internet Draft prepared (markdown GitHub workflow is relatively
> straightforward) and see if a working group like HTTPbis has interest in
> adopting it for further work / publication.
>
> Lucas
>
> On Tue, 16 Apr 2019, 10:44 Roberto Polli, <[email-masked]>
> wrote:
>
>> Hi Lucas,
>>
>>
>> Il giorno mar 16 apr 2019 alle ore 11:33 Lucas Pardue
>> <[email-masked]> ha scritto:
>> > On Tue, 16 Apr 2019, 10:29 Roberto Polli, <
>> [email-masked]> wrote:
>> >> as the Digest header is still used, I'm willing to refresh the old spec
>> >>
>> >> - https://tools.ietf.org/html/rfc3230
>> >>
>> >> using the new definitions of RFC723x (eg. selected representation)
>> >> and untagling it from delta-encoding.
>>
>> > Yes I would like to see an update.
>> > This spec is no longer consistent with other HTTP docs, and leads to
>> some circular arguments.
>> > Despite that I think it still has a valid use.
>> Great! Here's an editable google docs:
>>
>>   -
>> https://docs.google.com/document/d/1p8KBR_dQKfh7PgLTOYXg_htMAME8nnEblKRC0DSH5_o/edit
>>
>> > I'd like to see the document updated to align with the current HTTP
>> core work, I'm happy to help if I can.
>> I need some guidance on the ietf workflow (Eg. proposing the draft,
>> ..) as it's the first time
>> I'm doing that :D
>>
>> I hope that the technical side can converge quic-ly :D
>>
>> Have a nice day,
>> R:
>>
>>
>>
>>
>> >>
>> >> Hi @all,
>> >>
>> >>
>> >> Is there somebody interested in supporting that effort?
>> >>
>> >> Thanks and have a nice day,
>> >> R.
>> >>
>> >> --
>> >> Roberto Polli
>> >> Full Stack Developer
>> >> M. +39 3406522736
>> >> ·D
>> >> TEAM PER LA
>> >> TRASFORMAZIONE
>> >> DIGITALE
>> >> Presidenza del Consiglio dei Ministri
>> >> teamdigitale.governo.it
>> >>
>> >> Il Team per la Trasformazione Digitale, salvo eccezioni, comunica con
>> >> le altre Amministrazioni via posta elettronica ordinaria e non posta
>> >> elettronica certificata, in conformità a quanto previsto dall’art.47
>> >> del Codice dell’Amministrazione Digitale.
>> >>
>>
>>
>> --
>> Roberto Polli
>> Full Stack Developer
>> M. +39 3406522736
>> ·D
>> TEAM PER LA
>> TRASFORMAZIONE
>> DIGITALE
>> Presidenza del Consiglio dei Ministri
>> teamdigitale.governo.it
>>
>> Il Team per la Trasformazione Digitale, salvo eccezioni, comunica con
>> le altre Amministrazioni via posta elettronica ordinaria e non posta
>> elettronica certificata, in conformità a quanto previsto dall’art.47
>> del Codice dell’Amministrazione Digitale.
>>
>

---

From: Rob Sayre <[email-masked]>
Date: Tue, 23 Jul 2019 13:25:27 -0700

On Tue, Jul 23, 2019 at 1:12 PM Rob Sayre <[email-masked]> wrote:

> One issue that prevented the adoption of this header and related ones
> (e.g. Content-MD5) was that HTTP/1.1 trailers had a lot of interoperability

That last part is slightly wrong. I should have said it was impossible to
use for message bodies that couldn't be pre-calculated that were also
streamed. Roughly, these headers required a lot of buffering, and this
trade-off doesn't seem to be present anymore.

thanks,
Rob

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 23 Jul 2019 22:41:02 +0100

Hi Rob!

> This draft looks good to me.
Thanks.

I think using trailers should work these days, right?

HTTP/2 makes carriage of trailers a bit easier but their
applicability/usage really depends on which endpoints we are considering. I
defer to Dragana Damjanovic's nice salient description [1]

"Clients have been hesitant to implement HTTP trailers for years because of
security and consistency concerns with the trailers general case. Browsers
seek to render response bytes as soon as possible, but rendering is the
result of combining the response data with the semantics of the response
headers (e.g., the content-type header informs how a response is
displayed). Generic trailers allow header information to be added after the
rendering has begun, and this has traditionally created concerns about both
correctness and security for supporting trailers."

> trade-off doesn't seem to be present anymore.

Speaking personally, there is still a challenge of buffering here.
Streaming transfer does not strictly mean streaming consumption.
Progressive consumption implies an ability for progressive digest
calculation, and if that cannot happen then I don't know if sending the
digest as trailer does address trade-offs.

That said, documenting trade offs could be useful for people looking to use
Digest header. At IETF 105 we were asked to add some use cases to the
document. I wonder if you might have a candidate along the lines of this
discussion that we might add for consideration.

Kind regards
Lucas

[1]  https://www.fastly.com/blog/supercharging-server-timing-http-trailers

---

From: Rob Sayre <[email-masked]>
Date: Tue, 23 Jul 2019 18:08:38 -0700

> digest as trailer does address trade-offs.

I agree that the digest is meaningless if you're doing progressive
rendering, since you can't calculate it until you have the whole message.
But you can calculate the digest as a trailer if you're doing progressive
production.

Sending the digest as a trailer reduces costs for the sender, whether it's
an upload or download. If the message is small, it's also likely that TLS
is good enough in that case.

thanks,
Rob

---

From: Lucas Pardue <[email-masked]>
Date: Tue, 23 Jul 2019 22:41:54 -0400

> is good enough in that case.

I broadly agree but there are several nuances here that may not be
immediately obvious to the passing reader. So I'm inclined to write
something up.

I disagree that TLS is good enough, it provides integrity protection for
only a single transport hop. Digest allows for integrity protection of the
object over multiple hops and, importantly, at rest.

Cheers
Lucas

---

From: Rob Sayre <[email-masked]>
Date: Tue, 23 Jul 2019 19:46:03 -0700

On Tue, Jul 23, 2019 at 7:42 PM Lucas Pardue <[email-masked]>
wrote:

> Digest allows for integrity protection of the object over multiple hops
> and, importantly, at rest.
>

I agree with this idea very much. It's why I think the header should be
updated, and why I bothered to send feedback.

looking forward to the next revision,
Rob

---

From: Roberto Polli <[email-masked]>
Date: Wed, 24 Jul 2019 11:05:57 +0200

Hi Rob,

thanks for your feedback! I try to reply per-point.

0- Digest use cases
> Rob: If the message is small, it's also likely that TLS is good enough in that case
> Lucas: Digest allows for integrity protection of the object over multiple hops and, importantly, at rest.
Lucas is right. Digest is mainly used to provide  a basic form of
non-repudiation of origin.

1- send a digest header with the "payload checksum"
2- send a signature protected http message where the signature covers
the digest header;
3- store the message, digest and signature.

See https://github.com/httpwg/http-extensions/blame/master/draft-ietf-httpbis-digest-headers.md#L129
See https://github.com/httpwg/http-extensions/blame/master/draft-ietf-httpbis-digest-headers.md#L686-L690

1- about HTTP/1.1 trailers.
> One issue that prevented the adoption of this header and related ones (e.g. Content-MD5) was that
> HTTP/1.1 trailers had a lot of interoperability problems
> Sending the digest as a trailer reduces costs for the sender, whether it's an upload or download. .

If you mean https://tools.ietf.org/html/rfc7230#section-4.4, they
apply to transfer coded messages, so iiuc
the only reasonable way to send Digest in a trailer was if the message
was transfer-coded from the
beginning by the Origin-Server. Do you agree?

2- about huge payload
> This made it expensive for large files, or impossible to use for message bodies that couldn't be pre-calculated.
I stubbed a reply in the section "Server resource exhaustion for
calculating Digest on huge representation-data"
of  https://github.com/httpwg/http-extensions/issues/852#issuecomment-514541274
You can comment there!

Suggestion and clarifications hints are very welcome!

Your comments are very interesting
and appreciated!

Thanks Rob,
R:
--

---

From: Rob Sayre <[email-masked]>
Date: Wed, 24 Jul 2019 13:11:39 -0700

On Wed, Jul 24, 2019 at 2:06 AM Roberto Polli <
[email-masked]> wrote:

> (e.g. Content-MD5) was that

Yes, I think I agree. Additionally, I wanted to point out that this did not
work very well in practice.

It seems like a trailer might work quite a bit better over HTTP/2 and later.

> 2- about huge payload

I think the key point is that you can feed bytes to a hash function
incrementally--in fact, this is the exact use case given in RFC 7230,
section 4.4. I don't think it's a security concern, but I think it would
help to point out that the sender (server or client), should probably
provide this data in a trailer (or equivalent).

thanks,
Rob