Subject: HTTP Instance Digests and encoding

From: Lucas Pardue <[email-masked]>
Date: Wed, 16 May 2018 10:24:56 +0000

Hi,

We’ve been looking at HTTP Instance Digest as defined in RFC 3230. The document seems to skim over a detail that is causing us some internal debate when considering compression like gzip. The question boils down to whether a digest should be calculated on the pre-compressed object or post-compressed one.

Section 4.2 states:

The digest is computed on the entire instance associated with the message. The instance is a snapshot of the resource prior to the application of any instance manipulation or transfer-coding (see section 3). The byte order used to compute the digest is the transmission byte order defined for the content-type of the instance.

Note: the digest is computed before the application of any instance manipulation. If a range or a delta-coding [9] is used, the computation of the digest after the computation of the range or delta would not provide a digest useful for checking the integrity of the reassembled instance.

Section 3 defines the relevant items:

   instance          The entity that would be returned in a status-200
                     response to a GET request, at the current time, for
                     the selected variant of the specified resource,
                     with the application of zero or more content-
                     codings, but without the application of any
                     instance manipulations or transfer-codings.

   instance manipulation

                     An operation on one or more instances which may

                     result in an instance being conveyed from server to

                     client in parts, or in more than one response

                     message.  For example, a range selection or a delta

                     encoding.  Instance manipulations are end-to-end,

                     and often involve the use of a cache at the client.

In our usage, resources are compresses with gzip and have an accompanying Content-encoding: gzip response header. Treating this as an instance manipulation is beneficial, because we allows a server to precompute the digest independent of the format of the compression used to deliver the resource.

Kind regards
Lucas

----------------------------

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.

---------------------

---

From: Amos Jeffries <[email-masked]>
Date: Thu, 17 May 2018 01:00:31 +1200

On 16/05/18 22:24, Lucas Pardue wrote:
> Hi,
> 
>  
> 
> We’ve been looking at HTTP Instance Digest as defined in RFC 3230. The
> document seems to skim over a detail that is causing us some internal
> debate when considering compression like gzip. The question boils down
> to whether a digest should be calculated on the pre-compressed object or
> post-compressed one.
> 

Do not confuse Transfer-Encoding and Content-Encoding.

* T-E is only part of the message on-wire framing/encoding.

* C-E is a variant of the origin's resource.

> 
> Section 4.2 states:
> 
>  
> 
> The digest is computed on the entire instance associated with the
> message. The instance is a snapshot of the resource prior to the
> application of any instance manipulation or transfer-coding (see section
> 3). The byte order used to compute the digest is the transmission byte
> order defined for the content-type of the instance.
> 
>  
> 
> Note: the digest is computed before the application of any instance
> manipulation. If a range or a delta-coding [9] is used, the computation
> of the digest after the computation of the range or delta would not
> provide a digest useful for checking the integrity of the reassembled
> instance.
> 
>  
> 

... in terms *additional* to the RFC 2616 (now 723x) definitions for
HTTP terminology.

>                      instance manipulations or transfer-codings.

ie. C-E:gzip is considered part of the "instance". T-E:gzip is not. This
is consistent with HTTP definitions of the encoding types.

> In our usage, resources are compresses with gzip and have an
> accompanying Content-encoding: gzip response header. Treating this as an
> instance manipulation is beneficial, because we allows a server to
> precompute the digest independent of the format of the compression used
> to deliver the resource.

Each of your objects has two digests to compute. Just like it has two
deliverable variants for each resource; C-E:gzip and C-E:identity (aka
no C-E).

If you want to use gzip as instance manipulation that makes it
Transfer-Encoding, *not* Content-Encoding.

Amos

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 21 May 2018 09:46:30 +0000

Hi Amos,

Thanks for the response! This clarifies things for me.
  
> deliverable variants for each resource; C-E:gzip and C-E:identity (aka no C-E).
> 
> If you want to use gzip as instance manipulation that makes it Transfer-
> Encoding, *not* Content-Encoding.

To ask a simple question using an example, is the following behaviour valid?

1) An origin server is configured to serve the file example.txt with either C-E:identity or C-E:gzip. 
2) Client requests example.txt with Accept-Encoding: gzip request header.
3) Server performs "on-demand" gzip compression, server calculates an instance digest on the results. Server may cache the result of this operation (implementation and configuration dependent).
4) Server adds Content-Encoding and Digest response headers.

Kind regards
Lucas

---

From: Julian Reschke <[email-masked]>
Date: Mon, 21 May 2018 13:48:59 +0200

> 4) Server adds Content-Encoding and Digest response headers.

AFAIR, there were arguments about whether RFC 3230 actually got the 
terminology right (it tries to clarify RFC 2616 but IMHO added even more 
confusion).

If you are serious about this, you may want to update RFC 3230 based on 
the terminology in RFC 723*.

Best regards, Julian

---

From: Lucas Pardue <[email-masked]>
Date: Mon, 21 May 2018 19:08:24 +0000

Hi Julian,

> AFAIR, there were arguments about whether RFC 3230 actually got the terminology right (it tries to clarify RFC 2616 but IMHO added even more confusion).

> If you are serious about this, you may want to update RFC 3230 based on the terminology in RFC 723*.

Digest offers several benefit to end-to-end content integrity, transport integrity alone (TLS etc) still allows some error in the chain to flip something that could break the clients. This thread was prompted by an edge case in configuring an experiment. Given some of the ambiguities that seem open to creative interpretation, broad interop might be interesting :)

I'm not sure I understand the problems completely such that I could update 3230. Hypothetically, if I were to try that, I might wait for HTTPter to complete.

Lucas

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 22 May 2018 15:10:01 +1000

Hey Lucas,

Note that this is very old and hasn't been adopted, but Jeff Mogul's paper here might help illuminate the thinking behind "instances":
  http://wwwconference.org/proceedings/www2002/refereed/444/index.html

Cheers,

> On 16 May 2018, at 8:24 pm, Lucas Pardue <[email-masked]> wrote:
> 
> 
> Hi,
>  

--
Mark Nottingham   https://www.mnot.net/