Subject: #445: Transfer-codings - Alternate proposal

From: <[email-masked]>
Date: Tue, 8 Apr 2014 15:37:48 +0000

Below I've attempted to capture...

- the problems we are trying to solve with transport codings, and

- the issues with transport codings brought up in the threads.

Problems solved with transport codings...

  a) Allows compression of identity range requests [1-4]

  b) De-emphasizes on-the-fly C-E & associated issues [2, 5-6]

  c) Allows extensibility to new/better compression techniques  [7]

  d) Allows compression by intermediaries (e.g. Roy suggests CDNs could be more efficient this way [8]) (e.g. Amos seems to claim this can be a security benefit by doing hop-specific compression [9])

Issues with using transport codings...

  0) Various security issues (e.g. DoS) [10-18].

  1) Compression at a hop without context about the data. [18-22]

  2) Requires a frame format change* [23]

  3) 2-byte tax in every frame* [23]

  4) Results in a low compression factor if done frame-by-frame* [24]

  5) Reintroduces hop-by-hop headers** [9, 25-26]

  6) Poses a problem for adding signed headers in a future rev** [9]

  7) Implementation complexity [15]

  8) Disputed benefit of compression by CDNs [17]

 9) Many seem to simply hate TE/Transfer-Encoding (for whatever reason) [22-23, 27]

*Specific to Matthew's proposal [28]

**Specific to our proposal [29]

Below we suggest two potential solutions to these issues. Other suggested solutions are welcome. (Also, please also let me know if you think I failed to capture any of the issues already brought up.)

A potential Solution

- Issues 2-4 are w.r.t. Matthew's proposal so I'll let him address those concerns.

- We could simply solve issues 0, 1, 5, 6, 8 by adding a special restriction in HTTP/2 on Transfer-Encoding headers disallowing changes at any hop.  Essentially making Transfer-Encoding end-to-end. This is a bit ugly though because it changes semantics and would likely cause confusion. (Matthew also mentioned doing something similar for his proposal, but he didn't get much positive support [30].)

- I have no answer for issue 9.  Haters are gonna hate.

A potentially better Solution

- A variation on the above solution would be to introduce a new end-to-end coding called "Message-Encoding".  The key difference between "Content-Encoding" (C-E) and "Message-Encoding" (M-E) would be that C-E comes before the Range is applied and M-E comes after the Range is applied.

  + M-E would allow seeks (i.e. range requests) on the identity data and allow the partial content to be compressed.

  + The decision to compress is in exactly the same place as the decision for C-E. The server will be able to decide the most suitable technique for a particular situation (i.e. C-E or M-E).

  + M-E would also be the better, more natural, choice for on-the-fly compression because range requests wouldn't require the entire content to be compressed just to serve a portion of it.

  - The only downside is that intermediaries couldn't add compression, but nobody seems to be fighting strongly for that (ourselves included).

Although I know HTTP/2 isn't allowed to change the existing semantics of HTTP, it's not clear if adding HTTP/2 specific semantics is allowed??

References

[1] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1179.html Keith

[2] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0075.html Matthew

[3] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0104.html Matthew

[4] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1188.html Roland

[5] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1199.html Roland

[6] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1214.html Matthew

[7] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0067.html Matthew

[8] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1189.html Roy

[9] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0098.html Amos

[10] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0071.html Martin

[11] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0083.html Martin

[12] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0078.html Michael

[13] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0079.html Michael

[14] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0088.html Michael

[15] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0103.html Roberto

[16] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0105.html Roberto

[17] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1197.html Mark

[18] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0087.html Michael

[19] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0086.html Martin

[20] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1204.html Martin

[21] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0089.html Michael

[22] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1221.html Poul-Henning

[23] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0065.html Mark

[24] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0100.html Roland

[25] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0084.html Martin

[26] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0082.html Martin

[27] http://lists.w3.org/Archives/Public/ietf-http-wg/2014JanMar/1206.html Patrick

[28] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0059.html Matthew

[29] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0076.html Keith

[30] http://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0080.html Matthew

This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 09 Apr 2014 04:35:20 +1200

> - I have no answer for issue 9.  Haters are gonna hate.

-1 Downvote from me on this.

Making TE end-to-end effectively makes it a redundant duplicate of CE.

>   + M-E would also be the better, more natural, choice for on-the-fly compression because range requests wouldn't require the entire content to be compressed just to serve a portion of it.

This seems to be missing the point about Range requests problem.
Part of the bigger problem is that intermediary cachesneed to store the
messages and expect to be able to server sub-ranges out of it to meet
client requests. Whether they received a full object or not this is true.

For example:
 client #1 requests bytes 1000-4000 of a 5KB object at some URI.
 this gets stored by the cache.
 time passes...
 client #2 requests bytes 1500-2000 from the same URI.

 cache should be able to serve that client #2 Range request without
fetching an entirely new message from the origin.

Avoiding re-encoding it for client #2 would be a bonus, but is not
semantically compatible with any HTTP/1 hops in that direction.

>   - The only downside is that intermediaries couldn't add compression, but nobody seems to be fighting strongly for that (ourselves included).

I suspect nobody here is working on satellite link networks. Squid
project is regularly getting requests/pressure from those groups to do
things like force compression, improved caching, etc. anything that
might reduce bandwidth and need to transfer full copies of objects.

> Although I know HTTP/2 isn't allowed to change the existing semantics of HTTP, it's not clear if adding HTTP/2 specific semantics is allowed??

As I understand it we can only add new semantics for cases where there
is no interaction with HTTP/1. ie on the single HTTP/2 hop.

Amos

---

From: <[email-masked]>
Date: Thu, 10 Apr 2014 09:46:42 +0000

Hi Amos-

Thanks for your response. Since there is obviously little support for this proposal, we are putting our support behind Matthew's proposal.

However, I did want to respond to a couple of your comments, for completeness and because you bring up a few interesting points I hadn't thought of...

On Tuesday,08 April 2014 18:35, [email-masked] wrote:
>> ...  Essentially making Transfer-Encoding end-to-end. This is

>> a bit ugly though because it changes semantics and would likely cause

>> confusion. ...

> Making TE end-to-end effectively makes it a redundant duplicate of CE.

I disagree.  There would still be a subtle, but important, difference between TE and CE.

Content-Encoding is applied *before* a range is extracted.

- from RFC2616 Section 14.35.1: "Byte range specifications in HTTP apply to the sequence of bytes in the entity-body ..."

- from RFC2616 Section 7.2.1: "entity-body := Content-Encoding( Content-Type( data ) )"

Transfer-Encoding is applied *after* a range is extracted.

- from RFC2616 Section 4.3: "message-body = entity-body | <entity-body encoded as per Transfer-Encoding>"

As a result, there is no way to extract a range of the Identity content coding of data (i.e. original form) and then compress the result without using TE.  Although TE is not widely adopted in 1.1, it is part of the spec and we use it heavily for this very purpose.  Since HTTP/2 got rid of TE/Transfer-Encoding, we can't do that anymore.  Hence our fight for some sort of encoding that can be applied after a range is extracted.

>> ...

>> ...

> Part of the bigger problem is that intermediary cachesneed to store the messages and expect to be able to server sub-ranges out of it to meet client requests. Whether they received a full object or not this is true.

> ...

It seems we have two different range problems (ours is explained above).  (Also, we do not use proxies, so my knowledge here is limited...) In this case I would say the intermediary would remove the message encoding before caching and serve sub-ranges from that - optionally re-compressing with message encoding for whatever sub-range is requested that the proxy can fulfil.

>>   - The only downside is that intermediaries couldn't add compression ...

> I suspect nobody here is working on satellite link networks. Squid project is regularly getting requests/pressure

> from those groups to do things like force compression, improved caching, etc. anything that might reduce bandwidth

> and need to transfer full copies of objects.

Actually we do use satellite link networks.  We even still have to use modems to some countries.  So we can empathize with this exact problem.  (We just don't use intermediaries.)