Subject: intermediaries, implicit gzip, etags, no-transform

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Jun 2014 17:12:20 +0200

Hi there,

so I played around with Google's Data Compression proxy and Opera's 
Turbo today, and found:

- both implicitly use content-coding gzip, but do not touch the Etag, 
potentially breaking HTTP semantics

- GDCP adds a custom header field describing the original content length 
(which of course has not been registered)

- setting "cache-control: no-transform" on the response, does not change 
this behavior, thus both seem to violate a "MUST NOT" requirement from 
<http://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.5.7.2>

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Fri, 20 Jun 2014 10:04:03 -0700

> <http://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.5.7.2>

Good thing for them that there are no protocol police.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Jun 2014 20:44:07 +0200

> Good thing for them that there are no protocol police.

Nobody expects the protocol police.

---

From: James M Snell <[email-masked]>
Date: Fri, 20 Jun 2014 12:19:46 -0700

One would think that, by now, the benefits of actually following the
specs would be well understood. Not changing the Etag when applying
content-coding has a very real impact; as does not following proper
cache-control requirements. (and yes, sadly, google is the only one
with this problem).

---

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 20 Jun 2014 22:15:17 +0200

> <http://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.5.7.2>

Both GDC and Turbo communicates between endpoints of known  
implementations. MUST and MUST NOTs are really only relevant seen from the  
outside, i.e. beyond the proxy. The exception would be intermediaries  
looking at this traffic, which both protocols assumes there are none of.

The cache-control: no-transform is really only set on ad images, so it is  
the semantic equivalent to the high priority flag on emails.

The actual implementation of Turbo varies quite a lot between versions btw.

/Martin Nilsson

-- 
Using Opera's mail client: http://www.opera.com/mail/

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Jun 2014 22:23:05 +0200

On 2014-06-20 22:15, Martin Nilsson wrote:
> On Fri, 20 Jun 2014 17:12:20 +0200, Julian Reschke
> <[email-masked]> wrote:
>
>> Hi there,
>>
>> - GDCP adds a custom header field describing the original content
>> length (which of course has not been registered)
>>
>> - setting "cache-control: no-transform" on the response, does not
>> change this behavior, thus both seem to violate a "MUST NOT"
>> requirement from
> implementations. MUST and MUST NOTs are really only relevant seen from
> the outside, i.e. beyond the proxy. The exception would be
> intermediaries looking at this traffic, which both protocols assumes
> there are none of.

If the origin server say "no-transform", and the UA receives a 
transformed payload, that's non-conforming. I don't believe there's any 
wiggle room here.

> The cache-control: no-transform is really only set on ad images, so it
> is the semantic equivalent to the high priority flag on emails.

The spec is agnostic about what kind of payloads it's used on. It might 
be true that *in practice* it's mostly used for these, but that doesn't 
change the protocol requirement to leave the payload alone when it's set.

> The actual implementation of Turbo varies quite a lot between versions btw.

I tried with whatever the current Opera desktop version uses when I 
select the "Turbo option".

Best regards, Julian

---

From: Michael Piatek <[email-masked]>
Date: Fri, 20 Jun 2014 13:34:33 -0700

Hi Julian,

I'm an engineer at Google working on the compression proxy service.
Responses inline.

In general, we're happy to get bug reports from the community, but
it's difficult to track them in mailing lists. If you believe you've
found a bug, please do file an issue at http://crbug.com/ and mention
the data compression proxy in your report. We'll see it.

> potentially breaking HTTP semantics

If you have specific examples of real sites broken by this behavior,
please let me know and we'll investigate.

> <http://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.5.7.2>

This will be fixed. Thanks for the report.

-Michael

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Jun 2014 22:41:04 +0200

> please let me know and we'll investigate.

We just (two weeks ago) killed implicit gzip in HTTP/2, so we're pretty 
sure it's a serious problem.

In the meantime, can you define what you consider a "real" site?

> This will be fixed. Thanks for the report.

Thanks.

Best regards, Julian

---

From: Michael Piatek <[email-masked]>
Date: Fri, 20 Jun 2014 13:53:10 -0700

> In the meantime, can you define what you consider a "real" site?

A site that breaks when using the data compression proxy that is not
contrived to do so.

>>> <http://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.5.7.2>

Actually, an example URL would be helpful for this case as well. In my
testing, I observe the proxy respecting the no-transform directive;
i.e., response payloads carrying cache-control: no-transform are not
modified, nor are the protected headers.

---

From: Johnny Graettinger <[email-masked]>
Date: Fri, 20 Jun 2014 17:08:08 -0400

> sure it's a serious problem.

My recollection (and that of the meeting minutes
<https://github.com/httpwg/wg-materials/blob/master/interim-14-06/minutes.md>)
was
that we killed implicit gzip because there are gnarly HTTP/1 spec
conformance issues, and a similar outcome can be achieved if first-mover
HTTP/2 deployments refuse HTTP/2 requests with a-e: identity. I don't
recall it being substantiated as a "serious problem" in that conversation.

cheers,
-johnny

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Jun 2014 23:14:17 +0200

> modified, nor are the protected headers.

Is "Content-Encoding" in your set of "protected headers"?

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Jun 2014 23:17:39 +0200

On 2014-06-20 23:08, Johnny Graettinger wrote:
>     We just (two weeks ago) killed implicit gzip in HTTP/2, so we're
>     pretty sure it's a serious problem.
>
>
> <https://github.com/httpwg/wg-materials/blob/master/interim-14-06/minutes.md>) was

The serious problem is that changing the content-coding while not 
changing the ETag can break conditional requests and range requests. 
That falls under my definition of "serious".

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Fri, 20 Jun 2014 14:19:26 -0700

On 20 June 2014 14:08, Johnny Graettinger <[email-masked]> wrote:
> My recollection (and that of the meeting minutes) was that we killed
> implicit gzip because there are gnarly HTTP/1 spec conformance issues, and a
> similar outcome can be achieved if first-mover HTTP/2 deployments refuse
> HTTP/2 requests with a-e: identity. I don't recall it being substantiated as
> a "serious problem" in that conversation.

We killed it so that we could avoid having to talk about it any more.
That worked out well, didn't it?

---

From: Julian Reschke <[email-masked]>
Date: Fri, 20 Jun 2014 23:26:55 +0200

> That worked out well, didn't it?

Absolutely.

Not having implicit GZIP seems to be undesirable performance-wise. 
Having it is undesirable because of broken HTTP semantics.

Time to finally fix T-E?

Best regards, Julian

---

From: "Martin Nilsson" <[email-masked]>
Date: Fri, 20 Jun 2014 23:52:06 +0200

> transformed payload, that's non-conforming. I don't believe there's any  

That depends on where you draw the line for the UA box. If you split the  
browser into multiple processes, are they allowed to send different  
representation of the data between each other? What if you move some  
processes to different hosts?

It also depends on what your view is on the authority over data. Should  
the origin server overrule the user agents decision? If the user agent  
explicitly asked for compressed data, should the origin server be allowed  
to overrule that? Should no-transform defeat all network security  
appliances?

> change the protocol requirement to leave the payload alone when it's set.

In the case of current Turbo it's not HTTP so the protocol requirement  
doesn't apply.

/Martin Nilsson

---

From: Michael Piatek <[email-masked]>
Date: Fri, 20 Jun 2014 14:52:04 -0700

Yup, Content-Encoding is protected. (I'm looking at the list in 2616
sec 13.5.2). I see that the proxy is gzipping text no-transform
responses. Thanks.

Are you aware of any sites that break given this behavior? Often,
misconfigured origins simply mark all responses as no-transform, and
it would be unfortunate to lose the compression benefits of gzip given
that it's typically benign.

On Fri, Jun 20, 2014 at 2:14 PM, Julian Reschke <[email-masked]> wrote:
>>>>> change

---

From: Matthew Kerwin <[email-masked]>
Date: Sat, 21 Jun 2014 10:59:47 +1000

On 21 June 2014 07:26, Julian Reschke <[email-masked]> wrote:

> Not having implicit GZIP seems to be undesirable performance-wise. Having
> it is undesirable because of broken HTTP semantics.
>
> Time to finally fix T-E?
>
>
Yes please! Every time it comes up there's a big outcry of "but nobody uses
it!"... which is disheartening because the cry comes from the very people
who _could_ be using it.​

C.f. <https://issues.apache.org/bugzilla/show_bug.cgi?id=52860>

Although Roy's been trying to convince Apache for quite a while, it seems...

2009:
"The best solution is to implement transfer-encoding as an
http protocol filter module."
<https://issues.apache.org/bugzilla/show_bug.cgi?id=39727#c31>

2006:
"The best solution is to not mess with content-encoding at all, which
gets us out of both this consistency problem and related problems
with the entity-header fields (content-md5, signatures, etc.).
That is why transfer encoding was invented in the first place."
<
http://mail-archives.apache.org/mod_mbox/httpd-dev/200612.mbox/[email-masked]%3E

-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Amos Jeffries <[email-masked]>
Date: Sat, 21 Jun 2014 15:58:13 +1200

On 21/06/2014 9:52 a.m., Martin Nilsson wrote:
> On Fri, 20 Jun 2014 22:23:05 +0200, Julian Reschke wrote:
> 
>>
>> transformed payload, that's non-conforming. I don't believe there's
>> any wiggle room here.
> 
> explicitly asked for compressed data, should the origin server be
> allowed to overrule that? Should no-transform defeat all network
> security appliances?

no-transform is also used in the medical industry to protect sensitive
data. I wonder how many patients are getting wrong disgnosis due to
compression artifacts in their scan result images when sent to medical
experts via these services?
 I would hope none due to the medical IT departments due diligence on
the services used. But one never knows when an idiot will appear.

Amos

---

From: Julian Reschke <[email-masked]>
Date: Sat, 21 Jun 2014 08:21:37 +0200

On 2014-06-20 23:52, Martin Nilsson wrote:
> On Fri, 20 Jun 2014 22:23:05 +0200, Julian Reschke

What's relevant is what gets out of the UA, which includes code using 
XMLHttpRequest.

> allowed to overrule that? Should no-transform defeat all network

It's up to the origin server to decide whether it wants to support the 
requested content-coding. If it does not, *and* sends c-c no-transform, 
that's what it is.

And no, no-transform doesn't defeat network security appliances. Those 
appliance just can't transform *and* send a 2xx status.

Finally, let's not forget 
<http://greenbytes.de/tech/webdav/rfc7234.html#rfc.section.5.5.6>:

"5.5.6 Warning: 214 - "Transformation Applied"

This Warning code MUST be added by a proxy if it applies any 
transformation to the representation, such as changing the 
content-coding, media-type, or modifying the representation data, unless 
this Warning code already appears in the response."

Another thing I haven't seen in the HTTP responses I saw in Chrome and 
Opera.

>> The spec is agnostic about what kind of payloads it's used on. It
>> might be true that *in practice* it's mostly used for these, but that
>> doesn't change the protocol requirement to leave the payload alone
>> when it's set.
>

Are users who opt in to use of Opera Turbo aware that this can break 
existing protocol semantics?

Best regards, Julian

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sat, 21 Jun 2014 16:47:24 +0000

>experts via these services?

If no-transform is all that's protecting your or me from getting
diagnosed with a pregnancy, then the medical industry shouldn't be
allowed to get anywhere near computers to begin with.  (Yes, I did
read that study which showed they are clueless morons across the
board when it comes to reliability, security and integrity).

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Martin Nilsson" <[email-masked]>
Date: Sat, 21 Jun 2014 22:52:53 +0200

> What's relevant is what gets out of the UA, which includes code using  

And we can know if the request is coming from XHR. We can also know if the  
resource is inspected, saved, etc. since it is our UA code. I say "can"  
here since what we do differs between iterations. We are not currently  
mirroring the javascript state on the server side, but we have the code.

> Another thing I haven't seen in the HTTP responses I saw in Chrome and  

That status code doesn't appear to be registered with IANA yet, oddly  
enough (I'm looking at  
http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml  
). The response code doesn't have any semantic value here though, since  
the UA already knows that the resource is transformed. It does however  
look like a low risk to add.

> existing protocol semantics?

I think they are aware that using software features can result in bugs.

/Martin Nilsson

---

From: "Martin Nilsson" <[email-masked]>
Date: Sat, 21 Jun 2014 22:52:51 +0200

> the services used. But one never knows when an idiot will appear.

For these kind of use cases you need more end to end due diligence, or  
you'll could have problems at any layer with gamma settings, image library  
precision etc.

Isn't everyone running DICOM?

/Martin Nilsson

---

From: Greg Wilkins <[email-masked]>
Date: Sun, 22 Jun 2014 00:02:34 +0200

Julian,

did you notice if they correctly set the Vary header to indicate the
content can change depending on Accept-Encoding?

cheers

On 20 June 2014 17:12, Julian Reschke <[email-masked]> wrote:

> Hi there,
>
> so I played around with Google's Data Compression proxy and Opera's Turbo
> today, and found:
>
> this behavior, thus both seem to violate a "MUST NOT" requirement from <
> http://greenbytes.de/tech/webdav/rfc7230.html#rfc.section.5.7.2>
>
> Best regards, Julian
>
>
>
>
>

-- 
Greg Wilkins <[email-masked]>
http://eclipse.org/jetty HTTP, SPDY, Websocket server and client that scales
http://www.webtide.com  advice and support for jetty and cometd.

---

From: "Eric J. Bowman" <[email-masked]>
Date: Sat, 21 Jun 2014 17:50:55 -0600

"Martin Nilsson" wrote:
> 
> And we can know if the request is coming from XHR. We can also know
> if the resource is inspected, saved, etc. since it is our UA code.
>

And intermediaries? How are they to know any of this, since it isn't
*their* UA code?

>
> The response code doesn't have any semantic value here though, since
> the UA already knows that the resource is transformed.
>

Maybe not to you, but without those semantics, how can intermediaries
participate in the communication? That's where the semantic value comes
in -- even if it isn't relevant to you, interoperability is relevant to
the other components you don't have control over, is the point here.

-Eric

---

From: "Eric J. Bowman" <[email-masked]>
Date: Sat, 21 Jun 2014 17:50:23 -0600

"Martin Nilsson" wrote:
> 
> It also depends on what your view is on the authority over data.
> Should the origin server overrule the user agents decision?
>

Yes; with my content-publisher hat on, only I can be authoritative
about my data. If we make user-agents authoritative, why should I
bother to publish if I then have no mechanism to be authoritative about
my own content, when necessary?

Doing what the user-agent wants is a good tiebreaker, so to speak, but
not to the extent that it argues in favor of making anyone but the
publisher authoritative about the data.

-Eric

---

From: "Martin Nilsson" <[email-masked]>
Date: Sun, 22 Jun 2014 02:08:46 +0200

> the other components you don't have control over, is the point here.

The protocol is not designed to interop. The latest version released is a  
SPDY derived protocol not running on port 80. We plan to start encrypting  
it soon, if we aren't already in the latest release.

/Martin Nilsson

---

From: "Martin Nilsson" <[email-masked]>
Date: Sun, 22 Jun 2014 02:16:18 +0200

> my own content, when necessary?

I'm only talking about browsers here. The final interpretation of the data  
lies with the browser, and is not only built into browsers where addons  
and configuration changes modifies this, but into standards itself, like  
user style sheets for CSS.

/Martin Nilsson

---

From: "Eric J. Bowman" <[email-masked]>
Date: Sat, 21 Jun 2014 20:05:12 -0600

"Martin Nilsson" wrote:
> 
> I'm only talking about browsers here. The final interpretation of the
> data lies with the browser, and is not only built into browsers where
> addons and configuration changes modifies this, but into standards
> itself, like user style sheets for CSS.
> 

I'm not _not_ talking about browsers, but there's a big difference with
user style sheets, which can be shut off if the user feels something's
not right, or being hidden, or whatever. The final *display* of the
content is up to the browser, but what that content consists of must be
up to the content publisher. If that hadn't been the case on the Web up
to this point, everyone would still be using Compuserve and AOL -- at
least I'm not willing to discount the possibility that publisher
control over content was the undoing of those walled gardens. Without
publisher control, perhaps the Web would've died an untimely death
before ever intermediaries were needed to make it scale, intermediaries
which theoretically allow the publisher to maintain control of their
content and not have that authority usurped by what's best for browsers.

-Eric

---

From: Julian Reschke <[email-masked]>
Date: Sun, 22 Jun 2014 09:47:20 +0200

On 2014-06-21 22:52, Martin Nilsson wrote:
> ...
>> content-coding, media-type, or modifying the representation data,
>> unless this Warning code already appears in the response."
>>
> http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml ).
> The response code doesn't have any semantic value here though, since the
> UA already knows that the resource is transformed. It does however look
> like a low risk to add.
> ...

That's a warn code, not a status code.

> ...

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 22 Jun 2014 09:49:20 +0200

> content can change depending on Accept-Encoding?

I'll try to come up with a good set of tests, and that's a good thing to 
add...

Best regards, Julian

---

From: "Martin Nilsson" <[email-masked]>
Date: Sun, 22 Jun 2014 21:39:01 +0200

> That's a warn code, not a status code.

We certainly have products where it would make sense adding this.

/Martin Nilsson

---

From: Julian Reschke <[email-masked]>
Date: Mon, 23 Jun 2014 13:56:48 +0200

> I think they are aware that using software features can result in bugs.

Indeed.

As a matter of fact, one reason why I started this thread was that I 
want us (the WG) to agree on what are bugs and what aren't, with the 
goal to have the bugs actually fixed at some point.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 23 Jun 2014 14:26:46 +0200

> sec 13.5.2). I see that the proxy is gzipping text no-transform

Note that RFC 2616 is obsolete, and that section doesn't even exist in 
the new specs anymore (see 
<https://trac.tools.ietf.org/wg/httpbis/trac/ticket/60> for context).

> that it's typically benign.

I would argue that "punishing" people who set no-transform when they 
don't nee do is an incentive for them to check what they're doing. As a 
matter of fact, it's not up to the proxy to try to determine whether 
no-transform was sent due to incorrect configuration or stupidity.

Best regards, Julian