Subject: Optional Connection fields in 1xx messages and Upgrade requests

From: Alex Rousskov <[email-masked]>
Date: Wed, 14 Oct 2020 19:52:37 -0400

On 10/14/20 5:33 PM, Amos Jeffries wrote at
https://github.com/squid-cache/squid/pull/732#issuecomment-708672207:

> Your response answers my question about testing. That clearly indicates
> a buggy Voice service. If you can please document your tests and report
> to the relevant Google dev if you can find them and/or the HTTPbis WG so
> the spec authors can be aware of this case for the latest spec update.

I am not sure adjusting HTTP specs can help in this case, but I am
documenting the problem here per Amos request.

After adding support for HTTP Upgrade mechanism to Squid, we have
encountered several services that fail the Upgrade "handshake" (and,
hence, do not switch to a new protocol) only because Squid sent an
"extra" Connection header field or field value. There are two known cases:

* 101 (Switching Protocols) with "Connection: upgrade, keep-alive"

Client applications using websocket_client[1] library refused to upgrade
upon receiving a "Connection: upgrade, keep-alive" header field in an
HTTP/1.1 101 (Switching Protocols) response. The (unnecessary)
keep-alive addition was the culprit. I do not know whether sending two
Connection header fields would have worked better.

[1] https://pypi.org/project/websocket_client/

* HTTP/1.1 GET request with "Connection: keep-alive"

A Google Voice service[2] rejected Upgrade requests with a 400 (Bad
Request) response if the _last_ Connection header field did not have an
"upgrade" value. The current behavior has changed to resetting the TCP
connection.

[2] https://web.voice.telephony.goog/websocket

Both cases have one thing in common -- they follow HTTP RFC examples to
the letter but do not handle "extra" Connection headers in the
Upgrade-related messages (that those examples do not and should not have).

It may be a good idea to emphasize that both the upgrade-seeking request
and a 101 response may include additional Connection headers, especially
Connection: keep-alive. The agent participating in the upgrade MUST
support enough of HTTP to not be confused by additional Connection
headers (at least). The HTTP Upgrade mechanism is an HTTP mechanism.

HTH,

Alex.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Oct 2020 05:21:21 +0200

Hi Alex,

> headers (at least). The HTTP Upgrade mechanism is an HTTP mechanism.

Sadly I'm not surprised. That reminds me the (painful) period of the
websocket design that some on this list certainly remember as well. There
was a strong demand from some implementers to only use byte matching on
the wire instead of parsing HTTP messages, presumably because HTTP was
considered as the annoying protocol to switch away from, and I can easily
imagine that initial code parts written this way have not much evolved.

For having seen some HTTP code deployed in small IoT devices with just
a few tens of kB of RAM and essentially relying on strstr() to spot
header+value couples, I would suggest that each time we're dealing with
protocol variations targetting these use cases, we must be very careful
about how our messages are consumed. It's not necessarily easy nor even
possible most of the time, but this needs to be anticipated to ease
location of the problem at least.

Cheers,
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 15 Oct 2020 14:30:02 +1100

Sounds like it might need some grease...

> On 15 Oct 2020, at 2:21 pm, Willy Tarreau <[email-masked]> wrote:
> 
> Hi Alex,
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 15 Oct 2020 05:32:40 +0200

> Sounds like it might need some grease...

Good point. But it will need to be done on extremities at least (clients
and servers), because intermediaries are systematically pointed the finger
at since "look it doesn't work anymore when I insert $GATEWAY here".

Willy

---

From: Alex Rousskov <[email-masked]>
Date: Thu, 15 Oct 2020 09:22:59 -0400

> Sounds like it might need some grease...

Yes, ... 22+ years ago. I doubt it is practical to grease HTTP/1 today,
especially in an Upgrade area where agent proliferation is probably
still relatively high.

In a sense, Squid was doing exactly that -- lightly greasing Upgrade
messages. We had to remove that grease to achieve compatibility with
existing popular players (as Willy has anticipated).

Alex.