Subject: [hybi] Report of a dirty WS blocking issue with OfficeScan

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Mar 2012 11:18:50 +0100

Hi,

I'm posting this here as Jason is not subscribed to the list, please
keep him CCed for any response or for more details.

A few months ago I was contacted by Jason Strimpel (CCed) about an issue
with WS and haproxy. From the server side, the handshake correctly worked,
the server sent some data but the client never replied. To make the long
story short, after many tests, network captures, etc... it became obvious
that the data were not leaving the client for connections going to port
80. What made it look like it was haproxy was the fact that connecting
directly to the server on port 8000 used to work.

After inspecting all components deployed on the desktop, Jason found
that Trend Micro's OfficeScan was installed on the machine, and that
disabling it fixed the issue. It seems like this product inspects
outgoing traffic to port 80, doesn't understand HTTP 101+Upgrade and
blocks any further outgoing traffic without closing the connection
(probably that it considers 101 as 200 and the lack of content-length
as a close mode).

It would be nice to know if minor adaptations to the handshake could
make such connections fail cleanly and quickly. I wonder if just sending
"Content-length: 0" in responses would workaround such components. If
the product considers that 101 is a final response, at least with cl:0
it should wait for a new request and reject further client data as an
invalid request. Possibly that if this works, adding "Connection: close"
in the response would also handle the situation when the client expects
the server to speak first.

These are some of the tests we could possibly do with Jason as he has
access to the faulty device.

I wanted to report this here so that nobody needlessly wastes as much time
on a similar issue. We'll report here if we find an simple workaround for
the issue so that the connection fails cleanly.

Regards,
Willy

---

From: Tobias Oberstein <[email-masked]>
Date: Fri, 16 Mar 2012 04:01:52 -0700

Willy, thanks for reporting. Good to know to watch out for such things.

> After inspecting all components deployed on the desktop, Jason found that
> Trend Micro's OfficeScan was installed on the machine, and that disabling it
> fixed the issue. It seems like this product inspects outgoing traffic to port 80,
> doesn't understand HTTP 101+Upgrade and blocks any further outgoing traffic
> without closing the connection (probably that it considers 101 as 200 and the
> lack of content-length as a close mode).

Instead of putting the burden onto the whole WebSocket community, why not
make that single vendor fix their broken product?

Or - quick fix: uninstall it, safe money. Such products often do more harm
than good.

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Mar 2012 12:06:57 +0100

> make that single vendor fix their broken product?

Jason is contacting the vendor in parallel. But we wanted to make WS fail
as cleanly as possible on broken products so that they are easily spotted.
It took us 3 months to spot the faulty product. With a clean failure, it
would just have taken hours.

> Or - quick fix: uninstall it, safe money. Such products often do more harm

That's not the subject here. People don't chose what they use at their work,
and our engineering work consists in helping them complain to the right
person to get the product fixed or removed.

Regards,
Willy

---

From: "Richard L. Barnes" <[email-masked]>
Date: Fri, 16 Mar 2012 15:14:32 -0400

> would just have taken hours.

What would clean failure have looked like in this case?  A timeout?  And how would that have helped identify the guilty product?

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 16 Mar 2012 20:21:13 +0100

Hi Richard,

On Fri, Mar 16, 2012 at 03:14:32PM -0400, Richard L. Barnes wrote:
> What would clean failure have looked like in this case?  A timeout?  And how
> would that have helped identify the guilty product?

No, it should have been a connection close by the scanner, that would have
been detected by the client. When an error happens with an active close, it
is much easier to spot the culprit because both ends would observe the close,
so it's easy to find the middle point. But when everything gets stuck with
everyone thinking the data was properly transmitted, it's very hard to find
where the problem is.

Regards,
Willy

---

From: Greg Wilkins <[email-masked]>
Date: Mon, 26 Mar 2012 08:54:51 +1100

Willy,

I agree that fast fail is desirable, but in this case I cannot see how
we can really change the protocol in order to achieve this.   The
fault for your 3 month search really is that the product does not have
sufficient logging of it's actions to make it obvious that it is
rejecting traffic.

RFC2616 says:

   All 1xx (informational), 204 (no content), and 304 (not modified) responses
   MUST NOT include a message-body. All other responses do include a
   message-body, although it MAY be of zero length.

  4.4
   1.Any response message which "MUST NOT" include a message-body (such
     as the 1xx, 204, and 304 responses and any response to a HEAD
     request) is always terminated by the first empty line after the
     header fields, regardless of the entity-header fields present in
     the message.

  14.13 Applications SHOULD use this field [Content-length] to
indicate the transfer-length of
   the message-body, unless this is prohibited by the rules in section
   4.4.

I certainly have had issues with SOAP clients that have broken on
content-length:0 being sent with a 100 continue response, so I'd fear
that adding that header to 101's might break code that is written
against the RFC.

As for adding a Connection:close to the 101 response, I'm dubious
about it's effectiveness in this situation.   If the intermediary is
thinking this is a 200 response, this it will expect the server to
send a response and then close the connection.  I can't see how this
header would force the intermediary to close the connection on it's
own.  Perhaps if you could conduct some experiments with this product
and some similar ones to see if it would be effective, then that would
strengthen the case to consider it.

cheers