Subject: [hybi] ABNF for the method parameter's value of permessage-compression-04 conflicts with Sec-WebSocket-Extensions's ABNF

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 2 Nov 2012 14:17:08 +0900

In http://tools.ietf.org/html/rfc6455#section-9.1

         extension-param = token [ "=" (token | quoted-string) ]
             ;When using the quoted-string syntax variant, the value
             ;after quoted-string unescaping MUST conform to the
             ;'token' ABNF.

but I just noticed the value of the method extension parameter of the
permessage-compression doesn't always conform token ABNF after unescaping.

Please see history below. AFAIK, there was no reason to limit the contents
in quoted-string even to token. It's overkilling (sorry that I didn't try
to discuss this enough when the first change was made and I drafted
permessage-compress forgetting this text).

So, ... I'd like to relax the constraint in RFC 6455 and keep current
design of premessage-compression.

----

History

Until http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-07,
value ABNF was quoted-string.
http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-08 to -13,
it was token.
Since http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-14,
it's quoted-string with the constraint.

I remember that I recommended that characters to be used in extension
parameter value should be restricted to U+0020 - U+007E. That resulted in
quoted-string -> token change.

token -> quoted-string change was made IIRC so that existing library for
normal HTTP that always quotes any value can be used without fix.

See this thread for detail
http://www.ietf.org/mail-archive/web/hybi/current/msg08013.html.

---

From: Peter Thorson <[email-masked]>
Date: Wed, 7 Nov 2012 09:11:00 -0600

On Nov 2, 2012, at 24:17 , Takeshi Yoshino <[email-masked]> wrote:

> but I just noticed the value of the method extension parameter of the permessage-compression doesn't always conform token ABNF after unescaping.
> 
> Please see history below. AFAIK, there was no reason to limit the contents in quoted-string even to token. It's overkilling (sorry that I didn't try to discuss this enough when the first change was made and I drafted permessage-compress forgetting this text).
> 
> So, ... I'd like to relax the constraint in RFC 6455 and keep current design of premessage-compression.
> 
> ----
> 
> History
> 
> Until http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-07, value ABNF was quoted-string.
> http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-08 to -13, it was token.
> Since http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-14, it's quoted-string with the constraint.
> 
> I remember that I recommended that characters to be used in extension parameter value should be restricted to U+0020 - U+007E. That resulted in quoted-string -> token change.
> 
> token -> quoted-string change was made IIRC so that existing library for normal HTTP that always quotes any value can be used without fix.
> 
> See this thread for detail http://www.ietf.org/mail-archive/web/hybi/current/msg08013.html.

It seems that relaxing this constraint is required in order to allow extension parameters to have their own sub-parameters or to provide multiple parameter values for the server to choose between. Neither of these capabilities are easily doable given the constraints of the RFC6455 extension negotiation syntax. Are header fields this complex typically used in HTTP? Is this sort of complex negotiation likely to be needed for other extensions? Does mux required it also? If it is needed, I think allowing arbitrary quoted extension parameter values is a sane way to do it that is consistent with RFC2616 style parsing.

I'd also like to clarify cases where a server is intended to ignore/reject the extension negotiation vs fail the connection. Should a parse error in the request-method-list ABNF defined by permessage-compress or a syntactically valid header that violates one of the MUSTs in section 3 (i.e. a permessage-compress extension with zero or two parameters or a value >15 for window bits) result in a failure of the entire handshake or just a failure to negotiate the extension?

What about cases where the header is syntactically valid by the proposed relaxed RFC6455 rules but not by the request-method-list syntax?
Example (trailing empty method param): Sec-WebSocket-Extensions: permessage-compress; method="deflate; s2c_max_window_bits=10, deflate;"

In the case that there is a failure to negotiate an extension that is required by the client (and implimented by both endpoints) due to not agreeing on parameters (see example) what close code should the client fail the connection with? Still 1010? or something else? If 1010, what exactly would be appropriate to return in the text reason? The entire value with the required parameters?

Example: Sec-WebSocket-Extensions: permessage-compress; method="deflate; s2c_max_window_bits=10" with a server that doesn't support custom window sizes.

> - Compression breaks a number of optimizations that servers can use to reduce CPU and memory usage like outgoing message preparation/de-duplication.
> 
> Do you mean that even if some part of frame construction (copying, etc.) can be skipped, whole frame octets must always go through compressor and it consumes CPU and memory?

These optimizations allow an endpoint can send an identical message to n clients with constant rather than linear marginal memory usage and processing time (for frame construction, utf8 validation, masking, etc). For pubsub/broadcast with many clients this is significant. By default, permessage-compress creates a stream out of the otherwise independent compressed messages in one connection. The sliding window state becomes unique to each connection and required to build subsequent compressed frames. This prevents message preparation as identical messages may be compressed differently for each connection (depending on past messages) and must necessarily be sent through processing once for each connection rather than once for all.

In general this behavior is a good default, as I think in most cases sliding window takeover will be beneficial and at low concurrency levels the memory requirements are worth it. High concurrency services may need to think carefully about whether permessage-compress is worth it.

Disabling sliding window takeover from server to client can also be used to mitigate this behavior in those cases (at obvious compression effectiveness cost). However, presently it seems only the client can request that the server not reuse previous state? I haven't had a chance to test exactly what happens when a server unilaterally decides to reset the sliding window for every message it sends. Is this likely to cause problems? Does the negotiation need a way to signal that this is happening?

Peter

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 8 Nov 2012 17:14:37 +0900

On Thu, Nov 8, 2012 at 12:11 AM, Peter Thorson <[email-masked]>wrote:

> It seems that relaxing this constraint is required in order to allow
> extension parameters to have their own sub-parameters or to provide
> multiple parameter values for the server to choose between. Neither of
> these capabilities are easily doable given the constraints of the RFC6455
> extension negotiation syntax. Are header fields this complex typically used
> in HTTP?
>

AFAIK, no.

> Is this sort of complex negotiation likely to be needed for other
> extensions? Does mux required it also?
>

There was some discussion how we should pass extensions for logical
channels. Currently they're listed linearly together with mux and
extensions for physical connection but there was an idea of nesting
extension list (e.g. mux; logical_ext="compress; deflate, baz", foo, bar).

> If it is needed, I think allowing arbitrary quoted extension parameter
> values is a sane way to do it that is consistent with RFC2616 style parsing.
>
> I'd also like to clarify cases where a server is intended to ignore/reject
> the extension negotiation vs fail the connection. Should a parse error in
> the request-method-list ABNF defined by permessage-compress or a
> syntactically valid header that violates one of the MUSTs in section 3
> (i.e. a permessage-compress extension with zero or two parameters or a
> value >15 for window bits) result in a failure of the entire handshake or
> just a failure to negotiate the extension?
>

I think it's safe to reject only permessage-compress but keep handshake
going as far as core protocol level Sec-WebSocket-Extensions parser doesn't
fail. permessage-compress:
- with zero parameter
- with two or more parameters
- with unknown parameter(s)
can be safely ignored.

Of course, we can also fail them aggressively to let implementors know
their bugs.

----

http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-04#section-5.1
   A server MUST ignore "deflate" method descriptions that:
   o  have any method parameter unknown to the server
   o  have any method parameter with an invalid value
   o  is not supported by the server

http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-04#section-3
   If the server doesn't support any of the descriptions listed by the
client, the server MUST reject use of the Per-message Compression
Extension.

So,
- deflate method description with win > 15 will be ignored
- if there's no description available, reject the permessage-compress
extension

>
> What about cases where the header is syntactically valid by the proposed
> relaxed RFC6455 rules but not by the request-method-list syntax?
> Example (trailing empty method param): Sec-WebSocket-Extensions:
> permessage-compress; method="deflate; s2c_max_window_bits=10, deflate;"
>
>
permessage-compress
- with method parameter value that doesn't conform the request-method-list
ABNF
Is also safe to be ignored, I think.

As long as the server doesn't send back permessage-compress, there should
be no problem (assuming core WebSocket stack on the client has no problem).

> In the case that there is a failure to negotiate an extension that is
> required by the client (and implimented by both endpoints) due to not
> agreeing on parameters (see example) what close code should the client fail
> the connection with? Still 1010? or something else? If 1010, what exactly
> would be appropriate to return in the text reason? The entire value with
> the required parameters?
>
>
How the reason for 1010 should be like is not well specified for now. If
we're going to let UAs react to 1010 automatically, we need to define
something structured (1010 from server may be used as hint for client's
retrial), but IMO, it's sufficient if it contains some text reading that
developers can know what parameter should be supported.

> Example: Sec-WebSocket-Extensions: permessage-compress; method="deflate;
> s2c_max_window_bits=10" with a server that doesn't support custom window
> sizes.
>

My idea is that if the client can live without s2c_max_window_bits, it
should send
  Sec-WebSocket-Extensions: permessage-compress: method="deflate;
s2c_max_window_bits=10; deflate".

Otherwise, it should send value like your Example. If the server rejected
the permessage-compress (as specified in
http://tools.ietf.org/html/draft-ietf-hybi-permessage-compression-04#section-5.1.2,
if the server choose "deflate; s2c_max_window_bits=10" as accepted request,
it must include s2c_max_window_bits=10 in its method agreement. since the
server doesn't support s2c_max_window_bits, the only way the server can
take is rejecting whole permessage-compress request), for example, the
client would send a 1010 with reason phrase in natural language like
"permessage-compress with s2c_max_window_bits=10 is mandatory".

This approach requires clients to list combinatorial number of offers
(method descriptions), but the number of parameters looks reasonably small
for deflate's case.

This is necessary to allow clients to declare mandatoriness. There're
alternatives such as "mandatory" annotation, but they should be also as
complicated as this approach, maybe.

>
>  - Compression breaks a number of optimizations that servers can use to
>> reduce CPU and memory usage like outgoing message
>> preparation/de-duplication.
>>
>
> Do you mean that even if some part of frame construction (copying, etc.)
> can be skipped, whole frame octets must always go through compressor and it
> consumes CPU and memory?
>
>
> These optimizations allow an endpoint ...
>

Thanks for explanation.

>
> Disabling sliding window takeover from server to client can also be used
> to mitigate this behavior in those cases (at obvious compression
> effectiveness cost). However, presently it seems only the client can
> request that the server not reuse previous state? I haven't had a chance to
> test exactly what happens when a server unilaterally decides to reset the
> sliding window for every message it sends. Is this likely to cause
> problems? Does the negotiation need a way to signal that this is happening?
>
>
It should be no problem that the server resets compression context for new
message without client's request to do so. Backward reference symbols
coming from server refers only data inside the message that are also known
by the client (dynamic huffman code used are always introduced in the
message since a deflate block never straddles message boundary). The
compression context of the client which didn't request
s2c_no_context_takeover just remembers data of previous messages that will
never be referred by symbols coming from the server.

Thanks
Takeshi