Subject: Re: [hybi] With deflate-stream, Close frame doesn't work as an end of data marker

From: [email-masked]
Date: Wed, 9 Mar 2011 01:55:01 -0800

This thread, too, I am wondering if this has reached a consensus at this 
point. At least, 
Greg, Brian  and myself agreed to have only payload deflated. This aligns 
with the 
reasons for choosing masking in frames, too. Also, it resolves the 
original issue Takeshi
pointed out with regards to presence of trailing bytes after Close frame 
as in the subject 
which would cause RST in normal termination.

- Yutaka

[email-masked] wrote on 02/28/2011 11:37:17 PM:

> 
> Hi John, 
> 
> [email-masked] wrote on 02/28/2011 06:32:03 AM:
> 
> > [BTW - here is the thread where I first posted results of an
> > experiment - 
http://www.ietf.org/mail-archive/web/hybi/current/msg01810.html
> > ]
> > 
> > On Mon, Feb 28, 2011 at 4:52 AM, Greg Wilkins <[email-masked]> 
wrote:
> > > I'm +1 for compression within the frame.
> > >
> > > If we compress the framing, then intermediaries will have to
> > > decompress in order to see framing and to know when to forward etc.
> > > It also has significant debug issues as it makes it much harder to
> > > decode a partial frame capture.
> > >
> > > I understand the argument that compression will not be as good, but 
I
> > > think some compromise is necessary.
> > 
> > The framing is pretty small and it is hard to imagine much redundancy
> > from the header (the opcode and reserved bits will be constant, but
> > then you will interfere with the compression by all the constantly
> > changing lengths), so I think little is lost by compressing only the
> > payload. 
> 
> Just to be clear, the compression does not have to be per payload of 
> a data frame, but can be entire stream of messages application passes 
in. 
> At the end of the message, though, F_SYNC_FLUSH is necessary to prevent 
> receiver from waiting for the last compressed data to arrive from 
sender, 
> but we should be aware that the compressor still maintains 32kB(or 
whatever 
> it is configured for) of reference range in its context. 
> 
> > 
> > However, it does mean the implementation is more complicated, and I 
> 
> It is actually very simple, in python for example: 
> 
> 
>         zlibObj = zlib.compressobj() 
> 
>         def send(in_data): 
>                    buf = zlibObj.compress(in_data) 
>                    buf += zlibObj.flush(zilib.Z_SYNC_FLUSH) 
>                         : 
>                 (construct frames/fragments and put into a queue) 
> 
> Note: the 'zlibObj' has the same lifetime as that of WebSocket. 
> 
> 
> > don't believe this would be feasible using the stock inflate/deflate
> > in Java for example.  I would not agree to just compressing each
> > frame's payload independently -- my earlier experiments show this will
> > frequently increase the size of the data and result in only modest
> > savings compared to maintaining compression state over the life of the
> > connection (while we are discussing it, 
> 
> Again, lifetime of the compression state/context can be the same. It 
> just needs, Z_SYNC_FLUSH operation at the end of the 'WS-frame' to avoid 

> unnecessary delay. Even compression was done for whole stream, we still 
> need Z_SYNC_FLUSH per frame basis with the exact same reason - we cannot 

> delay the frame! (Unless we introduce nagle's algorithm like protocol 
> just for slightly better compression ratio, which I think would not be 
> feasible.) This, at least to me, looks more complicated. 
> 
> I am aware that, if application passes in is very small message 
> frequently, (Z_SYNC_FLUSH has to be done frequently), resulting 
> compression ratio would not be that great. However, since we can not 
> delay each frame, we need the Z_SYNC_FLUSH anyways. Therefore, there's 
> no benefit in having compression layer under the framing layer. 
> 
> As I mentioned earlier, the compression state can be shared between 
> text and binary messages since they are not interleaved (even 
fragmented), 
> though it may be better to have dedicated compression state if we can 
> afford memory. Either way, we need to define the spec in the document. 
> 
> 
> - Yutaka 
> 
> 
> > we should talk about the
> > parameters to use for deflate to minimize memory requirements on the
> > server -- experiments done for compression in SPDY [mentioned in the
> > linked thread] show you can get most of the benefit with more modest
> > memory requirements using lower parameters). 
> > 
> > It also means we have to agree on the details of how it should be
> > specified, which is why the super-simple deflate-stream was added to
> > the standard instead of trying to get agreement on something better.
> > 
> > --
> > John A. Tamplin
> > Software Engineer (GWT), Google
> > _______________________________________________
> > hybi mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/hybi
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi

---

From: Andy Green <[email-masked]>
Date: Wed, 09 Mar 2011 10:58:49 +0000

> Greg, Brian and myself agreed to have only payload deflated. This aligns

I'd be a bit happier with payload-only compression but I don't really 
mind.  It's a little bit less deadly than masking-outside-framing 
because you can at least turn off negotiation of compression inside the 
standard OK if you want to debug it.

> which would cause RST in normal termination.

Having implemented this now, I don't see the problem using 
Z_PARTIAL_FLUSH as the standard seems to recommend normally, and then 
Z_FULL_FLUSH when sending the close packet, and disabling any further 
packet issue on that connection.  So I don't think Takeshi's issue is 
more than an implementation problem, I'd welcome being educated if I 
missed any point.

-Andy

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 9 Mar 2011 21:06:35 +0900

On Wed, Mar 9, 2011 at 19:58, Andy Green <[email-masked]> wrote:

> I'd be a bit happier with payload-only compression but I don't really mind.
>  It's a little bit less deadly than masking-outside-framing because you can
> at least turn off negotiation of compression inside the standard OK if you
> want to debug it.
>
>

I prefer in-frame compression, too.

I sent out a proposal
http://www.ietf.org/mail-archive/web/hybi/current/msg06642.html last week.
It's similar to what Yutaka explained in this thread. Comments are welcome.

> Having implemented this now, I don't see the problem using Z_PARTIAL_FLUSH
> as the standard seems to recommend normally, and then Z_FULL_FLUSH when
> sending the close packet, and disabling any further packet issue on that
> connection.  So I don't think Takeshi's issue is more than an implementation
> problem, I'd welcome being educated if I missed any point.
>
>
Maybe, exhaustive (to be strict, recv with big buffer) recv call in your
code drains all the deflated data arrived (everything including Huffman
codes for some preceding frames, ones for close frame, 3 bit header BFINAL=0
BTYPE=00 and 00 00 ff ff) from the TCP stack.

Yes, since it's rare that octets for a single WebSocket frame are put in
separate TCP packets and delivered to application layer separately
(requiring separate recv-call), this is not a big problem. But with in-frame
compression, it becomes perfect.

---

From: Andy Green <[email-masked]>
Date: Wed, 09 Mar 2011 12:35:14 +0000

On 03/09/2011 12:06 PM, Somebody in the thread at some point said:

Hi -

>     Z_PARTIAL_FLUSH as the standard seems to recommend normally, and
>     then Z_FULL_FLUSH when sending the close packet, and disabling any
>     further packet issue on that connection.  So I don't think Takeshi's
>     issue is more than an implementation problem, I'd welcome being
>     educated if I missed any point.
>
>
> codes for some preceding frames, ones for close frame, 3 bit header
> BFINAL=0 BTYPE=00 and 00 00 ff ff) from the TCP stack.

Yes on both rx / inflate and tx / deflate case libwebsockets will always 
loop and go back to in/deflate() for the rest if the callback handling 
that indicates the zlib call filled its spill buffer (suggesting that it 
very likely has more to spill).  So if there are multiple buffers full 
of stuff to flush in both rx or tx path they will always all get sent on 
the wire or sent to the user callback as uncompressed rx data before 
moving on.

You can see the loop here for the tx path

http://git.warmcat.com/cgi-bin/cgit/libwebsockets/tree/lib/parsers.c#n1378

After issuing specifically the CLOSE opcode packet like that, the 
connection state is changed so nothing more can be emitted on the 
socket, and this works reliably to issue the CLOSE to the far end for 
sure without further zlib housekeeping traffic or any other traffic 
going out.

> (requiring separate recv-call), this is not a big problem. But with
> in-frame compression, it becomes perfect.

I think you must handle extreme compression cases anyway to keep an eye 
on latency if nothing else, consider he sends 200K of 0x00 and the 
compressor run-length encodes it, you receive a handful of compressed 
code bytes but must sit looping through 200K of decompressed content 
that represents one message.  Or you pass in 1K of high quality random 
and 3K of codes comes out of the compressor.

Like I say I think keeping framing out of compression is okay, just 
deflate-stream babbling after close isn't any reason for it AFAICS.

-Andy

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 9 Mar 2011 23:02:18 +0900

Hi Andy,

On Wed, Mar 9, 2011 at 21:35, Andy Green <[email-masked]> wrote:

> loop and go back to in/deflate() for the rest if the callback handling that
> indicates the zlib call filled its spill buffer (suggesting that it very
> likely has more to spill).  So if there are multiple buffers full of stuff
> to flush in both rx or tx path they will always all get sent on the wire or
> sent to the user callback as uncompressed rx data before moving on.
>
> connection state is changed so nothing more can be emitted on the socket,
> and this works reliably to issue the CLOSE to the far end for sure without
> further zlib housekeeping traffic or any other traffic going out.
>
>
I'm still not sure if we're on the same table. Please allow me to explain my
thoughts again. Sorry if i'm misunderstanding your point.

Sending a close frame is not a problem. What I'm taking as a (small, but)
problem is receiving a close frame. "Trailing bytes after Close frame" in
Yutaka's post doesn't mean "any further packet" or "any other traffic going"
on the WebSocket layer. It means trailing bytes generated by zlib.

Your code luckily drained the "close, 3 bit header, 00 00 ff ff" which
contains octets generated by zlib for performing Z_FULL_FLUSH. But please
imagine someone split them into two TCP packets e.g. "close codes, 3 bit
header" and "00 00 ff ff" and inserted some delay between them. Your
callback is run with eff_buf.token filled with "... close codes, 3 bit
header", and after that your callback is run again with eff_buf.token = "00
00 ff ff".

On the first run of callback, zlib can decompress all of close frame octets,
but there's 00 00 ff ff left in the TCP stack, so it's not ready to issue
close(2) on the socket.

> I think you must handle extreme compression cases anyway to keep an eye on
> latency if nothing else, consider he sends 200K of 0x00 and the compressor
> run-length encodes it, you receive a handful of compressed code bytes but
> must sit looping through 200K of decompressed content that represents one
> message.  Or you pass in 1K of high quality random and 3K of codes comes out
> of the compressor.
>

---

From: Andy Green <[email-masked]>
Date: Wed, 09 Mar 2011 14:31:56 +0000

On 03/09/2011 02:02 PM, Somebody in the thread at some point said:

Hi -

> I'm still not sure if we're on the same table. Please allow me to
> explain my thoughts again. Sorry if i'm misunderstanding your point.
>
> Sending a close frame is not a problem. What I'm taking as a (small,

OK.

> but) problem is receiving a close frame. "Trailing bytes after Close
> frame" in Yutaka's post doesn't mean "any further packet" or "any other
> traffic going" on the WebSocket layer. It means trailing bytes generated
> by zlib.
>
> contains octets generated by zlib for performing Z_FULL_FLUSH. But
> please imagine someone split them into two TCP packets e.g. "close
> codes, 3 bit header" and "00 00 ff ff" and inserted some delay between

OK.

> them. Your callback is run with eff_buf.token filled with "... close
> codes, 3 bit header", and after that your callback is run again with
> eff_buf.token = "00 00 ff ff".

OK I think I get your idea.  So it is possible at the receiver I get 
enough zlib codes in one packet that it emits the full close frame 
decompressed from inflate(), and I go off and interpret that and try to 
close the socket, but there is already some "dribble" from zlib in 
flight in a second packet which never gets serviced then.

The only thing is I set it to do Z_PARTIAL_FLUSH after every input is 
added to the compressor.  So it means the compressor cannot be holding 
much of anything at the time we come and ask it to send the CLOSE frame 
with Z_FULL_FLUSH.  CLOSE frame is limited to 125 payload.  So it seems 
it would always issue one final compressed output buffer in this case, 
including zlib trailer, which translates to one packet on the wire 
reliably.  I'm also using TCP_NODELAY to stop it aggregating packets and 
causing latency.

I agree it is a theoretical opportunity for something else to fragment 
the packet and make the situation you're describing, but I can't see it 
happening the thing is on a link with like 30-40 byte MTU which is what 
is needed including TCP /IP headers to break the packet there.

Are there other circumstances that can actually cause this I'm 
overlooking too easily?  Thanks for explaining your meaning to me.

-Andy

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 10 Mar 2011 00:49:26 +0900

On Wed, Mar 9, 2011 at 23:31, Andy Green <[email-masked]> wrote:

> OK I think I get your idea.  So it is possible at the receiver I get enough
> zlib codes in one packet that it emits the full close frame decompressed
> from inflate(), and I go off and interpret that and try to close the socket,
> but there is already some "dribble" from zlib in flight in a second packet
> which never gets serviced then.
>
Yes!

>
> The only thing is I set it to do Z_PARTIAL_FLUSH after every input is added
> to the compressor.  So it means the compressor cannot be holding much of
> anything at the time we come

I'm doing the same in pywebsocket (using Z_SYNC_FLUSH).

> and ask it to send the CLOSE frame with Z_FULL_FLUSH.

Yes, too (Z_SYNC_FLUSH).

>  CLOSE frame is limited to 125 payload.  So it seems it would always issue
> one final compressed output buffer in this case, including zlib trailer,
> which translates to one packet on the wire reliably.  I'm also using
> TCP_NODELAY to stop it aggregating packets and causing latency.
>
It makes sense.

>
> I agree it is a theoretical opportunity for something else to fragment the
> packet and make the situation you're describing, but I can't see it
> happening the thing is on a link with like 30-40 byte MTU which is what is
> needed including TCP /IP headers to break the packet there.
>
> Yes. Because of that, I'm not so eager to push this idea... But if we can
have better one, I think it's nice.

One method I can come up with
- by which we can make sure that we recv all the data zlib emit
- that can be applied to stream-based deflate
is putting BFINAL=1 block at the end of stream. zlib does this when Z_FINISH
is specified. That block contains full or some portion of close frame octets
or even empty one after close codes is ok. zlib returns EoF by some manner
when it consumed all the data terminated by BFINAL=1 block.

python's zlib wrapper doesn't have API to probe EoF now, so I can't apply
this to pywebsocket, but it seems it's applicable on many environment
(original C zlib, Java, etc.)

> Are there other circumstances that can actually cause this I'm overlooking
> too easily?  Thanks for explaining your meaning to me.
>
No problem. Thanks.

---

From: [email-masked]
Date: Wed, 9 Mar 2011 13:51:28 -0800

> overlooking too easily?

Yes. Since your receive buffer passed in recv(2) has a predetermined 
finite size and you may not be reading all data in the kernel space 
(TCP) buffer. Depending on how it is implemented and timing, the
receiver may successfully extract Close frame out of data you have just
read from TCP socket, and then close the socket without bothering to 
read the outstanding data in the kernel.

- Yutaka

>  Thanks for explaining your meaning to me.
> 
> -Andy

---

From: Andy Green <[email-masked]>
Date: Wed, 09 Mar 2011 22:13:24 +0000

On 03/09/2011 09:51 PM, Somebody in the thread at some point said:

Hi -

> read the outstanding data in the kernel.

It's a good point.  I can see how that's effectively fragmenting the 
thing entirely in userland at the receiver.

However that seems simple to solve since the extra junk data is already 
received in kernel.  Just before socket close time we just need to use a 
little poll() POLLIN loop with no timeout to drain it while there's 
stuff immediately pending, right?  Nothing more will come and close is 
always happy.

-Andy

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 9 Mar 2011 21:22:27 -0800

I'm going over the threads trying to figure out what i need to do, and I
will admit that this one has me a bit stumped. If we adopt yoshino-san's
proposal [1] to only compress the payload of the frame, does that put this
issue to rest?

[1] http://www.ietf.org/mail-archive/web/hybi/current/msg06674.html

On Wed, Mar 9, 2011 at 2:13 PM, Andy Green <[email-masked]> wrote:

> It's a good point.  I can see how that's effectively fragmenting the thing
> entirely in userland at the receiver.
>
> little poll() POLLIN loop with no timeout to drain it while there's stuff
> immediately pending, right?  Nothing more will come and close is always
> happy.
>
>
> -Andy

---

From: Salvatore Loreto <[email-masked]>
Date: Thu, 10 Mar 2011 08:59:04 +0200

(as co-chair I want to bring your attention to this)

Hi there,

the following changes (proposed by Takeshi Yoshino in [1])
has been largely discussed (for the technical reasons follow the 
discussions thread [1] and [2]):
the discussion has highlighted that the proposal brings some benefit and 
no side effects.

[1] http://www.ietf.org/mail-archive/web/hybi/current/msg06642.html
[2] http://www.ietf.org/mail-archive/web/hybi/current/msg06801.html
[3] http://www.ietf.org/mail-archive/web/hybi/current/msg06674.html

so unless anyone object strongly -07 will include the "yoshino-san's 
proposal" [3]

cheers
/Sal

-- 
Salvatore Loreto
www.sloreto.com

On 3/10/11 7:22 AM, Ian Fette (イアンフェッティ) wrote:
> I'm going over the threads trying to figure out what i need to do, and 
> I will admit that this one has me a bit stumped. If we adopt 
> yoshino-san's proposal [1] to only compress the payload of the frame, 
> does that put this issue to rest?
>
> On Wed, Mar 9, 2011 at 2:13 PM, Andy Green <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     It's a good point.  I can see how that's effectively fragmenting
>     the thing entirely in userland at the receiver.
>
>     However that seems simple to solve since the extra junk data is
>     already received in kernel.  Just before socket close time we just
>     need to use a little poll() POLLIN loop with no timeout to drain
>     it while there's stuff immediately pending, right?  Nothing more
>     will come and close is always happy.
>
>
>     -Andy
>     [email-masked] <mailto:[email-masked]>

---

From: Ian Fette (イアンフェッティ) <[email-masked]>
Date: Wed, 9 Mar 2011 23:00:10 -0800

seems straight forward enough.

-ian

On Wed, Mar 9, 2011 at 10:59 PM, Salvatore Loreto <
[email-masked]> wrote:

> the discussion has highlighted that the proposal brings some benefit and no
> side effects.
>
>
>
> Salvatore Loretowww.sloreto.com
>
>
>

---

From: Brian <[email-masked]>
Date: Wed, 9 Mar 2011 23:21:47 -0800

Sounds fantastic!

Brian

On Wed, Mar 9, 2011 at 10:59 PM, Salvatore Loreto
<[email-masked]> wrote:
> has been largely discussed (for the technical reasons follow the discussions
> thread [1] and [2]):

---

From: Andy Green <[email-masked]>
Date: Thu, 10 Mar 2011 08:56:03 +0000

On 03/10/2011 05:22 AM, Somebody in the thread at some point said:

Hi -

> proposal [1] to only compress the payload of the frame, does that put
> this issue to rest?
>

Payload only compression would be nice.

However this "dribbling compression codes after close" thing is not a 
problem in the real world that I can see and at most maybe needs a note 
that before closing a socket compressed like this, you may need to drain 
it in a loop polling it with no timeout and POLLIN until nothing is left 
before calling close().

Also one note on Yoshino's language to describe the compression actions 
it mixes Z_ nomenclature that zlib uses and hex packets on the wire, if 
this hex trailer that he mentions is a byproduct of Z_FINISH or whatever 
it might be clearer to an implementer that he can generate it like that 
and forget mentioning magic hex.

-Andy

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 10 Mar 2011 11:51:19 +0000

> reliably.  I'm also using TCP_NODELAY to stop it aggregating packets and 

You *cannot* depend on such a write translating to one output packet.

You also cannot depend on TCP_NODELAY preventing packet aggregation.

There are many corner cases, but here is a few simple ones: When there
is any bottleneck such as congestion, receiver too slow, receiver busy
or not enough bandwidth somewhere, the sender will delay sending, or
resend, and so may aggregate more data into previously queued packets.

This is an area where I have had to hit programmers over the head with
a cluebrick, when their program that "worked fine on our LAN" would
occasionally, rarely, drop connections when run over other networks or
even when run on different OSes with different TCP implementations.
It is a nightmare to debug and fix protocols that make that assumption.

TCP can, and occasionally does, split and join the byte stream at
arbitrary positions.  It should always by treated as doing so.

It is ok to optimise with awareness of packet boundaries, but not to
depend on it for reliability.

Even if your sending host does not, various tunnels, proxies, relays,
transcoders etc. in the middle may alter the packet splitting.

In TCP, write() does not reliably correspond to packets, and neither
does read().

When orderly close (close-message) is used to avoid TCP reset hazard,
it is important for the receiver to defer close() or shutdown(SHUT_RD)
until it knows it has _definitely_ read the _last byte_ from the socket.

The last byte may be split off and delayed by flow control history,
and when it finally arrives, if close or SHUT_RD is done too soon, the
last byte causes a TCP reset, breaking unfinished messages in the
other direction.

This isn't about "reading OS buffers"; it can also be data in flight.
So you can't solve this by exhaustively reading from the socket before
closing.

Apache handles this by waiting for TCP FIN, but WebSocket can't depend
on that.

-- Jamie

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 10 Mar 2011 12:06:33 +0000

> it in a loop polling it with no timeout and POLLIN until nothing is left 

Polling in that way will not make this reliable, because you cannot be
sure the OS has recieved the final bytes.  The assumption that the
close-message and trailing bytes always occupy the same packet is not
reliable.  It is easy to construct TCP scenarios where the last byte
is split off at the sender, due to timing at the receiver.

It is not likely to be a _significant_ problem in the real world.
Many WebSocket apps likely don't need orderly close semantics either.[1]

But if we don't care about making orderly close reliable in corner
cases, there is no point having orderly close at all, as it defeats
the point of it.

-- Jamie

[1] (I expect many "big" apps and toolkits will end up doing what I
did: treat WS as an unreliable connection, and do their own framing,
ordering, duplicate detection, keepalives and message retrying
at the application layer, due to WS's sort-of-but-not-really attempts
to be reliable, and because that matches up well with fallbacks to HTTP)

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 10 Mar 2011 21:12:06 +0900

On Thu, Mar 10, 2011 at 17:56, Andy Green <[email-masked]> wrote:

> On <03%2F10%2F2011> <03%2F10%2F2011> <03%2F10%2F2011>03/10/2011 05:22 AM,
> Somebody in the thread at some point said:
>
> Hi -
>
> problem in the real world that I can see and at most maybe needs a note that
> before closing a socket compressed like this, you may need to drain it in a
> loop polling it with no timeout and POLLIN until nothing is left before
> calling close().
>

As Jamie created a new thread, let's talk about this there.

>
> Also one note on Yoshino's language to describe the compression actions it
> mixes Z_ nomenclature that zlib uses and hex packets on the wire, if this
> hex trailer that he mentions is a byproduct of Z_FINISH or whatever it might
> be clearer to an implementer that he can generate it like that and forget
> mentioning magic hex.
>
>
Basically I agree your point here that we should use only deflate
terminology in definition, and append implementation note to it using zlib
nomenclature, while avoiding mentioning magic hex as you say.

In my proposal (
http://www.ietf.org/mail-archive/web/hybi/current/msg06642.html the bottom
of the message), I used one magic hex "0x00 0x00 0xff 0xff" to include
optimization on deflate adopted by PPP and explain what implementor have to
do quickly. That can be changed to ... something like "0 byte length block
with BTYPE=00 excluding headers", but it's not helpful in this context IMO.
This is originally a part of byproduct of zlib's de-facto standard way to
flush deflated stream, but operation introduced by RFC 1979 like cutting,
removing on sender, re-appending the bytes on receiver is out of scope of
RFC1951. Maybe, just referring RFC1979 in the explanation of receiver's
action would be fine?

---

From: Andy Green <[email-masked]>
Date: Thu, 10 Mar 2011 12:16:44 +0000

On 03/10/2011 12:12 PM, Somebody in the thread at some point said:

Hi -

>     Also one note on Yoshino's language to describe the compression
>     actions it mixes Z_ nomenclature that zlib uses and hex packets on
>     the wire, if this hex trailer that he mentions is a byproduct of
>     Z_FINISH or whatever it might be clearer to an implementer that he
>     can generate it like that and forget mentioning magic hex.
>
>
> terminology in definition, and append implementation note to it using
> zlib nomenclature, while avoiding mentioning magic hex as you say.
>
> In my proposal
> (http://www.ietf.org/mail-archive/web/hybi/current/msg06642.html the
> bottom of the message), I used one magic hex "0x00 0x00 0xff 0xff" to
> include optimization on deflate adopted by PPP and explain what
> implementor have to do quickly. That can be changed to ... something
> like "0 byte length block with BTYPE=00 excluding headers", but it's not
> helpful in this context IMO. This is originally a part of byproduct of
> zlib's de-facto standard way to flush deflated stream, but operation
> introduced by RFC 1979 like cutting, removing on sender, re-appending
> the bytes on receiver is out of scope of RFC1951. Maybe, just referring
> RFC1979 in the explanation of receiver's action would be fine?

Well my issue with it was I don't know what code to write as described, 
it sounds like I should fill up a four byte bufer with constants and 
write() it out.

I guess all the actions fall into simple zlib calls with the right Z_ 
flags, eg, it really means you must call the last guy with Z_FINISH. 
However else it's described it'll be helpful to implementers to give 
examples where it isn't clear in Z_ nomenclature as well.

-Andy

---

From: Andy Green <[email-masked]>
Date: Thu, 10 Mar 2011 12:21:54 +0000

On 03/10/2011 11:51 AM, Somebody in the thread at some point said:

Hi -

> resend, and so may aggregate more data into previously queued packets.

Hum fair enough, that is a good point.

If I understood it, it looks like the new language from Yoshino will 
make this moot, because knowing the link is compressed, we know we have 
to specifically wait a while anyway for Z_FINISH bytes to come before close.

-Andy

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 10 Mar 2011 21:34:57 +0900

On Thu, Mar 10, 2011 at 21:16, Andy Green <[email-masked]> wrote:

> On <03%2F10%2F2011>03/10/2011 12:12 PM, Somebody in the thread at some
> point said:
>
> Hi -
>
> Well my issue with it was I don't know what code to write as described, it
> sounds like I should fill up a four byte bufer with constants and write() it
> out.
>
> flags, eg, it really means you must call the last guy with Z_FINISH. However
> else it's described it'll be helpful to implementers to give examples where
> it isn't clear in Z_ nomenclature as well.
>
> -Andy
>

OK. So we can add an implementation note like this.

Senders using this extension MUST apply RFC 1951 DEFLATE to all the bytes of
> the application data portion of data frames. Senders MAY include multiple
> blocks in a frame. Senders MAY use blocks of any type as defined in RFC
> 1951. Senders MUST use the algorithm described in Section 2.1 of RFC1979 to
> align compressed data to byte boundary. Senders MAY keep using the same LZ77
> sliding window for multiple frames.

If you're using zlib for applying DEFLATE compression, specify Z_SYNC_FLUSH
when you flush the last octets of application data for each frame (in case
the last block is non-compressed block (BTYPE=00), append one empty
compressed block (BTYPE=01)), and then strip the last 4 octets from the
compressed octets.

Receivers MUST append 0x00 0x00 0xff 0xff to the application data portion
> and decompress it using DEFLATE to decode the original application data.
> Receivers MUST keep using the same LZ77 sliding window for all the frames of
> the same WebSocket connection. Receivers MUST assume the senders use 32768
> byte LZ77 sliding window.

If you're using zlib for applying DEFLATE compression, for each received
frame, concatenate 4 octets 0x00 0x00 0xff 0xff and the application data,
and then supply the resulting bytes to zlib and use the uncompressed data.

----

There might be some function prepared on zlib that performs these procedure,
but I don't know that (i'm not so familiar with the original C lang zlib).
And, e.g. zlib wrapper of Python which I'm using to write WebSocket server
doesn't expose such method at all. I thought simple
uncompress/compress/flush methods are at least available on most of
platforms and built this text.

---

From: John Tamplin <[email-masked]>
Date: Thu, 10 Mar 2011 11:31:16 -0500

On Thu, Mar 10, 2011 at 7:34 AM, Takeshi Yoshino <[email-masked]>wrote:

> Senders using this extension MUST apply RFC 1951 DEFLATE to all the bytes
>> of the application data portion of data frames. Senders MAY include multiple

zlib is just one implementation, and I don't think the spec should be
choosing one implementation instead of the algorithm.  Therefore, we
shouldn't be using zlib-specific terminology in the spec.

I don't object to compressing only the payload (though it does make
implementations more complicated, rather than just pushing a
compressor/decompressor onto the stream having to switch back and forth),
but couldn't the issue we are trying to solve here be done by just saying
that if deflate-stream is negotiated and a close frame is read, you cannot
initiate close processing until the final compressed block is read
completely?

In any case, I think deflate-stream is only a stop-gap solution to get some
form of compression in the spec, also serving as an example extension.  I
think realistically we are going to have to have the ability to avoid
compressing incompressible data in the stream.  Initially, where WebSockets
implementations only support plain text, this won't be much of a problem,
but eventually clients will be mixing binary data with their text and we
would prefer to be able to use compression where it helps and not use it
where it hurts.

---

From: Takeshi Yoshino <[email-masked]>
Date: Fri, 11 Mar 2011 09:54:45 +0900

On Fri, Mar 11, 2011 at 01:31, John Tamplin <[email-masked]> wrote:

>> If you're using zlib for applying DEFLATE compression, specify
>> Z_SYNC_FLUSH when you flush the last octets of application data for each
>> frame (in case the last block is non-compressed block (BTYPE=00), append one
>> empty compressed block (BTYPE=01)), and then strip the last 4 octets from
>> the compressed octets.
>>

So, the note including zlib terms should go somewhere else than the spec
text.

IMO at least we should explain what implementors actually should do to align
to octet boundary, and place that somewhere they can easily get to. The
current deflate-stream spec just says "Senders MUST NOT delay the
transmission of any portion of a WebSocket message because the deflate
encoding of the message does not end on a byte boundary". It's not so easy
for implementors to know what to do and be convinced what they're doing is
correct. The octet aligning method is NOT explained in RFC1951. If we could
add some text like "For example, you can do this by adding BTYPE=00 block"
(and at least add reference to zlib since it's not our invention),
implementors can understand what to do just by reading WebSocket spec and
RFC1951. No need to use Z_ stuff.

> initiate close processing until the final compressed block is read
That's the quick fix to the deflate-stream extension I proposed above. I
want a short note in the spec saying that the sender should finalize the
stream after the close frame by BFINAL=1 block".

Some may immediately understand the deflate stream should finalized after
the close by just reading "Senders using this extension MUST apply RFC 1951
encodings to all bytes of the data stream following the handshake including
both data and control messages.", but expecting implementors to guess is not
good, I think.

> compressing incompressible data in the stream.
>

I agree this point. I'm not objecting to having deflate-stream in the
initial spec.

>   Initially, where WebSockets implementations only support plain text, this
> won't be much of a problem, but eventually clients will be mixing binary
> data with their text and we would prefer to be able to use compression where
> it helps and not use it where it hurts.
>
> --
> John A. Tamplin