Subject: [hybi] Fragmented text message

From: "Kukosa, Tomas" <[email-masked]>
Date: Thu, 21 Jul 2011 10:28:51 +0200

Hi,

I have a one question regarding fragmented text messages.

If the text message is fragmented must be each fragment a valid UTF-8 string or only complete defragmented message must be a valid UTF-8 string?
I.e. may I during receiving decode each fragment by UTF-8 and than join strings or do I need to receive all fragments and then decode only defragmented message?

Thanks,
  Tomas

---

From: Brian <[email-masked]>
Date: Thu, 21 Jul 2011 01:47:11 -0700

Unless I'm mistaken, the fragmentation may occur in the middle of a
multi-byte character sequence.  Your code should be aware of that when
decoding.  My initial implementation buffers all fragments and then decodes
the whole message into a string at once.  I imagine you could probably
inspect the last four bytes of a fragment to determine whether there's a
partial utf-8 character.  If there is, you could buffer just those few bytes
and decode the rest of the fragment.  Then when the next fragment comes in,
prepend those bytes to the new payload and continue. Depending on your use
case and what you're optimizing for, it may be more efficient to just buffer
the whole message and then decode.

Brian

On Thu, Jul 21, 2011 at 1:28 AM, Kukosa, Tomas <
[email-masked]> wrote:

> If the text message is fragmented must be each fragment a valid UTF-8
> string or only complete defragmented message must be a valid UTF-8 string?
> I.e. may I during receiving decode each fragment by UTF-8 and than join
> strings or do I need to receive all fragments and then decode only
> defragmented message?
>
> Thanks,
>  Tomas
> _______________________________________________
> hybi mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/hybi
>

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 21 Jul 2011 18:22:25 +0900

Agreed. Nothing in the spec disallows splitting UTF-8 byte sequence into
separate frames.

Impatient receivers must use some intelligent UTF-8 decoder like Brian
explained to get code points decoded. For example in python,
codecs.getincrementaldecoder('utf-8')() does this. Maybe most of major
platforms have library like it.

We may also add a constraint that the text message must be fragmented at
UTF-8 byte sequence boundary, but it complicates fragmentation code. I'm not
for that.

---

From: Piotr Kulaga <[email-masked]>
Date: Thu, 21 Jul 2011 16:01:02 +0000

My understanding is that each websocket entity that uses UTF-8 must have a valid UTF-8 stream as a payload. This covers all UTF-8 frames including continuation frames.

I slightly more prefer approach where UTF-8 message must contain valid sequence rather than each continuation frame (simpler for intermediaries, endpoints that stream data to application still must handle partial UTF-8 code point encoding case). Fine with any approach as long as it is well defined.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Takeshi Yoshino
Sent: Thursday, July 21, 2011 2:22 AM
To: Brian
Cc: [email-masked]
Subject: Re: [hybi] Fragmented text message

Agreed. Nothing in the spec disallows splitting UTF-8 byte sequence into separate frames.

Impatient receivers must use some intelligent UTF-8 decoder like Brian explained to get code points decoded. For example in python, codecs.getincrementaldecoder('utf-8')() does this. Maybe most of major platforms have library like it.

We may also add a constraint that the text message must be fragmented at UTF-8 byte sequence boundary, but it complicates fragmentation code. I'm not for that.

On Thu, Jul 21, 2011 at 17:47, Brian <[email-masked]<mailto:[email-masked]>> wrote:
Unless I'm mistaken, the fragmentation may occur in the middle of a multi-byte character sequence.  Your code should be aware of that when decoding.  My initial implementation buffers all fragments and then decodes the whole message into a string at once.  I imagine you could probably inspect the last four bytes of a fragment to determine whether there's a partial utf-8 character.  If there is, you could buffer just those few bytes and decode the rest of the fragment.  Then when the next fragment comes in, prepend those bytes to the new payload and continue. Depending on your use case and what you're optimizing for, it may be more efficient to just buffer the whole message and then decode.

Brian

On Thu, Jul 21, 2011 at 1:28 AM, Kukosa, Tomas <[email-masked]<mailto:[email-masked]>> wrote:
If the text message is fragmented must be each fragment a valid UTF-8 string or only complete defragmented message must be a valid UTF-8 string?
I.e. may I during receiving decode each fragment by UTF-8 and than join strings or do I need to receive all fragments and then decode only defragmented message?

---

From: David Endicott <[email-masked]>
Date: Thu, 21 Jul 2011 12:11:03 -0400

Would that not then require that the websocket peer and any websocket aware
intermediaries examine and understand the content of the frames?   (ie. to
be able to tell if a UTF sequence is being split).

That seems, to me, to be highly impractical.  Or even impossible.

Also, I consider WS to be transport protocol and as such, it must be content
agnostic.

So long as the endpoints deliver via their API's a valid UTF-8 stream, what
does it matter if it's scattered on the wire?

On Thu, Jul 21, 2011 at 12:01 PM, Piotr Kulaga <[email-masked]>wrote:

>  My understanding is that each websocket entity that uses UTF-8 must have
> a valid UTF-8 stream as a payload. This covers all UTF-8 frames including
> continuation frames.****
>
> ** **
>
> I slightly more prefer approach where UTF-8 message must contain valid
> sequence rather than each continuation frame (simpler for intermediaries,
> endpoints that stream data to application still must handle partial UTF-8
> code point encoding case). Fine with any approach as long as it is well
> defined.****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf Of
> *Takeshi Yoshino
> *Sent:* Thursday, July 21, 2011 2:22 AM
> *To:* Brian
> *Cc:* [email-masked]
> *Subject:* Re: [hybi] Fragmented text message****
>
> ** **
>
> separate frames.****
> ** **
> platforms have library like it.****
>
> ** **
>
> for that.****
> ** **
> On Thu, Jul 21, 2011 at 17:47, Brian <[email-masked]> wrote:****
>
> the whole message and then decode.****
>
> ** **
>
> Brian****
>
> ** **
>
> [email-masked]> wrote:****
>
> defragmented message?****
>

---

From: John Tamplin <[email-masked]>
Date: Thu, 21 Jul 2011 13:25:43 -0400

> That seems, to me, to be highly impractical.  Or even impossible.

Correct -- it seems much better to allow fragmentation at any byte boundary.
 Defragmentation of a text message already requires keeping state -- having
to keep state for a partial character seems hardly any extra burden.

-- 
John A. Tamplin
Software Engineer (GWT), Google