Subject: RE: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP mes sages

From: "Lars-Erik Jonsson (EAB)" <[email-masked]>
Date: Thu, 3 Apr 2003 20:05:30 +0200

> However, what is NOT an implementation issue is that -
> looking at transport layer - a receiving endpoint must
> expect it may receive uncompressed messages interspersed
> with SigComp messages on the same port and demultiplex
> them using either of the above approaches (or any other
> working solutions). This needs to be spelled out in a
> document.

From what I remember, this was not the purpose of the
multiplexing mechanism in SigComp. The mechanism in
SigComp is not there to allow multiplexing of SigComp
messages and raw messages within the same connection, 
but only to identify whether the connection uses
SigComp or not. 

Carsten provided a clarification on this issue, which 
was rather clear, something we should put in the SigComp
implementer's guide.

/L-E
_______________________________________________
Rohc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rohc

---

From: [email-masked]
Date: Thu, 3 Apr 2003 18:17:02 -0600

Resend (original message exceeded 40 KB limit, so I removed older discussion texts.)
Zhigang
 
-----Original Message-----
From: Liu Zhigang.C (NRC/Dallas) 
Sent: April 03, 2003 6:08 PM
To: 'ext Mark Watson'; 'Price, Richard'; [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP mes sages

Mark,
 
This is exactly the concern I have regarding the SigComp performance implication. I see this as a question of *what* states to create. More precisely, a compressor should create only "useful" state(s) from the text part(s) of a message while avoiding creating "garbage" state(s) from the binary part(s). 
 
(Of course, it's possible that the binary part may also repeat across messages and thus become useful. But I wouldn't go there for simplicity.)
 
So, how can a compressor achieve that? Below are ideas I can think of. If they are correct, I can compile more "formal" text to be added to the SigComp user guide.
 
There are two steps involved. First, the compressor needs to know if current message to be compressed contain binary parts, and if yes, where they are located in the message. Second, the compressor needs to make sure it can selectively create states on the decompressor side.
 
For the first step, the best thing can happen is that the application (SIP in this case) gives hint to sigcomp compressor. This is possible, since it is a matter of local interface between application and sigcomp layer within the same endpoint. Otherwise (i.e. no help from application), the compressor has to figure it out by itself. In general, this seems to be a difficult problem to solve (if possible at all). But there are a few simple and practical approaches one can consider.
 
For example, the compressor can check the compression ratio after compressing the message. If it is close to 1, that is a hint that the current message probably contains large binary content. The compressor can choose not to create any state from current message. However, the compressor may miss a small text segment in the message that turns out being repeated later in the subsequent messages. The positive side is that at least the compressor can avoid creating "garbage states". 
 
Another factor to make a compressor's (or rather implementer's) life easier: we can assume that application messages have two parts, namely, a UTF-8 encoded part at the beginning followed by a binary part. (Note this is the case at least for SIP.) Then, the compressor can identify the UTF-8 part (by parsing) and use it to create state. This also has a good side-effect: compressor can avoid compressing the remaining binary part. 
 
Now to the second step: creating state on decompressor side. One issue I see is the limitation of decompression memory. On the decompressor side, the large binary content may overwrite text content previously stored in the ring/history buffer (part of the UDVM buffer). One solution is that the compressor should "segment" one large application message into multiple sigcomp messages, so that one of them contains only text data. This avoids the overwriting problem. (Note: I don't think this works for SigComp over UDP. But on the other hand, large SIP messages must be carried over TCP, not UDP.)
 
BR,
Zhigang

-----Original Message-----
From: ext Mark Watson [mailto:[email-masked]]
Sent: April 03, 2003 8:35 AM
To: 'Price, Richard'; Liu Zhigang.C (NRC/Dallas); [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP mes sages

Richard, all, 

The issue I would like to see addressed is as follows: 

The text part of SIP/SDP is obviously highly compressible using very simple algorithms which refer back to previous parts of the message/earlier messages. In fact, subsequent messages in a SIP dialog seem to consist almost entirely of repeated text from previous messages.

So, it seems important to me that large binary content in SIP messages does not destroy state that will be useful in compressing subsequent messages.

In a wireless environment, the principle advantage of compression is reduction in the time taken to actually send the message over the wireless link (this may even be more important than the overall reduction in signalling bandwidth).

Now if the message contains large binary content, then of course we are 'out of luck' on the message transfer time stakes, but this shouldn't affect subsequent messages.

I believe we should document recommendations to SIGCOMP compressor designers to ensure this aspect is not overlooked (although it is not immediately clear to me how the compressor (being unaware of the SIP message structure) might achieve this.)

Is this something that would be appropriate to item 4 below ? 

Regards...Mark

---

From: "Mark Watson" <[email-masked]>
Date: Fri, 4 Apr 2003 09:19:22 +0100

Resent to rohc as first try didn't work...

-----Original Message-----
From: Watson, Mark [MOP:EP10:EXCH] 
Sent: 03 April 2003 15:24
To: [email-masked]'; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP
messages

Zhigang,

If I understand the solution described by Robert correctly, then the problem
you describe below does not arise. Your problem seems to assume that
demultiplexing of compressed and uncompressed messages will be done before
the SIP parser is invoked, based on the 11111xxx delimiter.

My understanding is that messages are passed directly to the SIP parser.
This software must be aware that SIGCOMP may be used, and checks the first
octet of the input stream. If this is 11111xxx then it uses the sigcomp
delimeter OxFFFF to find the end of the compressed message, and then sends
it to the decompressor. If the first octets is not 11111xxx then it just
starts parsing as normal.

The only assumption is that the SIP parser has its own way of recognising
the end of an uncompressed message.

Regards...Mark

> From: [email-masked] [mailto:[email-masked]]
> Sent: 01 April 2003 14:44
> To: [email-masked]
> Subject: RE: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP
> messages
> 
> 
> Robert,
> 
> >  I don't think it is a problem with SigComp multiplexing, but 
> > more like a SIP internal issue. What if there is a TCP stream 
> > of SIP messages and the Content-Length header is missing or 
> > false? They will screw up the same way with or without 
> > SigComp. 
> 
> Since TCP is a reliable transport, this only happens for a
> mis-behaved SIP sender. But that is not the "multiplexing" problem 
> I mentioned below, which happens even "during normal functioning" 
> (see below).
> 
> > During normal functioning both SIP and SigComp 
> > provide their own flow separation mechanism and they work just fine.
> 
> I don't think so. That is what I intended to show by the example,
> i.e. the issue to "multiplex uncompressed application messages and 
> SigComp messages on the same port". One solution, as I mentioned
> before, is to require sigcomp receiver has the ability to parse
> uncompressed application messages and detect the end. 
> 
> (Of course, I saw Carsten's email and I need understand his
> point of view, which says there is no such "multiplexing" and
> it is not possible and necessary.)
> 
> BR,
> Zhigang
> Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
> This list is for NEW development of the application of SIP
> Use [email-masked] for questions on current sip
> Use [email-masked] for new developments of core SIP
>

---

From: "Mark Watson" <[email-masked]>
Date: Fri, 4 Apr 2003 09:19:44 +0100

-----Original Message-----
From: Watson, Mark [MOP:EP10:EXCH] 
Sent: 03 April 2003 15:35
To: 'Price, Richard'; [email-masked]'; [email-masked];
[email-masked]
Cc: [email-masked]; [email-masked]
Subject: RE: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP
mes sages

Richard, all,

The text part of SIP/SDP is obviously highly compressible using very simple
algorithms which refer back to previous parts of the message/earlier
messages. In fact, subsequent messages in a SIP dialog seem to consist
almost entirely of repeated text from previous messages.

So, it seems important to me that large binary content in SIP messages does
not destroy state that will be useful in compressing subsequent messages.

In a wireless environment, the principle advantage of compression is
reduction in the time taken to actually send the message over the wireless
link (this may even be more important than the overall reduction in
signalling bandwidth).

Now if the message contains large binary content, then of course we are 'out
of luck' on the message transfer time stakes, but this shouldn't affect
subsequent messages.

I believe we should document recommendations to SIGCOMP compressor designers
to ensure this aspect is not overlooked (although it is not immediately
clear to me how the compressor (being unaware of the SIP message structure)
might achieve this.)

Regards...Mark

> From: Price, Richard [mailto:[email-masked]]
> Sent: 26 March 2003 13:58
> To: [email-masked]'; [email-masked];
> [email-masked]
> mes sages
> Hi,
> I think that the issue with SigComp and large binary
> content SIP messages is part of a more general issue -
> namely that before using SigComp to compress a particular
> application we need to standardise the following:
> 
> 1. Mechanism for SigComp discovery
> 2. Resources offered by SigComp
> 3. How to delimit messages
> 4. When to save state
> 
> Currently the SIPPING WG has provided RFCs for 1 and 2,
> but not for 3 and 4. It would be useful to get a draft
> out for these ASAP, as it would clarify several issues on
> how to get SigComp working with SIP (including the issue
> with large binary content SIP messages).
> 
> > Unless someone objects, I'll submit it in ROHC.
> 
> I think that any WG draft should be a product of the
> SIPPING WG, because the proposed solution for delimiting
> SigComp messages will need to take into account the
> behaviour of SIP itself (or we won't be able to multiplex
> SigComp and SIP messages on the same port). However, it
> would be good to get the ROHC WG involved in generating
> the draft as well!
> 
> Regards,
> 
> Richard
> 
> P.S. For anyone who's interested, the mechanisms that
> particular applications need to provide before they can
> use SigComp are as follows...
> 
> ------------------------------------------------------
> 
> 1. Mechanism for SigComp discovery
> 
>    - How does the application know when to use SigComp?
> 
> For SIP this issue is already sorted (RFC 3486).
> 
> ------------------------------------------------------
> 
> 2. Resources offered by SigComp
> 
>    - Does the application need to offer more than the
>      minimum amount of resources?
> 
>    - Does the application need any extra state (e.g.
>      a static dictionary)?
> 
> For SIP we've already got a static dictionary (RFC 3485).
> However, we haven't currently discussed how much memory
> should be provided in a SigComp implementation for SIP.
> If we don't specify a value then SigComp defaults to the
> minimum possible (2K), which is rather tight for some of
> the more generously sized SIP messages!
> 
> ------------------------------------------------------
> 
> 3. How to delimit messages
> 
>    - How does the application distinguish between
>      SigComp messages and uncompressed messages?
> 
>    - Is the application allowed to use "continuous mode"?
> 
> This is the issue that's been raised for large binary
> content SIP messages. If we want to send a mix of
> uncompressed and SigComp messages within the same TCP
> connection, how do we delimit the end of one message and
> the start of another?
> 
> A related issue is how to handle very large SIP messages
> (> 64K). SigComp is able to compress large SIP messages
> by using "continuous mode", where the message boundaries
> are ignored completely and the application data is just
> transmitted as a stream. However, this mode of operation
> raises some additional security issues, so it's up to the
> SIPPING WG to decide whether or not to use it in SigComp
> for SIP.
> 
> ------------------------------------------------------
> 
> 4. When to save state
> 
>    - How much security is needed in order to save state?
> 
>    - Which messages belong to each state "compartment"?
> 
>    - How long should the application keep state?
> 
> This is quite an important issue that doesn't seem to
> have been discussed in the SIPPING WG. It's up to the
> application whether or not to let SigComp save state,
> and for interoperability it's important to have at
> least a general idea of how long to keep saved state,
> when to reject state due to insufficient security etc.
> 
> ------------------------------------------------------
> 
> > Sent: Tuesday, March 25, 2003 6:00 PM
> > To: [email-masked]; [email-masked]
> > Hi Carsten and Lars-Erik,
> > 
> > I've volunteered to submit a personal draft on this issue. From
> > ROHC WG chairs' point of view, do you think this is the way 
> > to proceed?
> > 
> > Of course, I understood during the IETF56 that Carsten had some
> > doubts on whether this is indeed an issue. My email below explained
> > the issue a bit. I don't know if it's clear. Either way, we 
> > can continue discuss this in ROHC mailing list
> > 
> http://www1.ietf.org/mail-archive/working-groups/sipping/curre
> nt/msg04115.html
> > 
> > BR,
> > Zhigang
> > 
> > > From: ext [email-masked] [mailto:[email-masked]]
> > > Sent: March 13, 2003 9:55 AM
> > > To: [email-masked]
> > > Cc: [email-masked]; Isomaki Markus (NRC/Helsinki);
> > > [email-masked]; [email-masked]; [email-masked]
> > > Subject: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP
> > > messages
> > > 
> > > 
> > > I agree. How to handling binary content is a generic issue for any
> > > application messages. What I have in mind is to give generic 
> > > descriptions/requirements (mainly on the SigComp receiver 
> parsing part),
> > > then touch on SIP (perhaps RTSP too) as a particular example.
> > > 
> > > Unless someone objects, I'll submit it in ROHC.
> > > 
> > > Zhigang
> > > 
> > > 
> > > > From: ext Gonzalo Camarillo 
> [mailto:[email-masked]]
> > > > Sent: March 13, 2003 5:36 AM
> > > > To: Liu Zhigang.C (NRC/Dallas)
> > > > Cc: [email-masked]; Isomaki Markus (NRC/Helsinki);
> > > > [email-masked]; [email-masked]
> > > > Subject: Re: [Sipping] SIGCOMP and large binary content 
> SIP messages
> > > > 
> > > > 
> > > > Zhigang,
> > > > 
> > > > you can submit that draft to ROHC. SIPPING can review 
> it afterwards.
> > > > IMO, the competence needed to generate this draft can 
> be found more
> > > > easily in ROHC than in SIPPING.
> > > > 
> > > > Besides, you could use SigComp to compress RTSP 
> messages, for instance
> > > > And RTSP is not handle by SIPPING either.
> > > > 
> > > > Gonzalo
> > > > 
> > > > [email-masked] wrote:
> > > > > 
> > > > > Yes, that is also what I had in mind for SIP. But my 
> point is that
> > > > > a SigComp receiver now needs to have some knowledge 
> about application
> > > > > message format and do parsing to determine the end of 
> an uncompressed
> > > > > message. This is something new (or at least 
> unspecified) in RFC 3320
> > > > > and needs to be spelled out in a new (standard track) 
> RFC. The document
> > > > > will be short, but I think it's needed for interoperability.
> > > > > 
> > > > > I'll write up an I-D and submit it after IETF 56. But 
> I'd like to know
> > > > > if folks in SIPPING think it should be done here or in ROHC.
> > > > > 
> > > > > Zhigang
> > > > > 
> > > > > > From: ext Adam Roach [mailto:[email-masked]]
> > > > > > Sent: March 11, 2003 9:03 PM
> > > > > > To: Liu Zhigang.C (NRC/Dallas); Adam Roach; Isomaki Markus
> > > > > > (NRC/Helsinki); [email-masked]; [email-masked]
> > > > > > Subject: RE: [Sipping] SIGCOMP and large binary 
> content SIP messages
> > > > > >
> > > > > >
> > > > > > > Having binary content in SIP messages may have 
> two impacts on
> > > > > > > SigComp:
> > > > > > >
> > > > > > > a) multiplexing of uncompressed messages with 
> SigComp messages on the
> > > > > > > same port.
> > > > > > ...
> > > > > > > For a), it is only an issue for SigComp/TCP, not 
> SigComp/UDP (where one
> > > > > > > SigComp message maps to one UDP packet). Adam's 
> point 1 and 2 are related
> > > > > > > to this. As to point 1, I think the SigComp 
> receiver still needs to know
> > > > > > > (somehow) the end of an uncompressed message, or 
> at least the end of the
> > > > > > > binary part of each SIP messages. Otherwise, a 
> bit pattern 11111 in
> > > > > > > the binary part will be mistaken by SigComp as 
> the beginning of a SigComp
> > > > > > > message. As to point 2, I would add that the 
> delimiter 0xFFFF can appear
> > > > > > > at the both the beginning and the end of a 
> SigComp message. So, the
> > > > > > > receiver needs to have the same parsing logic 
> just mentioned for bit pattern
> > > > > > > 11111.
> > > > > >
> > > > > > This isn't any more of an issue than it is without 
> SigComp. SIP
> > > > > > messages provide their own framing, and SigComp 
> messages do the
> > > > > > same. They can be trivially mixed on the same stream:
> > > > > >
> > > > > > 1. Peek at the first byte waiting in the stream.
> > > > > >
> > > > > > 2. If it is 0xf8 or higher, the next message in the 
> stream is
> > > > > >    a SigComp message. The message continues until a 
> 0xFFFF is
> > > > > >    encountered in the stream.
> > > > > >
> > > > > > 3. Otherwise, the next message in the stream is a 
> SIP message.
> > > > > >    Read the stream until a CR/LF/CR/LF is encountered. Parse
> > > > > >    the headers, and then read the number of bytes specified
> > > > > >    in the "Content-Length" header.
> > > > > >
> > > > > > 4. Return to step 1.
> > > > > >
> > > > > > /a
> > > > > >
> > > > > Sipping mailing list 
> https://www1.ietf.org/mailman/listinfo/sipping
> > > -- 
> > > Gonzalo Camarillo         Phone :  +358  9 299 33 71
> > > Oy L M Ericsson Ab        Mobile:  +358 40 702 35 35
> > > Telecom R&D               Fax   :  +358  9 299 30 52
> > > FIN-02420 Jorvas          Email :  [email-masked]
> > > Finland                   http://www.hut.fi/~gonzalo
> > >

---

From: "Price, Richard" <[email-masked]>
Date: Mon, 7 Apr 2003 17:27:16 +0100

> > Regarding point a), I don't see how this can be done without
> > talking about specific protocols. 
> 
> I agree if you mean the exact logic (see below). But what I had in 
> mind for a) is a generic or abstract logic as follows:
> 
> 0. Endpoint B informs endpoint A that it supports SigComp AND
> multiplexing of uncompressed message and SigComp message on
> a TCP connection. This should be part of the SigComp discovery.
> 
> 1. Endpoint B reads in one byte. 
> 
> 2. If the first 5 MSB of the byte is 11111. this is the beginning
> of a SigComp message or a SigComp message delimiter (0xFFFF). 
> Endpoint B enters "SigComp parsing mode" and processes byte stream
> according to SigComp until it encounters END-MESSAGE instruction 
> or Sigcomp delimiter. When that happens, go to 1. 
> 
> 3. Else, this is the beginning of an uncompressed message. Endpoint
> B enters "uncompressed parsing mode" and parses the byte stream 
> according to the delimiting scheme of the application messages. 
> When reaching the end of application message, go to 1. 
> 
> Now, back to your comment. I certainly agree with you that detecting 
> end of application message in Step 3 depends on a particular application 
> protocol. But other than that, the rest is pretty generic. We can 
> consider the above as the abstract logic that will be "instantiated" 
> when binding SigComp to a particular application.

Hi Zhigang,

Both of Step 1 and Step 2 are already documented in the SigComp RFC.
If Step 3 never arises then Endpoint B can decompress the entire
stream without any problem, using the delimiting scheme from RFC 3320.

Step 0 and Step 3 are not defined in RFC 3320, but this is ok because
both steps are 100% application-specific. If the endpoint ever reaches
Step 3 then there are at least four ways that it could proceed:

i)   Always fail (i.e. no multiplexing allowed at all).

ii)  Fail unless it's the very first byte of the stream. This is just
     enough to allow application and SigComp streams to be multiplexed
     on the same port, but not enough to multiplex messages on the same
     stream.

iii) Use SigComp-style delimiters for everything (i.e. for both the
     application and the SigComp messages).

iv)  Let the application decide where the message ends and when to
     hand the parsing back to SigComp.

It's up to particular applications to decide which strategy is the
most appropriate - currently I haven't seen a convincing reason why
we need anything more complex than ii) for the case of SIP.

Regards,

Richard