Subject: [Sipping] SIGCOMP and large binary content SIP messages

From: "Mark Watson" <[email-masked]>
Date: Fri, 7 Mar 2003 14:51:37 -0000

All,

We have had a couple of short threads on SIMPLE considering the implications
of including large binary content within SIP messages (using MIME) (by
large, I mean anything more than several times the size of a normal INVITE
and by binary, I mean arbitrary bytes, rather than text). An example would
be a JPEG or audio file. It's been suggested that a user might include one
of these as part of their presence information.

On the face of it, it seems that this might have implications for the
effectiveness of SIPCOMP, firstly because such data may not itself compress
well (compared to SIP headers and SDP), and secondly because of the impact
this might have on any dynamic dictionary.

There may be ways to design SIGCOMP compressors which can cope with this,
and if so it would be good to have some guidelines included in the SIGCOMP
documentation. This would stop people from naively designing SIGCOMP
compressors based on false assumptions about the nature of SIP messaging
(for example SIGCOMP has to date made the assumption that SIP messages are
ASCII encoded - RFC3322).

So, before raising this in ROCH, perhaps we need to discuss what those
assumptions about SIP messaging should be - i.e. to approximately define a
class of SIP messages for which app. designers can expect SIGCOMP to perform
well. Then if applications need to generate messages outside that class,
they can choose between (i) potentially poor SIGCOMP performance (which may
be acceptable in some cases) or (ii) using techniques like Content
Indirection to return their messages into the 'SIGCOMP friendly' class.

I know large content in SIP messages has been discussed before in various
contexts, but is there any chance we can come to a rough consensus on what a
'SIGCOMP-friendly' class of SIP messages should look like ?

Regards,

Mark Watson

-----Original Message-----
From: Watson, Mark [MOP:EP10:EXCH] 
Sent: 05 March 2003 10:33
To: [email-masked]'; [email-masked]
Subject: RE: [Simple] I-D ACTION:draft-lonnfors-simple-partial-notify-00.t
xt

Mikko, All, 
The correct place to document such recommendations about SIGCOMP would be in
some ROCH documentation, perhaps the SIGCOMP User Guide ? I notice that
RFC3322 (Signalling Compression Requirements & Assumptions) already assumes
that the signalling messages are ASCII encoded, so if this is not the case I
think it's fairly urgent to get some discussion going.
Before we go to ROCH, though, we need to make some decision on what kind of
content we see being commonly included in SIP messages - i.e. what does
SIGCOMP have to deal with in addition to SIP Headers and SDP ??
To my mind this is a discussion for SIPPING - does anyone object to moving
this discussion there ? (Let's avoid creating a cross-posted thread if
possible - they're always a nightmare).
...Mark 

> -----Original Message----- 
> From: [email-masked] [mailto:[email-masked]] 
> Sent: 04 March 2003 10:03 
> To: Watson, Mark [MOP:EP10:EXCH]; [email-masked] 
> Subject: RE: [Simple] I-D 
> ACTION:draft-lonnfors-simple-partial-notify-00.txt 
> 
> 
> Hi mark, 
> 
> Inline inside <ML>: 
> 
> 
> All, 
> I am not convinced by the discussion on SIGCOMP which has 
> been introduced into this draft (Section 1.1). 
> Firstly, is this even the correct place to describe these issues ?? 
> 
> <ML> 
> Yes, you are right. To move discussion forward we needed to 
> place text somewhere and in this case only place we could 
> place it was the partial notification draft. 
> </ML> 
> 
>  If we really expect to see large binary content in SIP 
> messages, then we should document the impacts for SIGCOMP 
> somewhere more general - not buried in a presence partial 
> notifications discussion. 
> 
> <ML> 
> Yes 
> </ML> 
> 
> If we do not expect such content, we should document that 
> fact somewhere more general too. We need to make a decision 
> one way or the other because it has important implications 
> for the design of SIGCOMP compressors. 
> 
> <ML> 
> This is probably true. 
> </ML> 
> 
> Secondly, two of the solutions referred to involve the 
> SIGCOMP layer having knowledge of the content type, which is 
> a major architectural change. Presently, the SIGCOMP layer 
> does not parse SIP, MIME etc. 
> 
> <ML> 
> We have been talking about this issue with SIGCOMP author Z. 
> Liu (as at least I not an expert in this are) and these were 
> mentioned as possible solutions for this problem. I am also 
> not convinced that these two solutions are the best ones and 
> if other people feel same I am quite happy to drop them. 
> </ML> 
> 
> Thirdly, the last solution mentioned involves leaving 
> 'binary' content uncompressed, but this does not necessarily 
> mean that this content will not mess up the dynamic 
> dictionary. The compression scheme would need to detect 
> 'uncompressible' parts of the message and leave these out of 
> the dictionary. 
> 
> <ML> 
> As far as I know this should be quite easy thing to 
> accomplish for a SIGCOMP implementation. Of course it should 
> be specified (recommended) somewhere what is appropriate 
> threshold value for data which should be considered 'uncompressible'. 
> </ML> 
> 
> So, if we expect binary content in SIP, we should document 
> the impacts in SIGCOMP somewhere in the sip/sipping WGs. We 
> should decide whether we expect SIGCOMP to cope with such 
> binary content in an efficient way (i.e. if we think the same 
> content will be sent multiple times) or whether we are happy 
> for SIGCOMP to leave such content uncompressed (i.e. if we 
> think the same content will generally only be sent once). 
> 
> Finally, I think we concluded in the last discussion on this 
> that it was not appropriate to include in-line binary content 
> in the presence document itself due to encoding issues - 
> there is still some reference to this in the draft. Large 
> binary content would need to referenced from the presence 
> document, and we left it open whether this reference could be 
> to other MIME parts within the same SIP message. 
> 
> <ML> 
> Yes, this is our mistake. I also agree that large binary data 
> should not be in-line with other presence data. This will be 
> fixed in next version 
> </ML> 
> 
> Thanks for comments 
> - Mikko 
> 
> Regards, 
> Mark 
> > -----Original Message----- 
> > From: [email-masked] [mailto:[email-masked]] 
> > Sent: 24 February 2003 11:45 
> > Cc: [email-masked] 
> > Subject: [Simple] I-D 
> > ACTION:draft-lonnfors-simple-partial-notify-00.txt 
> > 
> > 
> > A New Internet-Draft is available from the on-line 
> > Internet-Drafts directories. 
> > 
> > 
> >       Title           : Partial Notification of Presence 
> Information 
> >       Author(s)       : J. Costa-Requena, E. Leppanen et al. 
> >       Filename        : draft-lonnfors-simple-partial-notify-00.txt 
> >       Pages           : 24 
> >       Date            : 2003-2-21 
> >       
> > A Presence service implemented using SIMPLE has some 
> constraints for 
> > delivering presence information to devices with low data processing 
> > capabilities, small display, and limited battery power. Other 
> > limitations can be caused by the interface between the terminal and 
> > the network, i.e. over radio links with high latency and low 
> > bandwidth. This memo presents a solution that aids in reducing the 
> > impact of those constrains and increasing efficiency, by 
> introducing 
> > a new MIME-type æpartial notificationsÆ and a Require header 
> > extension æpartial-notificationÆ. 
> > 
> > A URL for this Internet-Draft is: 
> > http://www.ietf.org/internet-drafts/draft-lonnfors-simple-part 
> > ial-notify-00.txt 
> > 
> > To remove yourself from the IETF Announcement list, send a 
> message to 
> > ietf-announce-request with the word unsubscribe in the body 
> > of the message. 
> > 
> > Internet-Drafts are also available by anonymous FTP. Login 
> > with the username 
> > "anonymous" and a password of your e-mail address. After 
> logging in, 
> > type "cd internet-drafts" and then 
> >       "get draft-lonnfors-simple-partial-notify-00.txt". 
> > 
> > A list of Internet-Drafts directories can be found in 
> > http://www.ietf.org/shadow.html 
> > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt 
> > 
> > 
> > Internet-Drafts can also be obtained by e-mail. 
> > 
> > Send a message to: 
> >       [email-masked]. 
> > In the body type: 
> >       "FILE 
> > /internet-drafts/draft-lonnfors-simple-partial-notify-00.txt". 
> >       
> > NOTE: The mail server at ietf.org can return the document in 
> >       MIME-encoded form by using the "mpack" utility.  To use this 
> >       feature, insert the command "ENCODING mime" before the "FILE" 
> >       command.  To decode the response(s), you will need 
> "munpack" or 
> >       a MIME-compliant mail reader.  Different MIME-compliant 
> > mail readers 
> >       exhibit different behavior, especially when dealing with 
> >       "multipart" MIME messages (i.e. documents which have 
> been split 
> >       up into multiple messages), so check your local 
> documentation on 
> >       how to manipulate these messages. 
> >               
> >               
> > Below is the data which will enable a MIME compliant mail reader 
> > implementation to automatically retrieve the ASCII version of the 
> > Internet-Draft. 
> > 
>

---

From: [email-masked]
Date: Mon, 10 Mar 2003 22:21:19 +0200

Hi Mark,
 
I think it will really be a big deficiency if SigComp can't handle somehow binary content in SIP messages. Presence (both PUBLISH and NOTIFY) is one thing that can potentially have binary attachments, and not all of them are going to be transported via content indirection. However, even more important case is SIP MESSAGE. Already today mobile phones transfer images of size ~50 kB using MMS, and I assume the same would happen if SIP were available. And again, not all of this would use HTTP and CI. 
 
In addition to the dynamic dictionary problem I believe that there is a problem with SIP/SigComp over TCP (which would be almost always the trasport if binary content is included). I've understood that SigComp assumes that some special non-ASCII character is used to indicate the start and end of a compressed message. Obviously this does not work if part of the message itself is non-ASCII. So is there even a bigger problem than just non-friendliness that needs to be fixed?
 
Markus

-----Original Message-----
From: ext Mark Watson [mailto:[email-masked]]
Sent: 07 March, 2003 16:52
To: [email-masked]'
Subject: [Sipping] SIGCOMP and large binary content SIP messages

All, 

We have had a couple of short threads on SIMPLE considering the implications of including large binary content within SIP messages (using MIME) (by large, I mean anything more than several times the size of a normal INVITE and by binary, I mean arbitrary bytes, rather than text). An example would be a JPEG or audio file. It's been suggested that a user might include one of these as part of their presence information.

On the face of it, it seems that this might have implications for the effectiveness of SIPCOMP, firstly because such data may not itself compress well (compared to SIP headers and SDP), and secondly because of the impact this might have on any dynamic dictionary.

There may be ways to design SIGCOMP compressors which can cope with this, and if so it would be good to have some guidelines included in the SIGCOMP documentation. This would stop people from naively designing SIGCOMP compressors based on false assumptions about the nature of SIP messaging (for example SIGCOMP has to date made the assumption that SIP messages are ASCII encoded - RFC3322).

So, before raising this in ROCH, perhaps we need to discuss what those assumptions about SIP messaging should be - i.e. to approximately define a class of SIP messages for which app. designers can expect SIGCOMP to perform well. Then if applications need to generate messages outside that class, they can choose between (i) potentially poor SIGCOMP performance (which may be acceptable in some cases) or (ii) using techniques like Content Indirection to return their messages into the 'SIGCOMP friendly' class.

I know large content in SIP messages has been discussed before in various contexts, but is there any chance we can come to a rough consensus on what a 'SIGCOMP-friendly' class of SIP messages should look like ?

Regards, 

Mark Watson 

-----Original Message----- 
From: Watson, Mark [MOP:EP10:EXCH] 
Sent: 05 March 2003 10:33 
To: [email-masked]'; [email-masked] 
Subject: RE: [Simple] I-D ACTION:draft-lonnfors-simple-partial-notify-00.t xt 

Mikko, All, 
The correct place to document such recommendations about SIGCOMP would be in some ROCH documentation, perhaps the SIGCOMP User Guide ? I notice that RFC3322 (Signalling Compression Requirements & Assumptions) already assumes that the signalling messages are ASCII encoded, so if this is not the case I think it's fairly urgent to get some discussion going.

Before we go to ROCH, though, we need to make some decision on what kind of content we see being commonly included in SIP messages - i.e. what does SIGCOMP have to deal with in addition to SIP Headers and SDP ??

To my mind this is a discussion for SIPPING - does anyone object to moving this discussion there ? (Let's avoid creating a cross-posted thread if possible - they're always a nightmare).

...Mark 

> From: [email-masked] [ mailto:[email-masked]] 
> > From: [email-masked] [ mailto:[email-masked]]

---

From: [email-masked]
Date: Tue, 11 Mar 2003 10:03:44 -0600

Hi,

I was alerted by Markus about this. As a co-author of SigComp, I'd
like to give some clarifications.

Having binary content in SIP messages may have two impacts on SigComp: 

a) multiplexing of uncompressed messages with SigComp messages on the
same port.

b) performance of SigComp. 

For a), it is only an issue for SigComp/TCP, not SigComp/UDP (where one
SigComp message maps to one UDP packet). Adam's point 1 and 2 are related
to this. As to point 1, I think the SigComp receiver still needs to know 
(somehow) the end of an uncompressed message, or at least the end of the 
binary part of each SIP messages. Otherwise, a bit pattern 11111 in 
the binary part will be mistaken by SigComp as the beginning of a SigComp
message. As to point 2, I would add that the delimiter 0xFFFF can appear
at the both the beginning and the end of a SigComp message. So, the
receiver needs to have the same parsing logic just mentioned for bit pattern
11111.

I think a) is an interoperability issue and should be clarified by an I-D, 
probably standard track. If people agree, then the next question is if 
the I-D should be done here in SIPPING WG or in ROHC WG.

For b), it's more of an implementation issue. It can be bundled together
in the above document as recommendations.

BR,
Zhigang

> From: ext Adam Roach [mailto:[email-masked]]
> Sent: March 10, 2003 5:19 PM
> To: Isomaki Markus (NRC/Helsinki); [email-masked];
> [email-masked]
> Subject: RE: [Sipping] SIGCOMP and large binary content SIP messages
> 
> 
> Before this spins out of control...
> 
> 1. SigComp works just fine with binary content. The only impact
>    that the assumption cited from 3322 has had on SigComp is
>    that the first byte of a message is examined to determine
>    whether it is compressed. Since binary content in a SIP message
>    will still have a text header, this causes no problems.
> 
> 2. The TCP delimiter (0xFFFF) in SigComp is applied to the stream
>    *after* it is compressed, not before. It has no harmful
>    interactions with the content being sent. (And, yes, it is
>    escaped if it is discovered in the compressed stream).
> 
> 3. When the underlying binary data *can* be compressed, the efficiency
>    of such compression is detemined by the compressor that the sender
>    uses. At the SigComp interop event two weeks ago, everyone I tested
>    with was using general-purpose compressors that would have 
> generally
>    done a decent job with such binary data.
> 
> 4. When the underlying binary data *cannot* be compressed much further
>    (e.g. mp3s, jpgs, etc), then it's pretty much a forgone conclusion
>    that the compressor won't do a good job compressing it. We can't
>    solve the problem of compressing uncompressible data.
> 
> Mark: an informational draft along the lines of what you propose
> sounds like it might provide some needed guidance. I would add
> to it the fact that compressors can decide to send a message
> uncompressed if they determine that it has a detrimental effect
> on the dynamic dictionary or actually grows in size. Further,
> it would be a rather straightforward excercise to design a
> compressor and complementary bytecodes that would compress the
> headers, and just pass through the uncompressible binary content
> untouched (solving both the dynamic dictonary problem and the
> growing message problem). Finally, I'll point out that any
> message in excess of 64 kb cannot be compressed with SigComp
> in any fashion, since RFC 3320 mandates a 64 kb output buffer
> size at the UDVM.
> 
> /a
> 
> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, March 10, 2003 14:21
> To: [email-masked]; [email-masked]
> Subject: RE: [Sipping] SIGCOMP and large binary content SIP messages
> 
> 
> Hi Mark,
> 
> I think it will really be a big deficiency if SigComp can't 
> handle somehow
> binary content in SIP messages. Presence (both PUBLISH and 
> NOTIFY) is one
> thing that can potentially have binary attachments, and not 
> all of them are
> going to be transported via content indirection. However, even more
> important case is SIP MESSAGE. Already today mobile phones 
> transfer images
> of size ~50 kB using MMS, and I assume the same would happen 
> if SIP were
> available. And again, not all of this would use HTTP and CI. 
> 
> In addition to the dynamic dictionary problem I believe that 
> there is a
> problem with SIP/SigComp over TCP (which would be almost 
> always the trasport
> if binary content is included). I've understood that SigComp 
> assumes that
> some special non-ASCII character is used to indicate the 
> start and end of a
> compressed message. Obviously this does not work if part of 
> the message
> itself is non-ASCII. So is there even a bigger problem than just
> non-friendliness that needs to be fixed?
> 
> Markus
> 
> > >       Filename        : 
> draft-lonnfors-simple-partial-notify-00.txt 
> > >       Pages           : 24 
> > >       Date            : 2003-2-21 
> > >       
> _______________________________________________
> Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
> This list is for NEW development of the application of SIP
> Use [email-masked] for questions on current sip
> Use [email-masked] for new developments of core SIP
> 
_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Adam Roach <[email-masked]>
Date: Tue, 11 Mar 2003 21:03:29 -0600

> From: [email-masked] [mailto:[email-masked]]
> 
...
> receiver needs to have the same parsing logic just mentioned for bit
pattern

This isn't any more of an issue than it is without SigComp. SIP
messages provide their own framing, and SigComp messages do the
same. They can be trivially mixed on the same stream:

1. Peek at the first byte waiting in the stream.

2. If it is 0xf8 or higher, the next message in the stream is
   a SigComp message. The message continues until a 0xFFFF is
   encountered in the stream.

3. Otherwise, the next message in the stream is a SIP message.
   Read the stream until a CR/LF/CR/LF is encountered. Parse
   the headers, and then read the number of bytes specified
   in the "Content-Length" header.

4. Return to step 1.

/a

---

From: Adam Roach <[email-masked]>
Date: Tue, 11 Mar 2003 21:14:09 -0600

I will reiterate that compressors can decide to send a message
uncompressed if they determine that it has a detrimental effect
on the dynamic dictionary. They could also send them compressed,
but with an indication that the dynamic dictionary is not to
be updated. Finally, it would be a rather straightforward
excercise to design a compressor and complementary bytecodes
that would compress the headers, and just pass through any binary
content untouched (and without impact to the any dynamic dictionary).

As I said before, I think an informational document describing
the issues and techniques used to mitigate them would be useful.
I fully intended to stay out of this conversation until it was
implied that something was actually *broken* as opposed to merely
being sub-optimal when implemented thoughtlessly.

/a

-----Original Message-----
From: Mark Watson [mailto:[email-masked]]
Sent: Tuesday, March 11, 2003 8:38
To: 'Adam Roach'; [email-masked]'; [email-masked]
Subject: RE: [Sipping] SIGCOMP and large binary content SIP messages

Adam, 
I am not sure I agree with the statement 'SigComp works just fine with
binary content' - although I guess it depends what you mean by 'just fine'.
A very basic compression algorithm which replaces repeated text with a
reference back to a previous occurance, and which keeps a dynamic dictionary
of previous messages (within some reasonable size limit of a few k), will
really work very well with SIP headers & SDP. Throw some binary content in
and this blots out the dictionary, dramatically cutting performance for
subsequent SIP messages.
Ok, so it still 'works', in that nothing actually breaks, but it doesn't
work 'as well' as it should. This could be resolved if the compressor
designer had known to expect this kind of message i.e. if we document the
conditions under which we expect Sigcomp to perform well. 
Of course the inefficiency in compressing subsequent SIP messages probably
costs few extra bytes when compared to the size of the previous binary
content - you might argue that if you have enough bandwidth for the binary
content itself you shouldn't worry about some inefficienies in the
'aftermath' of this content.
However at least in a wireless context it is not just bandwidth which is a
problem but the transmission time for the messages - this may not be
important for a message containing some content (the user may expect this to
take some time to arrive), but it is important for session establishement
messaging.
Finally, I think different algorithms have a different 'run in' time before
they become really efficient (e.g. LZW-based algorithms will need to see a
good few messages before they build up a good codebook). If you tailor your
compression algorithm to SIP/SDP-type messaging then you can achieve high
compression much faster. For example it would be really useful if I knew
that the second occurance of a given SIP URI could always be replaced by a
reference in full. But you need to know what 'SIP/SDP-type messaging' looks
like to get this right.
...Mark 
> From: Adam Roach [mailto:[email-masked]] 
> Sent: 10 March 2003 23:19 
> To: [email-masked]'; Watson, Mark [MOP:EP10:EXCH]; 
> [email-masked]

---

From: "Mark Watson" <[email-masked]>
Date: Tue, 11 Mar 2003 14:38:14 -0000

Adam,

...Mark

---

From: [email-masked]
Date: Thu, 13 Mar 2003 09:54:31 -0600

I agree. How to handling binary content is a generic issue for any
application messages. What I have in mind is to give generic 
descriptions/requirements (mainly on the SigComp receiver parsing part),
then touch on SIP (perhaps RTSP too) as a particular example.

Unless someone objects, I'll submit it in ROHC.

Zhigang

> From: ext Gonzalo Camarillo [mailto:[email-masked]]
> Sent: March 13, 2003 5:36 AM
> To: Liu Zhigang.C (NRC/Dallas)
> Cc: [email-masked]; Isomaki Markus (NRC/Helsinki);
> [email-masked]; [email-masked]
> Subject: Re: [Sipping] SIGCOMP and large binary content SIP messages
> 
> 
> Zhigang,
> 
> you can submit that draft to ROHC. SIPPING can review it afterwards.
> IMO, the competence needed to generate this draft can be found more
> easily in ROHC than in SIPPING.
> 
> Besides, you could use SigComp to compress RTSP messages, for instance
> And RTSP is not handle by SIPPING either.
> 
> Gonzalo
> 
> [email-masked] wrote:
> > 
> > Yes, that is also what I had in mind for SIP. But my point is that
> > a SigComp receiver now needs to have some knowledge about 
> application
> > message format and do parsing to determine the end of an 
> uncompressed
> > message. This is something new (or at least unspecified) in RFC 3320
> > and needs to be spelled out in a new (standard track) RFC. 
> The document
> > will be short, but I think it's needed for interoperability.
> > 
> > I'll write up an I-D and submit it after IETF 56. But I'd 
> like to know
> > if folks in SIPPING think it should be done here or in ROHC.
> > 
> > Zhigang
> > 
> > > Sent: March 11, 2003 9:03 PM
> > > To: Liu Zhigang.C (NRC/Dallas); Adam Roach; Isomaki Markus
> > > (NRC/Helsinki); [email-masked]; [email-masked]
> > > > receiver needs to have the same parsing logic just 
> mentioned for bit
> > > pattern
> > > > 11111.
> > >
> -- 
> Gonzalo Camarillo         Phone :  +358  9 299 33 71
> Oy L M Ericsson Ab        Mobile:  +358 40 702 35 35
> Telecom R&D               Fax   :  +358  9 299 30 52
> FIN-02420 Jorvas          Email :  [email-masked]
> Finland                   http://www.hut.fi/~gonzalo
>

---

From: [email-masked]
Date: Tue, 25 Mar 2003 11:59:30 -0600

Hi Carsten and Lars-Erik,

I've volunteered to submit a personal draft on this issue. From
ROHC WG chairs' point of view, do you think this is the way to proceed?

Of course, I understood during the IETF56 that Carsten had some
doubts on whether this is indeed an issue. My email below explained
the issue a bit. I don't know if it's clear. Either way, we can continue
discuss this in ROHC mailing list
http://www1.ietf.org/mail-archive/working-groups/sipping/current/msg04115.html

BR,
Zhigang

> From: ext [email-masked] [mailto:[email-masked]]
> Sent: March 13, 2003 9:55 AM
> To: [email-masked]
> [email-masked]; [email-masked]; [email-masked]
> Subject: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP
> messages
> 
> 
> > From: ext Gonzalo Camarillo 
> [mailto:[email-masked]]
> > > Sipping mailing list  
https://www1.ietf.org/mailman/listinfo/sipping
Rohc mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rohc

---

From: [email-masked]
Date: Wed, 26 Mar 2003 19:00:32 -0600

Richard,

I agree on most of your comments. What I want to clarify and ask
for input is the way to organize documents. Based on comments
from Gonzalo and my own view, we should have *two* documents:

Document-1. Not specific to SIP. Deal with how to apply SigComp
to application protocol whose messages may contain binary (i.e.
non UTF-8 encoded) data. It should contain at least two parts:
a) multiplexing between uncompressed and SigComp messages on
the same TCP port, and b) performance implication. Part a) should
be normative and b) should be just recommendations.

Document-2. SIP Specific. Deal with issues on how to apply SigComp 
to SIP, e.g. default DMS, compartment mapping/management, how to
deal with binary data in SIP messages, etc.

If we agree on the two documents approach, I think Document-1 should
be ROHC WG document as it is not SIP specific. As to document-2, I
tend to agree with you that it should be done in SIPPING WG. But
I really don't have strong feeling, as long as it is done. 

During the ROHC meeting in IETF-56, I've volunteered to be the editor
of document-2. I see you've made many good points here. So, here is 
my proposal. I'll write a skeleton draft with TOC, fill in some sections
that looks obvious to me (e.g. DMS, binary content, and perhaps 
compartment mapping) and then send to you so you can change and fill with 
other issues you pointed out below. I'll try to do that in the next 2~3 weeks.
Hopefully, people will agree which WG to home the document by then. :-)

BR,
Zhigang

> From: ext Price, Richard [mailto:[email-masked]]
> Sent: March 26, 2003 7:58 AM
> To: Liu Zhigang.C (NRC/Dallas); [email-masked];
> [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: RE: [rohc] RE: [Sipping] SIGCOMP and large binary content SIP
> messages
> 
> 
> Hi,
> 
> I think that the issue with SigComp and large binary
> content SIP messages is part of a more general issue -
> namely that before using SigComp to compress a particular
> application we need to standardise the following:
> 
> 1. Mechanism for SigComp discovery
> 2. Resources offered by SigComp
> 3. How to delimit messages
> 4. When to save state
> 
> Currently the SIPPING WG has provided RFCs for 1 and 2,
> but not for 3 and 4. It would be useful to get a draft
> out for these ASAP, as it would clarify several issues on
> how to get SigComp working with SIP (including the issue
> with large binary content SIP messages).
> 
> I think that any WG draft should be a product of the
> SIPPING WG, because the proposed solution for delimiting
> SigComp messages will need to take into account the
> behaviour of SIP itself (or we won't be able to multiplex
> SigComp and SIP messages on the same port). However, it
> would be good to get the ROHC WG involved in generating
> the draft as well!
> 
> Regards,
> 
> Richard
> 
> P.S. For anyone who's interested, the mechanisms that
> particular applications need to provide before they can
> use SigComp are as follows...
> 
> ------------------------------------------------------
> 
> 1. Mechanism for SigComp discovery
> 
>    - How does the application know when to use SigComp?
> 
> For SIP this issue is already sorted (RFC 3486).
> 
> ------------------------------------------------------
> 
> 2. Resources offered by SigComp
> 
>    - Does the application need to offer more than the
>      minimum amount of resources?
> 
>    - Does the application need any extra state (e.g.
>      a static dictionary)?
> 
> For SIP we've already got a static dictionary (RFC 3485).
> However, we haven't currently discussed how much memory
> should be provided in a SigComp implementation for SIP.
> If we don't specify a value then SigComp defaults to the
> minimum possible (2K), which is rather tight for some of
> the more generously sized SIP messages!
> 
> ------------------------------------------------------
> 
> 3. How to delimit messages
> 
>    - How does the application distinguish between
>      SigComp messages and uncompressed messages?
> 
>    - Is the application allowed to use "continuous mode"?
> 
> This is the issue that's been raised for large binary
> content SIP messages. If we want to send a mix of
> uncompressed and SigComp messages within the same TCP
> connection, how do we delimit the end of one message and
> the start of another?
> 
> A related issue is how to handle very large SIP messages
> (> 64K). SigComp is able to compress large SIP messages
> by using "continuous mode", where the message boundaries
> are ignored completely and the application data is just
> transmitted as a stream. However, this mode of operation
> raises some additional security issues, so it's up to the
> SIPPING WG to decide whether or not to use it in SigComp
> for SIP.
> 
> ------------------------------------------------------
> 
> 4. When to save state
> 
>    - How much security is needed in order to save state?
> 
>    - Which messages belong to each state "compartment"?
> 
>    - How long should the application keep state?
> 
> This is quite an important issue that doesn't seem to
> have been discussed in the SIPPING WG. It's up to the
> application whether or not to let SigComp save state,
> and for interoperability it's important to have at
> least a general idea of how long to keep saved state,
> when to reject state due to insufficient security etc.
> 
> ------------------------------------------------------
> 
> > Sent: Tuesday, March 25, 2003 6:00 PM
> > To: [email-masked]; [email-masked]
> > Cc: [email-masked]; [email-masked]
> > Subject: RE: [rohc] RE: [Sipping] SIGCOMP and large binary 
> content SIP
> > messages
> > 
> > 
> > the issue a bit. I don't know if it's clear. Either way, we 
> > can continue discuss this in ROHC mailing list
> > 
> > > > > a SigComp receiver now needs to have some knowledge 
> about application
> > > > > message format and do parsing to determine the end of 
> an uncompressed
> > > > > and needs to be spelled out in a new (standard track) 
> RFC. The document
> > > > > I'll write up an I-D and submit it after IETF 56. But 
> I'd like to know
> > > > > > > Having binary content in SIP messages may have 
> two impacts on
> > > > > > > SigComp:
> > > > > > >

---

From: [email-masked]
Date: Thu, 27 Mar 2003 19:50:29 -0600

Lars-Erik,

IMO, binary content in application message is a general issue for
SigComp multiplexing. It's not specific to SIP. I already explained 
this in rohc and sipping mailing list. But here is a simple example. 

Let's say endpoint A already knows endpoint B supports SigComp on TCP 
port number 5000. A sends, on port 5000, uncompressed message 1 followed 
by message 2 compressed using SigComp. From RFC 3320, message 2 starts
with bit pattern 11111. So, if message 1 has no binary content, endpoint
B can easily detect the beginning of message 2 (i.e. end of message 1)
by detecting 11111. However, if message 1 contains binary content,
that means the (uncompressed) binary part may contain byte 11111xxx.
This "mimicking" will make endpoint B mistakenly think a new SigComp
message starts at that byte. Obviously, things will be messed up
from that point.

This is what I referred to as the multiplexing issue. I hope I explained
it better this time for you. 

Anyway, I'll submit a personal draft to help the discussion, although the 
above example should make it obvious already.

Zhigang

> From: ext Lars-Erik Jonsson (EAB)
> [mailto:[email-masked]]
> Sent: March 27, 2003 2:18 AM
> Cc: [email-masked]
> > My question is the handling of binary content in application 
> > messages. In my view, this is a generic issue although it started
> > with SIP. My thought was to have a *separate* document that
> > addresses this issue (and perhaps other generic SigComp "bugs" if
> > we find out).
> 
> First of all, it is not yet clear to me that this is a general issue,
> at least I have not been able to see consensus on that. Secondly, if
> we have general unclear issues with SigComp, those would go into the
> Implementer's guide at this point.
> 
> Currently, I therefore think we can and should avoid having even more
> documents.  However, you can always provide input by writing a draft,
> that is up to you.
> 
> Regarding the SigComp&SIP document, it does not matter much
> in which WG it is "hosted", as long as it is written and well 
> reviewed 
> by people from both sides.
> 
> 
> Rgds,
> /L-E

---

From: [email-masked]
Date: Fri, 28 Mar 2003 09:36:09 +0200

SIP has this header called content-length that tells how long the body of a sip message is regardless of its binary of UTF-8 contents.

The SIP parser will read the content-length value and then take that many octets from whatever follows the sip headers as body. If the body has binary with pattern 11111, the parser doesn't care and doesn't even look in the body.

Any protocol that wants to make use of sigcomp should fix its own problems.

Regards,
Hisham

> Sent: Friday, March 28, 2003 3:50 AM
> To: [email-masked]

---

From: [email-masked]
Date: Fri, 28 Mar 2003 13:59:25 +0530

HI,
I have some doubts.
Content length will come into picture only after decompression and parsing
of  the incomming message.
But if the parsing or decompression fails , what will happen to the
subsequent messages in the stream.

Regards
mahesh

[email-masked] on 03/28/2003 01:06:09 PM

To:   [email-masked], [email-masked]
cc:   [email-masked], [email-masked] (bcc: Mahesh Govind/HSSBLR)

SIP has this header called content-length that tells how long the body of a
sip message is regardless of its binary of UTF-8 contents.

The SIP parser will read the content-length value and then take that many
octets from whatever follows the sip headers as body. If the body has
binary with pattern 11111, the parser doesn't care and doesn't even look in
the body.

Regards,
Hisham

---

From: [email-masked]
Date: Fri, 28 Mar 2003 08:24:09 +0100

Hi Zhigang,

 I don't think it is a problem with SigComp multiplexing, but more like a SIP internal issue. What if there is a TCP stream of SIP messages and the Content-Length header is missing or false? They will screw up the same way with or without SigComp. During normal functioning both SIP and SigComp provide their own flow separation mechanism and they work just fine.
 As for other applications the requirement is that the messages must not start with 11111xxx and the end of the messages must be detected. If that is assured binary content is not a problem anymore. Besides, whatever pattern you choose to distinguish SigComp it might appear in application messages <evil> for example if we attach a Sigcomp message as application/octet stream </evil>. Only the possibility for screwing up will be lower.

Best Regards,
Robert

> Sent: March 28,2003 2:50