Subject: RE: [Sipping] SIGCOMP and large binary content SIP messages

From: Adam Roach <[email-masked]>
Date: Mon, 10 Mar 2003 17:18:34 -0600

Before this spins out of control...

1. SigComp works just fine with binary content. The only impact
   that the assumption cited from 3322 has had on SigComp is
   that the first byte of a message is examined to determine
   whether it is compressed. Since binary content in a SIP message
   will still have a text header, this causes no problems.

2. The TCP delimiter (0xFFFF) in SigComp is applied to the stream
   *after* it is compressed, not before. It has no harmful
   interactions with the content being sent. (And, yes, it is
   escaped if it is discovered in the compressed stream).

3. When the underlying binary data *can* be compressed, the efficiency
   of such compression is detemined by the compressor that the sender
   uses. At the SigComp interop event two weeks ago, everyone I tested
   with was using general-purpose compressors that would have generally
   done a decent job with such binary data.

4. When the underlying binary data *cannot* be compressed much further
   (e.g. mp3s, jpgs, etc), then it's pretty much a forgone conclusion
   that the compressor won't do a good job compressing it. We can't
   solve the problem of compressing uncompressible data.

Mark: an informational draft along the lines of what you propose
sounds like it might provide some needed guidance. I would add
to it the fact that compressors can decide to send a message
uncompressed if they determine that it has a detrimental effect
on the dynamic dictionary or actually grows in size. Further,
it would be a rather straightforward excercise to design a
compressor and complementary bytecodes that would compress the
headers, and just pass through the uncompressible binary content
untouched (solving both the dynamic dictonary problem and the
growing message problem). Finally, I'll point out that any
message in excess of 64 kb cannot be compressed with SigComp
in any fashion, since RFC 3320 mandates a 64 kb output buffer
size at the UDVM.

/a

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Monday, March 10, 2003 14:21
To: [email-masked]; [email-masked]
Subject: RE: [Sipping] SIGCOMP and large binary content SIP messages

Hi Mark,

I think it will really be a big deficiency if SigComp can't handle somehow
binary content in SIP messages. Presence (both PUBLISH and NOTIFY) is one
thing that can potentially have binary attachments, and not all of them are
going to be transported via content indirection. However, even more
important case is SIP MESSAGE. Already today mobile phones transfer images
of size ~50 kB using MMS, and I assume the same would happen if SIP were
available. And again, not all of this would use HTTP and CI. 

In addition to the dynamic dictionary problem I believe that there is a
problem with SIP/SigComp over TCP (which would be almost always the trasport
if binary content is included). I've understood that SigComp assumes that
some special non-ASCII character is used to indicate the start and end of a
compressed message. Obviously this does not work if part of the message
itself is non-ASCII. So is there even a bigger problem than just
non-friendliness that needs to be fixed?

Markus

-----Original Message-----
From: ext Mark Watson [mailto:[email-masked]]
Sent: 07 March, 2003 16:52
To: [email-masked]'
Subject: [Sipping] SIGCOMP and large binary content SIP messages

All, 
We have had a couple of short threads on SIMPLE considering the implications
of including large binary content within SIP messages (using MIME) (by
large, I mean anything more than several times the size of a normal INVITE
and by binary, I mean arbitrary bytes, rather than text). An example would
be a JPEG or audio file. It's been suggested that a user might include one
of these as part of their presence information.
On the face of it, it seems that this might have implications for the
effectiveness of SIPCOMP, firstly because such data may not itself compress
well (compared to SIP headers and SDP), and secondly because of the impact
this might have on any dynamic dictionary.
There may be ways to design SIGCOMP compressors which can cope with this,
and if so it would be good to have some guidelines included in the SIGCOMP
documentation. This would stop people from naively designing SIGCOMP
compressors based on false assumptions about the nature of SIP messaging
(for example SIGCOMP has to date made the assumption that SIP messages are
ASCII encoded - RFC3322).
So, before raising this in ROCH, perhaps we need to discuss what those
assumptions about SIP messaging should be - i.e. to approximately define a
class of SIP messages for which app. designers can expect SIGCOMP to perform
well. Then if applications need to generate messages outside that class,
they can choose between (i) potentially poor SIGCOMP performance (which may
be acceptable in some cases) or (ii) using techniques like Content
Indirection to return their messages into the 'SIGCOMP friendly' class.
I know large content in SIP messages has been discussed before in various
contexts, but is there any chance we can come to a rough consensus on what a
'SIGCOMP-friendly' class of SIP messages should look like ?
Regards, 
Mark Watson 

-----Original Message----- 
From: Watson, Mark [MOP:EP10:EXCH] 
Sent: 05 March 2003 10:33 
To: [email-masked]'; [email-masked] 
Subject: RE: [Simple] I-D ACTION:draft-lonnfors-simple-partial-notify-00.t
xt 

Mikko, All, 
The correct place to document such recommendations about SIGCOMP would be in
some ROCH documentation, perhaps the SIGCOMP User Guide ? I notice that
RFC3322 (Signalling Compression Requirements & Assumptions) already assumes
that the signalling messages are ASCII encoded, so if this is not the case I
think it's fairly urgent to get some discussion going.
Before we go to ROCH, though, we need to make some decision on what kind of
content we see being commonly included in SIP messages - i.e. what does
SIGCOMP have to deal with in addition to SIP Headers and SDP ??
To my mind this is a discussion for SIPPING - does anyone object to moving
this discussion there ? (Let's avoid creating a cross-posted thread if
possible - they're always a nightmare).
...Mark 

> -----Original Message----- 
> From: [email-masked] [mailto:[email-masked]] 
> Sent: 04 March 2003 10:03 
> To: Watson, Mark [MOP:EP10:EXCH]; [email-masked] 
> Subject: RE: [Simple] I-D 
> ACTION:draft-lonnfors-simple-partial-notify-00.txt 
> 
> 
> Hi mark, 
> 
> Inline inside <ML>: 
> 
> 
> All, 
> I am not convinced by the discussion on SIGCOMP which has 
> been introduced into this draft (Section 1.1). 
> Firstly, is this even the correct place to describe these issues ?? 
> 
> <ML> 
> Yes, you are right. To move discussion forward we needed to 
> place text somewhere and in this case only place we could 
> place it was the partial notification draft. 
> </ML> 
> 
>  If we really expect to see large binary content in SIP 
> messages, then we should document the impacts for SIGCOMP 
> somewhere more general - not buried in a presence partial 
> notifications discussion. 
> 
> <ML> 
> Yes 
> </ML> 
> 
> If we do not expect such content, we should document that 
> fact somewhere more general too. We need to make a decision 
> one way or the other because it has important implications 
> for the design of SIGCOMP compressors. 
> 
> <ML> 
> This is probably true. 
> </ML> 
> 
> Secondly, two of the solutions referred to involve the 
> SIGCOMP layer having knowledge of the content type, which is 
> a major architectural change. Presently, the SIGCOMP layer 
> does not parse SIP, MIME etc. 
> 
> <ML> 
> We have been talking about this issue with SIGCOMP author Z. 
> Liu (as at least I not an expert in this are) and these were 
> mentioned as possible solutions for this problem. I am also 
> not convinced that these two solutions are the best ones and 
> if other people feel same I am quite happy to drop them. 
> </ML> 
> 
> Thirdly, the last solution mentioned involves leaving 
> 'binary' content uncompressed, but this does not necessarily 
> mean that this content will not mess up the dynamic 
> dictionary. The compression scheme would need to detect 
> 'uncompressible' parts of the message and leave these out of 
> the dictionary. 
> 
> <ML> 
> As far as I know this should be quite easy thing to 
> accomplish for a SIGCOMP implementation. Of course it should 
> be specified (recommended) somewhere what is appropriate 
> threshold value for data which should be considered 'uncompressible'. 
> </ML> 
> 
> So, if we expect binary content in SIP, we should document 
> the impacts in SIGCOMP somewhere in the sip/sipping WGs. We 
> should decide whether we expect SIGCOMP to cope with such 
> binary content in an efficient way (i.e. if we think the same 
> content will be sent multiple times) or whether we are happy 
> for SIGCOMP to leave such content uncompressed (i.e. if we 
> think the same content will generally only be sent once). 
> 
> Finally, I think we concluded in the last discussion on this 
> that it was not appropriate to include in-line binary content 
> in the presence document itself due to encoding issues - 
> there is still some reference to this in the draft. Large 
> binary content would need to referenced from the presence 
> document, and we left it open whether this reference could be 
> to other MIME parts within the same SIP message. 
> 
> <ML> 
> Yes, this is our mistake. I also agree that large binary data 
> should not be in-line with other presence data. This will be 
> fixed in next version 
> </ML> 
> 
> Thanks for comments 
> - Mikko 
> 
> Regards, 
> Mark 
> > -----Original Message----- 
> > From: [email-masked] [mailto:[email-masked]] 
> > Sent: 24 February 2003 11:45 
> > Cc: [email-masked] 
> > Subject: [Simple] I-D 
> > ACTION:draft-lonnfors-simple-partial-notify-00.txt 
> > 
> > 
> > A New Internet-Draft is available from the on-line 
> > Internet-Drafts directories. 
> > 
> > 
> >       Title           : Partial Notification of Presence 
> Information 
> >       Author(s)       : J. Costa-Requena, E. Leppanen et al. 
> >       Filename        : draft-lonnfors-simple-partial-notify-00.txt 
> >       Pages           : 24 
> >       Date            : 2003-2-21 
> >       
> > A Presence service implemented using SIMPLE has some 
> constraints for 
> > delivering presence information to devices with low data processing 
> > capabilities, small display, and limited battery power. Other 
> > limitations can be caused by the interface between the terminal and 
> > the network, i.e. over radio links with high latency and low 
> > bandwidth. This memo presents a solution that aids in reducing the 
> > impact of those constrains and increasing efficiency, by 
> introducing 
> > a new MIME-type æpartial notificationsÆ and a Require header 
> > extension æpartial-notificationÆ. 
> > 
> > A URL for this Internet-Draft is: 
> > http://www.ietf.org/internet-drafts/draft-lonnfors-simple-part 
> > ial-notify-00.txt 
> > 
> > To remove yourself from the IETF Announcement list, send a 
> message to 
> > ietf-announce-request with the word unsubscribe in the body 
> > of the message. 
> > 
> > Internet-Drafts are also available by anonymous FTP. Login 
> > with the username 
> > "anonymous" and a password of your e-mail address. After 
> logging in, 
> > type "cd internet-drafts" and then 
> >       "get draft-lonnfors-simple-partial-notify-00.txt". 
> > 
> > A list of Internet-Drafts directories can be found in 
> > http://www.ietf.org/shadow.html 
> > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt 
> > 
> > 
> > Internet-Drafts can also be obtained by e-mail. 
> > 
> > Send a message to: 
> >       [email-masked]. 
> > In the body type: 
> >       "FILE 
> > /internet-drafts/draft-lonnfors-simple-partial-notify-00.txt". 
> >       
> > NOTE: The mail server at ietf.org can return the document in 
> >       MIME-encoded form by using the "mpack" utility.  To use this 
> >       feature, insert the command "ENCODING mime" before the "FILE" 
> >       command.  To decode the response(s), you will need 
> "munpack" or 
> >       a MIME-compliant mail reader.  Different MIME-compliant 
> > mail readers 
> >       exhibit different behavior, especially when dealing with 
> >       "multipart" MIME messages (i.e. documents which have 
> been split 
> >       up into multiple messages), so check your local 
> documentation on 
> >       how to manipulate these messages. 
> >               
> >               
> > Below is the data which will enable a MIME compliant mail reader 
> > implementation to automatically retrieve the ASCII version of the 
> > Internet-Draft. 
> > 
> 
_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: Pekka Pessi <[email-masked]>
Date: Tue, 11 Mar 2003 20:37:43 +0200

>   will still have a text header, this causes no problems.

	The problem is, as Zhigang said, determining the message
	boundaries within a single stream. If SigComp messages and
	uncompressed SIP messages are multiplexed within the same
	stream, decompressor dispatcher has to rely on application (SIP
	parser) to determine message boundaries. If the SIP message were
	nice ASCII or UTF-8, the decompressor dispatcher could do this
	by itself, but the SIP message body may contain "binary" data
	like HTML (which is Latin-1 by default).

....
>Finally, I'll point out that any

	On the other hand, I'd like to point out that a SIP message can
	be sent as a several SigComp messages.

						Pekka