Subject: RE: [Sip] SIPit21: BNF future-proofing problem?

From: "T Satyanarayana-A12694" <[email-masked]>
Date: Thu, 22 Nov 2007 13:43:21 +0800

This would mean a large change to existing parsers and formatters. And
also, the "escaping" and "unescaping" overheads.

Changing the header-value BNF may be more acceptable?

Regards
Satya T

-----Original Message-----
From: Christer Holmberg [mailto:[email-masked]] 
Sent: Wednesday, November 21, 2007 11:13 PM
To: Christer Holmberg; Robert Sparks
Cc: sip List
Subject: RE: [Sip] SIPit21: BNF future-proofing problem?

Hi,
 
Another alternative would be to define an escaping mechanism for
quoted-strings in SIP messages, and require all characters not allowed
by the header-value ABNF to be escaped.
 
Regards,
 
Christer 

________________________________

From: Christer Holmberg [mailto:[email-masked]]
Sent: Wed 21/11/2007 15:33
To: Robert Sparks
Cc: sip List
Subject: RE: [Sip] SIPit21: BNF future-proofing problem?

Hi,

>So to restate the problem that was reported:
>
>Right now, extension-header does not allow a new header that contains a

>quoted-string.
>
>We have defined extensions (and are likely to define new
>ones) that use quoted-string (anything that uses name-addr for 
>instance).
>
>This inconsistency has caused interoperability problems in real 
>implementations.
>
>My original note was to suggest that we change extension-header to 
>actually allow the headers we're going to define.
>
>Christer's response could be read as a proposal to change
>quoted- string to achieve the same goal.

Yes.

Unless someone can show a use-case where the x00-x20 characters would be
needed, for backward compability reasons I think it's better to restrict
quoted-string than to extend extension-header.

Regards,

Christer

> > Hi,
> >
> > I am still a little unclear what Robert is proposing, but is there 
> > really a need to be able to use characters between x00 and
> x20 in SIP
> > messages? Characters between x21 and x7F, and between x80
> and xBF, are
> > ok since they are covered by TEXT-UTF8char and UTF8-CONT.
> >
> > Regards,
> >
> > Christer
> >
> >
> > ________________________________
> >
> >     From: Hisham Khartabil [mailto:[email-masked]]
> >     Sent: 21. marraskuuta 2007 7:30
> >     To: Robert Sparks
> >     Cc: sip List
> >     Subject: Re: [Sip] SIPit21: BNF future-proofing problem?
> >    
> >    
> >     I think I did misread Robert's original email. I thought header 
> > values having quoted strings are currently not allowed and Robert 
> > wanted to change RFC2822. Now I realise that you can have
> header field
> > values with quoted string. Therefore I support the correction that 
> > Robert is proposing.
> >    
> >     Hisham
> >    
> >    
> >     On 21/11/2007, Hisham Khartabil <[email-masked]>
> > wrote:
> >
> >             This is a big change if we do adopt it. It will
> cause a lot of
> > problem to parsers that handle extenion headers today and is not 
> > backwards compatible. Why isn't the extension header as is defined 
> > today not sufficient?
> >                            
> >             Hisham
> >            
> >            
> >                                             On 20/11/2007, Robert 
> > Sparks <[email-masked] > wrote:
> >
> >                     The BNF in 3261 says the following:
> >                    
> >                     extension-header  =  header-name HCOLON
> header-value
> >                     header-value      =  *(TEXT-UTF8char / UTF8-CONT
> > / LWS)
> >                    
> >                     This is intended to be the catch-all
> field for all future
> > extensions
> >                     - older parsers working against this
> BNF shouldn't barf
> >                     when we introduce a new header field.
> >                    
> >                     Now, we may have new fields in the
> future that look like:
> >                    
> >                     NewHeader = new-header-name HCOLON quoted-string
> >                    
> >                     And down inside quoted-string, we get:
> >                    
> >                           quoted-string  =  SWS DQUOTE
> *(qdtext / quoted-pair ) DQUOTE
> >                           qdtext         =  LWS / %x21 / %x23-5B /
> > %x5D-7E
> >                                             / UTF8-NONASCII
> >                           quoted-pair  =  "\" (%x00-09 / %x0B-0C
> >                                            / %x0E-7F)
> >                    
> >                     So, for instance, we could have inside
> a quoted string the 2 byte
> >                     sequence \ NULL
> >                    
> >                     This does not parse against
> header-value above...
> >                    
> >                     Is this a problem? Some of the SIPit21
> participants argued that it
> > is.
> >                    
> >                     The projects I've been involved in
> don't parse unknown headers and
> >                     the stacks will just hand up an
> unparsed bucket of bits (the only
> > rules
> >                     used are those necessary to identify
> the next header-field
> > starting).
> >                    
> >                     Would it be worth the effort to make
> the BNF reflect that rather
> > than
> >                     continuing with the incongruity that we
> currently specify?
> >                    
> >                     RjS
> >                    
> >                    
> >                     _______________________________________________
> >                     Sip mailing list 
> > https://www1.ietf.org/mailman/listinfo/sip
> > <https://www1.ietf.org/mailman/listinfo/sip>
> >                     This list is for NEW development of the
> core SIP Protocol
> >                     Use [email-masked]
> for questions on current sip
> >                     Use [email-masked] for new
> developments on the application of sip
> >                    
> >
> >
> >            
>
>

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol Use
[email-masked] for questions on current sip Use
[email-masked] for new developments on the application of sip

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol Use
[email-masked] for questions on current sip Use
[email-masked] for new developments on the application of sip

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: "Christer Holmberg" <[email-masked]>
Date: Tue, 4 Dec 2007 14:13:43 +0100

Hi,

>also, the "escaping" and "unescaping" overheads.

Existing parsers would work, in the sense that they would not report error due to unallowed characters. 

But, if you are interested in the actual escapted value you would of course have to add the "unescaping" functionality.

>Changing the header-value BNF may be more acceptable?

Any solution which EXTENDS what we have today may cause problems with existing parsers.

Regards,

Christer

-----Original Message-----
From: Christer Holmberg [mailto:[email-masked]]
Sent: Wednesday, November 21, 2007 11:13 PM
To: Christer Holmberg; Robert Sparks
Cc: sip List
Subject: RE: [Sip] SIPit21: BNF future-proofing problem?

Hi,

Regards,

Christer

Hi,

Yes.

Regards,

Christer

---

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 05 Dec 2007 17:15:07 -0500

This strikes me as a problem in theory and not in practice. AFAICT, the 
only characters you can't use are ascii 0x00-0x19 (0x20 is space and is 
possible through the LWS construct as Christer pointed out). Is there a 
practical use case for any of these? They are:

   Decimal   Octal   Hex    Binary     Value
        -------   -----   ---    ------     -----
          000      000    000   00000000      NUL    (Null char.)
          001      001    001   00000001      SOH    (Start of Header)
          002      002    002   00000010      STX    (Start of Text)
          003      003    003   00000011      ETX    (End of Text)
          004      004    004   00000100      EOT    (End of Transmission)
          005      005    005   00000101      ENQ    (Enquiry)
          006      006    006   00000110      ACK    (Acknowledgment)
          007      007    007   00000111      BEL    (Bell)
          008      010    008   00001000       BS    (Backspace)
          009      011    009   00001001       HT    (Horizontal Tab)
          010      012    00A   00001010       LF    (Line Feed)
          011      013    00B   00001011       VT    (Vertical Tab)
          012      014    00C   00001100       FF    (Form Feed)
          013      015    00D   00001101       CR    (Carriage Return)
          014      016    00E   00001110       SO    (Shift Out)
          015      017    00F   00001111       SI    (Shift In)
          016      020    010   00010000      DLE    (Data Link Escape)
          017      021    011   00010001      DC1 (XON) (Device Control 1)
          018      022    012   00010010      DC2       (Device Control 2)
          019      023    013   00010011      DC3 (XOFF)(Device Control 3)
          020      024    014   00010100      DC4       (Device Control 4)
          021      025    015   00010101      NAK    (Negative 
Acknowledgement)
          022      026    016   00010110      SYN    (Synchronous Idle)
          023      027    017   00010111      ETB    (End of Trans. Block)
          024      030    018   00011000      CAN    (Cancel)
          025      031    019   00011001       EM    (End of Medium)
          026      032    01A   00011010      SUB    (Substitute)
          027      033    01B   00011011      ESC    (Escape)
          028      034    01C   00011100       FS    (File Separator)
          029      035    01D   00011101       GS    (Group Separator)
          030      036    01E   00011110       RS    (Request to 
Send)(Record Separator)
          031      037    01F   00011111       US    (Unit Separator)

I don't think this is a real problem.

-Jonathan R.

Robert Sparks wrote:
> extension-header  =  header-name HCOLON header-value
> header-value      =  *(TEXT-UTF8char / UTF8-CONT / LWS)
> 
> This is intended to be the catch-all field for all future extensions - 
> older parsers working against this BNF shouldn't barf
> Now, we may have new fields in the future that look like:
> 
>       quoted-string  =  SWS DQUOTE *(qdtext / quoted-pair ) DQUOTE
>       qdtext         =  LWS / %x21 / %x23-5B / %x5D-7E
>                         / UTF8-NONASCII
> So, for instance, we could have inside a quoted string the 2 byte 
> sequence \ NULL
> 
> This does not parse against header-value above...
> 
> Is this a problem? Some of the SIPit21 participants argued that it is.
> 
> The projects I've been involved in don't parse unknown headers and the 
> stacks will just hand up an unparsed bucket of bits (the only rules
> used are those necessary to identify the next header-field starting).
> 
> Would it be worth the effort to make the BNF reflect that rather than 
> continuing with the incongruity that we currently specify?
> 
> RjS
> 
> 

-- 
Jonathan D. Rosenberg, Ph.D.                   499 Thornall St.
Cisco Fellow                                   Edison, NJ 08837
Cisco, Voice Technology Group
[email-masked]
http://www.jdrosen.net                         PHONE: (408) 902-3084
http://www.cisco.com

---

From: [email-masked]
Date: Thu, 6 Dec 2007 15:09:21 -0500

From: Jonathan Rosenberg <[email-masked]>

HT (horizontal tab) should be included, but that's also in LWS.  As
for the remaining characters, I don't think that anyone has ever
*intended* that they be used in SIP headers, and their inclusion in
quoted-pair was a typographical error.

Dale

---

From: "Christer Holmberg" <[email-masked]>
Date: Thu, 6 Dec 2007 22:18:09 +0100

Hi,
 
No matter how we'll fix this, I guess it would be a valid essential correction?
 
Regards,
 
Christer

Lähettäjä: [email-masked] [mailto:[email-masked]]
Lähetetty: to 6.12.2007 21:09
Vastaanottaja: [email-masked]
Aihe: Re: [Sip] SIPit21: BNF future-proofing problem?

Dale

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 13 Dec 2007 00:05:56 -0500

Correction, yes. Essential? Hardly.

This is another question around essential corrections. I think they not 
only need to clearly represent a bug or a fix, but need to be one that 
is important enough to merit documenting. Just like software, sometimes 
you don't ever bother fixing those sev 5s. Ever.

-Jonathan R.

---

From: Thomas Froment <[email-masked]>
Date: Thu, 13 Dec 2007 09:43:04 +0100

Jonathan Rosenberg wrote:
> This is another question around essential corrections. I think they 
> not only need to clearly represent a bug or a fix, but need to be one 
> that is important enough to merit documenting. Just like software, 
> sometimes you don't ever bother fixing those sev 5s. Ever.
Just would like to raise a small warning with the "essential" vs 
"non-essential" debate... Sometimes the most "costly" bugs (in terms of 
interoperability problems) are the smallest stupid nits.
Just one example, the FIRST proxy example of the RFC 3665 is broken... 
Contact of subsequent request is not correctly pushed in the Request URI 
with transport parameter, so, at SIPIT, I already have seen many UAs
that behave incorrectly because of this... Ok this is clearly written in 
3261, but you know how developers do: "don't have time to read this long 
RFC, I will just look at the example :p"... And
the place you generally look at is 3665, and then, it is broken (!)... 
This is not "fixed" and I assume it will probably never be... By the way 
this is a question: would it be interesting for somebody to
fix 3665 examples, and if yes, how to proceed?
+
Thomas

---

From: "Christer Holmberg" <[email-masked]>
Date: Thu, 13 Dec 2007 14:09:08 +0100

My opinion is that people who only implement based on examples, without reading the actual protocol specification, have themselves to blame - and they are going to run into problems sooner or later, no matter how correct the exmaples are. 

If people are too lazy to read 3261 I don't think there is anything IETF can do - or even should spend time on doing.

Regards,

Christer 

> From: Thomas Froment [mailto:[email-masked]] 
> Sent: 13. joulukuuta 2007 10:43
> To: Jonathan Rosenberg
> Cc: Christer Holmberg; [email-masked]
> Subject: Re: VS: [Sip] SIPit21: BNF future-proofing problem?
> 
> Just would like to raise a small warning with the "essential" 
> vs "non-essential" debate... Sometimes the most "costly" bugs 
> (in terms of interoperability problems) are the smallest stupid nits.
> Contact of subsequent request is not correctly pushed in the 
> Request URI with transport parameter, so, at SIPIT, I already 
> have seen many UAs that behave incorrectly because of this... 
> Ok this is clearly written in 3261, but you know how 
> developers do: "don't have time to read this long RFC, I will 
> just look at the example :p"... And the place you generally 
> look at is 3665, and then, it is broken (!)... 
> This is not "fixed" and I assume it will probably never be... 
> By the way this is a question: would it be interesting for 
> somebody to fix 3665 examples, and if yes, how to proceed?
> +
> Thomas
> 
> 
> >
> > -Jonathan R.
> >
> > Christer Holmberg wrote:
> >> Hi,
> >>  
> >> No matter how we'll fix this, I guess it would be a valid 
> essential 
> >> correction?
> >>  
> >> Regards,
> >>  
> >> Christer
> >>
> >>    only characters you can't use are ascii 0x00-0x19 (0x20 
> is space 
> >> and is

---

From: Thomas Froment <[email-masked]>
Date: Thu, 13 Dec 2007 14:16:33 +0100

> If people are too lazy to read 3261 I don't think there is anything IETF can do - or even should spend time on doing.
I agree, but if you don't care about the first example of the most basic 
example of an RFC being broken and not fixed, you should not be 
surprised if interoperability problems occur
with new implementations.. This was the topic of interoperability 
workshop: new implementors seem to all fall in the same "traps".
I just wanted to outline that "small fix" is not synonym of "non-essential".

+

---

From: "Christer Holmberg" <[email-masked]>
Date: Thu, 13 Dec 2007 18:44:20 +0100

This is how it should be:
 
1. I implement RFC3261 (and whatever extensions needed)
 
2. I start to test, and I realize that the outcome is not according to the example.
 
3. I send an e-mail to some SIP list, get a reply that there is a bug in the example and that it was good that I found it, and that it will be reported in some bugzilla.
 
4. I interop with other people who have done the same
 
Of course it is bad if there are bugs in an RFC, but my point is that fixing an example call flow is probably not going to help interoperability very much in the long run, if people don't read the protocol spec...
 
Regards,
 
Christer

From: Thomas Froment [mailto:[email-masked]]
Sent: Thu 13/12/2007 14:16
To: Christer Holmberg
Cc: [email-masked]
Subject: Re: VS: [Sip] SIPit21: BNF future-proofing problem?

+

---

From: "Christer Holmberg" <[email-masked]>
Date: Sat, 15 Dec 2007 13:35:37 +0100

Hi,

>you don't ever bother fixing those sev 5s. Ever.

So, who makes the decission whether a bug (I guess we all agree it IS a bug) is "important enough"?

My understanding is that this bug was found at SIPit, so it is not pure "theoretical".

I do agree with you that it shouldn't occur that often, because I see no reason why one would need to use the affected characters. 

But, never the less, the fix MAY be important from a parser implementation perspective.

Regards,

Christer

-Jonathan R.

--
Jonathan D. Rosenberg, Ph.D.                   499 Thornall St.
Cisco Fellow                                   Edison, NJ 08837
Cisco, Voice Technology Group
[email-masked]
http://www.jdrosen.net <http://www.jdrosen.net/>                          PHONE: (408) 902-3084
http://www.cisco.com <http://www.cisco.com/>

---

From: "T Satyanarayana-A12694" <[email-masked]>
Date: Sat, 15 Dec 2007 20:53:52 +0800

Just to correct the context here :

This bug was not FOUND at SIPit, in any test scenario. But, was rather pointed out by me as an issue in the spec.

While it is unlikely that we would face this issue in a "real" situation, as Christer said, it will be helpful (and unambiguous) for parser implementors.

Regards
Satya T

-----Original Message-----
From: Christer Holmberg [mailto:[email-masked]] 
Sent: Saturday, December 15, 2007 6:06 PM
To: Jonathan Rosenberg
Cc: [email-masked]
Subject: RE: VS: [Sip] SIPit21: BNF future-proofing problem?

Hi,

Regards,

Christer

-Jonathan R.

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol Use [email-masked] for questions on current sip Use [email-masked] for new developments on the application of sip

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sat, 15 Dec 2007 08:39:08 -0500

Christer Holmberg wrote:
>> not only need to clearly represent a bug or a fix, but need to be
>> one that is important enough to merit documenting. Just like
>> software, sometimes you don't ever bother fixing those sev 5s.
>> Ever.
> 

The working group, by consensus.

> I do agree with you that it shouldn't occur that often, because I see
> no reason why one would need to use the affected characters.
> 

If you cannot think of a use case when this would come up, then how can 
you claim it may be important?

-Jonathan R.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sat, 15 Dec 2007 08:40:26 -0500

You are confusing 'small' with 'non-essential'. I agree small bugs can 
be essential to fix, and the one you pointed out (which is not an 
rfc3261 bug) is small and essential.

But this BNF problem that was raised is small but NON-ESSENTIAL, and its 
because I don't believe its essential, that I don't think it merits 
addressing.

-Jonathan R.