Subject: [Sip] SIPit21: BNF future-proofing problem?

From: Robert Sparks <[email-masked]>
Date: Mon, 19 Nov 2007 15:35:03 -0600

The BNF in 3261 says the following:

extension-header  =  header-name HCOLON header-value
header-value      =  *(TEXT-UTF8char / UTF8-CONT / LWS)

This is intended to be the catch-all field for all future extensions  
- older parsers working against this BNF shouldn't barf
when we introduce a new header field.

Now, we may have new fields in the future that look like:

NewHeader = new-header-name HCOLON quoted-string

And down inside quoted-string, we get:

       quoted-string  =  SWS DQUOTE *(qdtext / quoted-pair ) DQUOTE
       qdtext         =  LWS / %x21 / %x23-5B / %x5D-7E
                         / UTF8-NONASCII
       quoted-pair  =  "\" (%x00-09 / %x0B-0C
                        / %x0E-7F)

So, for instance, we could have inside a quoted string the 2 byte  
sequence \ NULL

This does not parse against header-value above...

Is this a problem? Some of the SIPit21 participants argued that it is.

The projects I've been involved in don't parse unknown headers and  
the stacks will just hand up an unparsed bucket of bits (the only rules
used are those necessary to identify the next header-field starting).

Would it be worth the effort to make the BNF reflect that rather than  
continuing with the incongruity that we currently specify?

RjS

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: "Christer Holmberg" <[email-masked]>
Date: Tue, 20 Nov 2007 08:18:12 +0100

Hi Robert,

I am not really sure what you are proposing, but if we would mandate
parsers not to parse unknown headers something is REALLY wrong with the
BNF... 

Regards,

Christer

> -----Original Message-----
> From: Robert Sparks [mailto:[email-masked]] 
> Sent: 19. marraskuuta 2007 23:35
> To: sip List
> Subject: [Sip] SIPit21: BNF future-proofing problem?
> 
> - older parsers working against this BNF shouldn't barf when 
> we introduce a new header field.
> 
> So, for instance, we could have inside a quoted string the 2 
> byte sequence \ NULL
> 
> The projects I've been involved in don't parse unknown 
> headers and the stacks will just hand up an unparsed bucket 
> of bits (the only rules used are those necessary to identify 
> the next header-field starting).
> 
> Would it be worth the effort to make the BNF reflect that 
> rather than continuing with the incongruity that we currently specify?
> 
> RjS
> 
> 
> This list is for NEW development of the core SIP Protocol Use 
> [email-masked] for questions on current sip

---

From: "Hisham Khartabil" <[email-masked]>
Date: Wed, 21 Nov 2007 15:53:58 +1100

This is a big change if we do adopt it. It will cause a lot of problem to
parsers that handle extenion headers today and is not backwards compatible.
Why isn't the extension header as is defined today not sufficient?

Hisham

---

From: "Hisham Khartabil" <[email-masked]>
Date: Wed, 21 Nov 2007 16:29:39 +1100

I think I did misread Robert's original email. I thought header values
having quoted strings are currently not allowed and Robert wanted to change
RFC2822. Now I realise that you can have header field values with quoted
string. Therefore I support the correction that Robert is proposing.

Hisham

---

From: "Christer Holmberg" <[email-masked]>
Date: Wed, 21 Nov 2007 08:08:07 +0100

Hi,

I am still a little unclear what Robert is proposing, but is there
really a need to be able to use characters between x00 and x20 in SIP
messages? Characters between x21 and x7F, and between x80 and xBF, are
ok since they are covered by TEXT-UTF8char and UTF8-CONT.

Regards,

Christer

________________________________

	From: Hisham Khartabil [mailto:[email-masked]] 
	Sent: 21. marraskuuta 2007 7:30
	To: Robert Sparks
	Cc: sip List
	Subject: Re: [Sip] SIPit21: BNF future-proofing problem?

	I think I did misread Robert's original email. I thought header
values having quoted strings are currently not allowed and Robert wanted
to change RFC2822. Now I realise that you can have header field values
with quoted string. Therefore I support the correction that Robert is
proposing. 
	 
	Hisham 

	On 21/11/2007, Hisham Khartabil <[email-masked]>
wrote: 

		This is a big change if we do adopt it. It will cause a
lot of problem to parsers that handle extenion headers today and is not
backwards compatible. Why isn't the extension header as is defined today
not sufficient? 
				 
		Hisham

						On 20/11/2007, Robert
Sparks <[email-masked] > wrote: 

			extension-header  =  header-name HCOLON
header-value
			header-value      =  *(TEXT-UTF8char / UTF8-CONT
/ LWS) 
			
			This is intended to be the catch-all field for
all future extensions
			- older parsers working against this BNF
shouldn't barf
			when we introduce a new header field.
			
			Now, we may have new fields in the future that
look like: 

			
			      quoted-string  =  SWS DQUOTE *(qdtext /
quoted-pair ) DQUOTE
			      qdtext         =  LWS / %x21 / %x23-5B /
%x5D-7E 
			                        / UTF8-NONASCII
			      quoted-pair  =  "\" (%x00-09 / %x0B-0C
			                       / %x0E-7F)
			
			So, for instance, we could have inside a quoted
string the 2 byte
			sequence \ NULL 
			
			This does not parse against header-value
above...
			
			Is this a problem? Some of the SIPit21
participants argued that it is.
			
			The projects I've been involved in don't parse
unknown headers and
			the stacks will just hand up an unparsed bucket
of bits (the only rules
			used are those necessary to identify the next
header-field starting).
			
			Would it be worth the effort to make the BNF
reflect that rather than 
			continuing with the incongruity that we
currently specify?
			
			RjS

			_______________________________________________
			Sip mailing list
https://www1.ietf.org/mailman/listinfo/sip
<https://www1.ietf.org/mailman/listinfo/sip> 
			This list is for NEW development of the core SIP
Protocol
			Use [email-masked] for
questions on current sip
			Use [email-masked] for new developments on the
application of sip

---

From: Robert Sparks <[email-masked]>
Date: Wed, 21 Nov 2007 08:20:04 -0600

So to restate the problem that was reported:

Right now, extension-header does not allow a new header that contains  
a quoted-string.

We have defined extensions (and are likely to define new ones) that  
use quoted-string (anything that uses name-addr for instance).

This inconsistency has caused interoperability problems in real  
implementations.

My original note was to suggest that we change extension-header to  
actually allow the headers we're going to define.

Christer's response could be read as a proposal to change quoted- 
string to achieve the same goal.

RjS

---

From: "Christer Holmberg" <[email-masked]>
Date: Wed, 21 Nov 2007 15:33:48 +0100

Hi, 

>Right now, extension-header does not allow a new header that 
>contains a quoted-string.
> 
>We have defined extensions (and are likely to define new 
>ones) that use quoted-string (anything that uses name-addr 
>for instance).
> 
>This inconsistency has caused interoperability problems in 
>real implementations.
> 
>My original note was to suggest that we change 
>extension-header to actually allow the headers we're going to define.
> 
>Christer's response could be read as a proposal to change 
>quoted- string to achieve the same goal.

Yes.

Unless someone can show a use-case where the x00-x20 characters would be
needed, for backward compability reasons I think it's better to restrict
quoted-string than to extend extension-header.

Regards,

Christer

> > values having quoted strings are currently not allowed and Robert 
> > wanted to change RFC2822. Now I realise that you can have 
> header field 
> > values with quoted string. Therefore I support the correction that 
> > Robert is proposing.
> > 	
> > 	Hisham
> > 	
> > 	
> > 		This is a big change if we do adopt it. It will 
> cause a lot of 
> > problem to parsers that handle extenion headers today and is not 
> > backwards compatible. Why isn't the extension header as is defined 
> > today not sufficient?
> > 				
> > 		Hisham
> > 		
> > 		
> > 						On 20/11/2007, Robert
> > 			This is intended to be the catch-all 
> field for all future 
> > extensions
> > 			the stacks will just hand up an 
> unparsed bucket of bits (the only 
> > rules
> > 			used are those necessary to identify 
> the next header-field 
> > starting).
> > 			
> > 			Would it be worth the effort to make 
> the BNF reflect that rather 
> > than

---

From: Adam Roach <[email-masked]>
Date: Wed, 21 Nov 2007 08:42:23 -0600

> quoted-string than to extend extension-header.

Certainly you mean 0x00-0x19, right? I'd like to be able to use spaces 
inside quoted strings.

/a

---

From: Adam Roach <[email-masked]>
Date: Wed, 21 Nov 2007 08:44:20 -0600

> Certainly you mean 0x00-0x19, right? I'd like to be able to use spaces 

Erm... make that 0x00 - 0x1F (oops).

/a

---

From: "Christer Holmberg" <[email-masked]>
Date: Wed, 21 Nov 2007 18:42:12 +0100

Hi,
 
>Certainly you mean 0x00-0x19, right? I'd like to be able to use spaces

The ABNF (see below) acutally doesn't allow 0x20 as such. BUT, you are also allowed to use LWS, which can contain spaces :)

TEXT-UTF8-TRIM  =  1*TEXT-UTF8char *(*LWS TEXT-UTF8char)
TEXT-UTF8char   =   %x21-7E / UTF8-NONASCII
UTF8-NONASCII   =  %xC0-DF 1UTF8-CONT
                                    /  %xE0-EF 2UTF8-CONT
                                    /  %xF0-F7 3UTF8-CONT
                                    /  %xF8-Fb 4UTF8-CONT
                                    /  %xFC-FD 5UTF8-CONT
UTF8-CONT       =  %x80-BF

Regards,

Christer

---

From: "Christer Holmberg" <[email-masked]>
Date: Wed, 21 Nov 2007 18:42:53 +0100

Hi,
 
Another alternative would be to define an escaping mechanism for quoted-strings in SIP messages, and require all characters not allowed by the header-value ABNF to be escaped.
 
Regards,
 
Christer 

From: Christer Holmberg [mailto:[email-masked]]
Sent: Wed 21/11/2007 15:33
To: Robert Sparks
Cc: sip List
Subject: RE: [Sip] SIPit21: BNF future-proofing problem?

Hi,

Yes.

Regards,

Christer

---

From: "T Satyanarayana-A12694" <[email-masked]>
Date: Thu, 22 Nov 2007 13:51:42 +0800

This would mean a large change to existing parsers and formatters. And
also, the "escaping" and "unescaping" overheads.

Changing the header-value BNF may be more acceptable?

Regards
Satya T

-----Original Message-----
From: Christer Holmberg [mailto:[email-masked]]
Sent: Wednesday, November 21, 2007 11:13 PM
To: Christer Holmberg; Robert Sparks
Cc: sip List
Subject: RE: [Sip] SIPit21: BNF future-proofing problem?

Hi,
 
Another alternative would be to define an escaping mechanism for
quoted-strings in SIP messages, and require all characters not allowed
by the header-value ABNF to be escaped.
 
Regards,
 
Christer 

Hi,

>Right now, extension-header does not allow a new header that contains a

>quoted-string.
>ones) that use quoted-string (anything that uses name-addr for 
>instance).
>

Yes.

Regards,

Christer

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol Use
[email-masked] for questions on current sip Use
[email-masked] for new developments on the application of sip

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol Use
[email-masked] for questions on current sip Use
[email-masked] for new developments on the application of sip

---

From: "Christer Holmberg" <[email-masked]>
Date: Thu, 22 Nov 2007 15:14:44 +0100

Hi, 

Existing parsers would not reject the message due to unsupported
characters, because you would only use characters already allowed.

Of course, parser would need to be able to unescapte those characters.
But, again, I wonder whether the characters we are talking about are
really needed in the first place...

>Changing the header-value BNF may be more acceptable?

Extending the definition is for sure going to have impacts on many
existing parsers.

Regards,

Christer

> Another alternative would be to define an escaping mechanism 
> for quoted-strings in SIP messages, and require all 
> characters not allowed by the header-value ABNF to be escaped.
>  
> Regards,
>  
> Christer 
> 
> contains a
> Unless someone can show a use-case where the x00-x20 
> characters would be needed, for backward compability reasons 
> I think it's better to restrict quoted-string than to extend 
> extension-header.
> 
> Regards,
> 
> Christer
> 
> 
> 
> 
> > > Hi,
> > >

---

From: [email-masked]
Date: Sun, 25 Nov 2007 11:53:06 -0500

From: Robert Sparks <[email-masked]>

The whole situation is rather icky.  I can see five problems:

1. header-value generates solo UTF8-CONT, the extension bytes of UTF-8
characters, which are the range x80-BF.  Why this is so is unclear --
the syntax cannot generate a solo UTF-8 initial byte which would
govern the extension byte, but the syntax also does not admit the
(single-byte) encodings of a lot of the characters in the ISO-8859-*
character sets, so the syntax does not permit embedding the one-byte
ISO-8859 encodings.  It appears to me that the inclusion of UTF8-CONT
in the production is unintended.

2. quoted-string admits (most of) x00-1F even though extension-header
does not.

3. Since quoted-string is used in many defined headers, we are
already in the position of having defined headers that cannot be
parsed as extension-header as a catch-all mechanism.

4. Given that there is no common character encoding within which all
of these productions can be uniformly interpreted, the only overall
description that can be given of the encoding of SIP headers is
"*OCTET".  And yet SIP headers are not intended to be a binary
protocol.

5. In quoted-string, a backslash is permitted to quote any ASCII
character, but not any Unicode character x80 or higher.  (Despite that
the backslash is not used to quote a solo UTF-8 initial byte.)  This
leads to the peculiar result that some letters used in (e.g.) French
can be preceeded by backslash in quoted-string, but others cannot.

Based on RFC 3261 section 7:

   SIP is a text-based protocol and uses the UTF-8 charset (RFC 2279 [7]).

my understanding is that the intention for SIP headers is that they
are sequences of Unicode characters encoded using UTF-8.  I see no
reason to abandon that principle and I've not heard of any instance
where anyone has done so deliberately.

To hold to that principle and clean up the above problems, the BNF
would need to be revised to be:

   extension-header  =  header-name HCOLON header-value
   header-value      =  *(TEXT-UTF8char / LWS)

	  quoted-string  =  SWS DQUOTE *(qdtext / quoted-pair ) DQUOTE
	  qdtext         =  LWS / %x21 / %x23-5B / %x5D-7E
			    / UTF8-NONASCII
	  quoted-pair  =  "\" (%x20-7E / UTF8-NONASCII)

or equivalently

	  quoted-pair  =  "\" (SP / TEXT-UTF8char)

Relative to other proposals, in a sense I'm proposing that
extension-header and quoted-string be contracted so that they
coincide, as I think that makes all the rules consistent and
conceptually coherent and does not exclude any current usage.  This
leaves me in the dark why interoperability problems have been seen.
Robert, can you show us some examples?

Dale