Subject: [Sip] bis-06 BNF issues

From: Rohan Mahy <[email-masked]>
Date: Sun, 3 Feb 2002 09:26:28 -0800 (Pacific Standard Time)

Hi,

I'm also in the middle of a detailed bis-06 review, and haven't finished,
but since I noticed that none of the other comments focussed on the BNF, I
redirected my efforts there.  I found several errors or questions on my
first pass, so here they are.

The last time I thoroughly read the BNF was bis-04, so I was a bit
surprised by some of the implications of the "explicit whitespace" BNF.
Please accept my apologies if this has been discussed already.

1) The thing that surprised me most is that "quoted-string" allows CRLF.
So the following are legal according to the BNF in bis-06:

To: "
 Bob
	" <[email-masked]>

Server: MyUA/1.0 ( "A comment
 that spans
 many lines" )

WWW-Authenticate: Digest realm = " my
  realm "  <etc....>

Was this the intent?  If so, it certainly seems to complicate parsing and
line-folding (you cannot collapse LWS into a single SP)  Lines 5812-5813
seem to indicate that quoted text should not contain CRLF.

2) Also, I think that SWS is either misdefined or used improperly instead
of LWS in many places.  Some examples:

To: "Bob"
<sip:[email-masked]>

is currently valid, because LAQUOT contains SWS, and SWS does not
necessarily contain SP or HT after a CRLF.  For that matter, SWS doesn't
have to contain anything, so this is also valid:

To: "Bob"<sip:[email-masked]>

I am assuming the later was intended, as I think this is a good thing.  It
just might not be obvious to readers/implementors.

3) Likewise the definition of "word" includes SP and HT.  this may be OK,
but I don't think it is such a good idea:

Call-ID: ab cd@ef gh

4) Ampersands (&) seem to disappeer in the PDF version.  perhaps this is
tex escaping issue?  for example, lines: 5771 (reserved), 5817 (headers),
5818 (pchar)

5) Reason-Phrase in 5822 seems to have -- instead of |  .  Also, shouldn't
you be able to include UTF8-NONASCII and UTF8-CONT in a reason phrase?

6) Shouldn't message-header include an extension-header ?

7) display-name in line 5838 has an extra trailing ")"

8) qvalue (line 5838) has an error, it should be:

	("0" [ "." 1*3DIGIT ] ) |
        ("1" [ "." 1*3("0") ] )

unless you want  ;q=0.

9) the definition of "challenge" in 5855 needs some LWS:

WWW-Authenticate: Digestrealm="foo.com",etc....

	= "Digest" LWS digest-challenge

or (preferably) we can resurrect auth-scheme from RFC2617 and use an IANA
registration for sip-auth-schemes or something.

10) Also, I know its late, but I'd like to propose that extension-method
names be limited to upalpha, or alphanum.  We really don't want/need method
names as in this example:

	*L.~ sip:[email-masked] SIP/2.0

11) no defintion of "psuedonym" (line 5872 -- warn-agent), plus warn-agent
should use existing definition of hostport (COLONs SWS is not apropos
here).

12) line 5818 (absoluteURI) should use ":" not COLON (don't need
whitespace in an http URL)

13) telephone-subscriber should match definition in RFC2806 which also
allows several other characters, for example:

	*  #  A  B  C  D   p  w  (  )

thanks,
-rohan

_______________________________________________
Sip mailing list  https://www1.ietf.org/mailman/listinfo/sip
This list is for NEW development of the core SIP Protocol
Use [email-masked] for questions on current sip
Use [email-masked] for new developments on the application of sip

---

From: Jonathan Rosenberg <[email-masked]>
Date: Sun, 03 Feb 2002 21:30:15 -0500

Thanks for the comments. Responses inline.

> 1) The thing that surprised me most is that "quoted-string" allows CRLF.

This has been allowed for as long as I remember. Its even allowed
according to rfc2543. Quoting that:

 quoted-string  =  ( <"> *(qdtext | quoted-pair ) <"> )
        qdtext         =  <any TEXT-UTF8 except <">>

and then, for TEXT-UTF8:

TEXT-UTF8  =  <any UTF-8 character encoding, except CTLs,
                      but including LWS>

which explicitly calls out that LWS is allowed. THus, this is not a
change.

> seem to indicate that quoted text should not contain CRLF.

No, these lines merely indicate that you cannot escape a CRLF by itself.
THat is:

From: "there is an \
esaped CRLF here" <sip:user@domain>

The intent, to my knowledge, has been to allow LWS in a quoted string.

> just might not be obvious to readers/implementors.

SWS is broken in bis-06, we got many comments to that regard. The new
definition is the following:

LWS = [*WSP CRLF] 1*WSP ; linear whitespace
SWS = [LWS] ; sep whitespace

WSP is from rfc2234, its SP / HT.

> but I don't think it is such a good idea:

It shouldnt matter, but this does represent a change from bis-04 and
bis-05. Thank goodness for bugzilla, I found the entry matching this
change. We can't use token, since it rules out usage of a v6 address
after the @ sign. However, I believe SP and HT shouldn't be in there,
and some comments in bugzilla even expressly said it should not. Thus, I
believe this is an error. I will remove.

> tex escaping issue?  for example, lines: 5771 (reserved), 5817

Yeah, I got all of those. It was a bug in our latex to text converter.
Henning fixed it. 

> 
> 5) Reason-Phrase in 5822 seems to have -- instead of |  .

Already fixed that one. We've convereted to full rfc2234 compliance, by
the way, so these are all going to be / instead of |.

>  Also,
> shouldn't

Yeah, I would think thats quite important for internationalization. Good
catch on that one. I will add them both.

> 6) Shouldn't message-header include an extension-header ?

Doh! This doesn't appear to be in rfc2543 or any later versions of bis.
I cannot believe we missed it. I will add it. I defined it as:

extension-header = header-name HCOLON header-value
header-name = token
header-value =  *(TEXT-UTF8CHAR / UTF8-CONT / LWS)

which is pretty much anything but the ascii controls; except that space
and tabs and CRLF are restricted to be valid linear white space. This
allows you to frame an extension header no matter what its contents.

> 7) display-name in line 5838 has an extra trailing ")"

Thanks, fixed.

>         ("0" [ "." 1*3DIGIT ] ) |

Interesting. This is copied right from http 1.1, which has the same bnf
exactly, and allows 0. I will assume this is OK as a result.....
> registration for sip-auth-schemes or something.

Yeah, I fixed all of that already. It allows for other schemes.

>         *L.~ sip:[email-masked] SIP/2.0

Another example of inheritances from HTTP. 1.1 also allows token there.
As with q-value, I will note this in bugzilla, but wont change for now.
I could imagine people doing things like:

SET_DATA

which is already beyond upalpha and alphanum. Although its aesthetically
ugly, I dont see anything wrong with tokens there from a processing
perspective.

> should use existing definition of hostport (COLONs SWS is not apropos

Huh? pseudonym is defined a few lines down.

As per SWS, there is nothing wrong with SWS, since it is still
completely parseable (warn-text would start with "). However, it is odd
given the usage of SP as the separator otherwise. I'll change to
hostport.

> whitespace in an http URL)

Yup, and also segment should use ; instead of SEMI
> allows several other characters, for example:

Yes. That one has been fixed. The spec now points to rfc2806 for the
BNF. Not sure thats the right thing either, since that bnf allows for
some stuff that is not allowed in the user part... For now, it
references 2806 and says you need to escape stuff that is not allowed.

-Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.            72 Eagle Rock Avenue
Chief Scientist                         First Floor
dynamicsoft                             East Hanover, NJ 07936
[email-masked]                 FAX: (973) 952-5050
http://www.jdrosen.net                  PH:  (973) 952-5000
http://www.dynamicsoft.com

---

From: Henning Schulzrinne <[email-masked]>
Date: Sun, 03 Feb 2002 22:28:26 -0500

> which explicitly calls out that LWS is allowed. THus, this is not a

Also, RFC 2822, the granddaddy of all these syntax patterns, allows this
explicitly, in BNF and wording:

Section 3.2.5:

qtext           =       NO-WS-CTL /     ; Non white space controls

                        %d33 /          ; The rest of the US-ASCII
                        %d35-91 /       ;  characters not including "\"
                        %d93-126        ;  or the quote character

qcontent        =       qtext / quoted-pair

quoted-string   =       [CFWS]
                        DQUOTE *([FWS] qcontent) [FWS] DQUOTE
                        [CFWS]

   A quoted-string is treated as a unit.  That is, quoted-string is
   identical to atom, semantically.  Since a quoted-string is allowed to
   contain FWS, folding is permitted.  Also note that since quoted-pair
   is allowed in a quoted-string, the quote and backslash characters may
   appear in a quoted-string so long as they appear as a quoted-pair

> believe this is an error. I will remove.

Yes, we had this discussion explicitly on the list. No spaces in
Call-ID.

> allows you to frame an extension header no matter what its contents.

Below is the 2822 definition for header-value. Obviously, we'd have to
replace ASCII by UTF-8.

utext           =       NO-WS-CTL /     ; Non white space controls
                        %d33-126 /      ; The rest of US-ASCII
                        obs-utext

unstructured    =       *([FWS] utext) [FWS]

> ugly, I dont see anything wrong with tokens there from a processing

Heck, we may need a provision for methods that correspond to the
unprintable @&%$#* bubbles in comics.

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 04 Feb 2002 00:16:08 -0500

Henning Schulzrinne wrote:

> unstructured    =       *([FWS] utext) [FWS]

Is there any real reason why you would allow controls? The definition
above that I have given seems simpler...

-Jonathan R.

---

From: Henning Schulzrinne <[email-masked]>
Date: Mon, 04 Feb 2002 09:08:16 -0500

> above that I have given seems simpler...

Don't know; my general answer is that it is probably wise to avoid
restricting the character set without a good reason. I'll ask the editor
of 2822 to see if there was a special reason.