Subject: Semantic meaning of double quotation marks delimiting quoted-string

From: Geoffrey Sneddon <[email-masked]>
Date: Sun, 28 Oct 2007 18:17:21 +0000

Hi,

Earlier today I posted <http://gsnedders.com/http-entity-tags- 
confusion>, and having discussed it with various people, have  
concluded that it'd be best addressed by RFC 2616bis.

In short, the post raises the issue of the meaning of double quotes.  
If you parse them out, you end up with Etag: W/"a" and Etag: "W/a"  
equivalent. If you don't, you can end up with character sets such as  
"UTF-8" (i.e., including the quotation marks) in headers like Content- 
Type: text/plain;charset="UTF-8". What is the expected behaviour for  
quoted-string (or, if need be, for each and every specification that  
uses quoted-string)?

All the best

Geoffrey Sneddon

---

From: Julian Reschke <[email-masked]>
Date: Sun, 28 Oct 2007 19:32:39 +0100

Geoffrey Sneddon wrote:
> 
> Hi,
> 
> Earlier today I posted 
> <http://gsnedders.com/http-entity-tags-confusion>, and having discussed 
> it with various people, have concluded that it'd be best addressed by 
> RFC 2616bis.
> 
> In short, the post raises the issue of the meaning of double quotes. If 
> you parse them out, you end up with Etag: W/"a" and Etag: "W/a" 
> "UTF-8" (i.e., including the quotation marks) in headers like 
> Content-Type: text/plain;charset="UTF-8". What is the expected behaviour 
> for quoted-string (or, if need be, for each and every specification that 
> uses quoted-string)?
> ...

The simple answer is: the double quotes are part of the entity tag. So a 
response header such as

	ETag: x

would simply be invalid and should be ignored.

Best regards, Julian

(PS: I wouldn't be opposed to making that more obvious)

---

From: Geoffrey Sneddon <[email-masked]>
Date: Sun, 28 Oct 2007 19:27:15 +0000

On 28 Oct 2007, at 18:32, Julian Reschke wrote:

>> In short, the post raises the issue of the meaning of double  
>> quotes. If you parse them out, you end up with Etag: W/"a" and  
>> Etag: "W/a" equivalent. If you don't, you can end up with  
>> character sets such as "UTF-8" (i.e., including the quotation  
>> marks) in headers like Content-Type: text/plain;charset="UTF-8".  
>> What is the expected behaviour for quoted-string (or, if need be,  
>> for each and every specification that uses quoted-string)?
>> ...
>
> The simple answer is: the double quotes are part of the entity tag.  
> So a response header such as
>
> 	ETag: x
>

I am aware — but how is the receiving end meant to deal with them? Is  
it meant to keep the quotation marks around any quoted-string, even  
when that therefore results in non-exist things like a character set  
called "UTF-8" (with quotes)? Or does the behaviour need to be  
specific to each and every use of quoted-string need to have it  
defined separately?

- Geoffrey.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 28 Oct 2007 21:53:37 +0100

Geoffrey Sneddon wrote:
>> The simple answer is: the double quotes are part of the entity tag. So 
>> a response header such as
>>
>>     ETag: x
>>
> I am aware — but how is the receiving end meant to deal with them? Is it 
> meant to keep the quotation marks around any quoted-string, even when 
> that therefore results in non-exist things like a character set called 
> "UTF-8" (with quotes)? Or does the behaviour need to be specific to each 
> and every use of quoted-string need to have it defined separately?

That may be the case.

Do you have a specific header in mind? I just followed the grammar for 
"Accept-Charset", and as far as I can tell if you have double quotes in 
a charset value, it should be considered part of the charset name:

   Accept-Charset = "Accept-Charset" ":"
            1#( ( charset | "*" )[ ";" "q" "=" qvalue ] )

(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.2>)

  charset = token

(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.3.4>)

     token          = 1*<any CHAR except CTLs or separators>
     separators     = "(" | ")" | "<" | ">" | "@"
                    | "," | ";" | ":" | "\" | <">
                    | "/" | "[" | "]" | "?" | "="
                    | "{" | "}" | SP | HT

     CHAR           = <any US-ASCII character (octets 0 - 127)>

(<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.2.2>)

Now that really smells like something that is implemented differently in 
practice...

BR, Julian

---

From: Geoffrey Sneddon <[email-masked]>
Date: Sun, 28 Oct 2007 21:39:53 +0000

On 28 Oct 2007, at 20:53, Julian Reschke wrote:

> Geoffrey Sneddon wrote:
>>> The simple answer is: the double quotes are part of the entity  
>>> tag. So a response header such as
>>>
>>>     ETag: x
>>>
>> I am aware ? but how is the receiving end meant to deal with them?  
>> Is it meant to keep the quotation marks around any quoted-string,  
>> even when that therefore results in non-exist things like a  
>> character set called "UTF-8" (with quotes)? Or does the behaviour  
>> need to be specific to each and every use of quoted-string need to  
>> have it defined separately?
>
> That may be the case.

I was referring to my original example, i.e.,

> If you don't [parse the quotation marks out], you can end up with  

Which really is the question: what are we meant to do with the  
delimiting quotation marks in quoted-string?

If we take UTF-8 as a string, we can escape this as a quoted-string  
in several ways, including:

- "UTF-8"
- "\U\T\F\-\8"

Now, are we meant to unescape every quoted-string we come across  
(therefore including entity-tag), or only some? I think we can all  
agree that "\U\T\F\-\8" is not, in itself, a valid character set. If  
only some, which? As it stands now, it is not clear if you should  
ever unescape them.

- Geoffrey.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 28 Oct 2007 23:05:32 +0100

Geoffrey Sneddon wrote:
>> character sets such as "UTF-8" (i.e., including the quotation marks) 
>> in headers like Content-Type: text/plain;charset="UTF-8".
> 

Well, in general keep them.

> If we take UTF-8 as a string, we can escape this as a quoted-string in 
> several ways, including:
> 
> - "UTF-8"
> - "\U\T\F\-\8"
> 
> (therefore including entity-tag), or only some? I think we can all agree 
> that "\U\T\F\-\8" is not, in itself, a valid character set. If only 
> some, which? As it stands now, it is not clear if you should ever 
> unescape them.

I agree this is fuzzy, but I *disagree* that this fuzziness applies to ETag.

As far as I can tell, the quotes are an integral part of the ETag. It's 
invalid to leave them out.

Best regards, Julian

---

From: Geoffrey Sneddon <[email-masked]>
Date: Sun, 28 Oct 2007 22:10:02 +0000

On 28 Oct 2007, at 22:05, Julian Reschke wrote:

>> If we take UTF-8 as a string, we can escape this as a quoted- 
>> string in several ways, including:
>> - "UTF-8"
>> - "\U\T\F\-\8"
>> agree that "\U\T\F\-\8" is not, in itself, a valid character set.  
>> If only some, which? As it stands now, it is not clear if you  
>> should ever unescape them.
>
> As far as I can tell, the quotes are an integral part of the ETag.  
> It's invalid to leave them out.

My point is that such behaviour, like when to keep them and not,  
should be defined.

All the best,

Geoffrey Sneddon.

---

From: David Morris <[email-masked]>
Date: Sun, 28 Oct 2007 15:51:00 -0800 (PST)

My reading of the grammer you included is that the " character is a
separator and therefore not part of the charset name which by those
productions is a token.

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 29 Oct 2007 03:06:42 +0100

> delimiting quotation marks in quoted-string?

Whatever suits you best for processing the value I would say, as long as
the semantics of the grammar is preserved.

In case of entity-tag, the parser joining W/ and the dequoted etag value
as a single string is not a valid operation as it loses the distinction
between the weak marker and the etag opaqueue-tag value which are
distinct values in the grammar. If the parser removes quoting then the
weakness indication and the opaqueue-tag value needs to be stored
separately in order to be able to properly process the entity-tag. A
parser is not allowed to remove semantic structure of the parsed data,
only quoting/escaping of parsed elements, repeated LWS or other
redundant information having no semantic meaning.

> in several ways, including:

Yes, for use within a quoted-string element only. See quoted-string for
the meaning of this.. and yes, the two are equal.

And no, neither Accept-Charset or Content-Type is using quoted-string
for the charset.. but yes, the general parser structure of Content-Type
allows for quoted-string in parameter values.

> (therefore including entity-tag), or only some?

To compare two quoted-string elements you need to dequote them including
removing escapes, but in practice it doesn't matter much as people are
not usually escaping things within quoted-string unless needed (but
sometimes forget when needed, partly due to poor specifications, already
fixed).

This is quite notable in for example Digest authentication where proper
handling of quoted-string is required for the hashes to compute properly
as they are based on the value as such and not the quoted-string
representation. (i.e a login name with " or \ in it..)

It's in theory also needed for ETag processing, but it's less noticeable
as impacts on the protocol of getting this wrong is pretty minimal.

Regards
Henrik

---

From: "Frank Ellermann" <[email-masked]>
Date: Mon, 29 Oct 2007 08:37:46 +0100

Julian Reschke wrote:

> Now that really smells like something that is implemented 
> differently in practice...

...putting it mildly.  value := token / quoted-string  is a
MIME definition, and the MIME charset parameter takes a 
<value>, not only a <token>.  ISO_646.irv:1991 is an alias
with a colon, not permitted in a <token>.  Admittedly I'd
prefer US-ASCII for this character set. ;-)

 Frank

---

From: Geoffrey Sneddon <[email-masked]>
Date: Mon, 29 Oct 2007 18:37:04 +0000

> as impacts on the protocol of getting this wrong is pretty minimal.

Can we put something like the above quotation into the spec, so it is  
actually spelt out somewhere (as it currently isn't, at all)?

All the best,

Geoffrey Sneddon.