Subject: non CHAR characters in headers

From: Adrien de Croy <[email-masked]>
Date: Fri, 12 Jun 2009 17:30:59 +1200

I'm seeing a response from a server where the Content-Type value is not 
ASCII

the bytes are 0xFF 0xD0 0xFE etc.

This is illegal correct?  I thought if unicode were being put into 
headers, it needs to be encoded according to one of several standards 
e.g. RFC 2047.  There's no "=?" in there anywhere.

What is a proxy supposed to do in such cases?  There's no information 
upon which to assume the proper format of the value.  I see 4 options.

1. pass bytes through as is and let the client deal with it.
2. strip the header
3. block  the response with an error
4. try and convert.

I'm currently using option 1.  Is this deemed problematic?

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 12 Jun 2009 15:17:19 +0900

> e.g. RFC 2047. There's no "=?" in there anywhere.

I general, I'm rather sympathetic to putting Unicode in headers where it 
makes sense. But for Content-Type, it doesn't make sense at all (unless 
it's part of a parameter to some exotic content type, but I think even 
most parameters for content types are ASCII-only by design, because they 
are simple predefined tokens or numbers).

Also, the right way to use Unicode in HTTP headers, if at all, is to use 
UTF-8. Neither 0xFF nor 0xFE can occur in UTF-8. So this is doubly suspect.

> 3. block the response with an error

What do you mean with 4? It looks like you're getting garbage, what do 
you want to convert that to?

> I'm currently using option 1. Is this deemed problematic?

I don't know what the spec says, but for headers that the proxy is not 
concerned about, it's probably the right thing to do. The client will 
have to deal with (which may just mean being robust against) such cases 
anyway. Where the proxy is concerned about the header, the situation may 
be a bit different (in this case e.g. if there is a Vary by Content-Type).

Regards,    Martin.

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:[email-masked]

---

From: Julian Reschke <[email-masked]>
Date: Fri, 12 Jun 2009 12:11:03 +0200

> I'm seeing a response from a server where the Content-Type value is not 

As in "Content-Type: abc/xyz", and "abc/xyz" containing non-ASCII 
characters?

> e.g. RFC 2047.  There's no "=?" in there anywhere.

This was sort-of allowed in RFC2616 for headers that used the TEXT ABNF 
rule, and Content-Type doesn't do that.

BTW, in HTTPbis, we have reduced the situations where this is allowed 
(as it wasn't used anyway); I think the only place where we currently 
still have it is for the Warn header in Part 6. (I guess that's a TODO...).

> ...

> ...

I don't think so; see:

"Historically, HTTP has allowed field-content with text in the 
ISO-8859-1 [ISO-8859-1] character encoding (allowing other character 
sets through use of [RFC2047] encoding). In practice, most HTTP header 
field-values use only a subset of the US-ASCII charset [USASCII]. Newly 
defined header fields SHOULD constrain their field-values to US-ASCII 
characters. Recipients SHOULD treat other (obs-text) octets in 
field-content as opaque data." -- 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-06.html#rfc.section.4.2.p.3>

BR, Julian