Subject: Re: PROPOSAL: i74: Encoding for non-ASCII headers

From: Mark Nottingham <[email-masked]>
Date: Wed, 2 Apr 2008 16:54:59 +1100

On 29/03/2008, at 3:54 PM, Roy T. Fielding wrote:

> I'd rather defer this issue until much later in the process,

I know people are getting fatigued by this one, and I'd like to at  
least get a shared understanding of the scope of this issue, and maybe  
carve off some parts to make it more manageable. Here are a few  
statements that I believe capture where we're at; if you disagree,  
please say so (hopefully without reopening the entire discussion);

1. We are considering allowing UTF-8 in content, specifically (a) in  
newly defined headers, and/or (b) in places where TEXT is now.

2. We intend to remove the "blanket" RFC2047 encoding associated with  
TEXT and (if kept) move it to the definitions of the individual rules,  
so that it's clear where such encoding may occur. Candidates for this  
include Reason-Phrase, filename-parm, warn-text, as well as the  
comments in field-content.

3. If RFC2047 encoding is used / referenced, we need to more carefully  
specify its use; e.g., regarding what encoding forms are allowable,  
line length limits, charsets used, folding.

4. From also deserves a look.

5. Either the definition of TEXT or CTL may need the C1 control  
characters added.

As Roy states, #1 should be approached conservatively. I think we can  
quickly get a decision on 2, 3, and 5. If we later decide to allow  
UTF-8, we can readjust the text (and TEXT) to suit, but if we're going  
to punt on this, I'd like to at least nail down the parts of it that  
we know we have to deal with.

--
Mark Nottingham     http://www.mnot.net/

---

From: Julian Reschke <[email-masked]>
Date: Thu, 03 Apr 2008 20:58:40 +0200

Mark Nottingham wrote:
> I know people are getting fatigued by this one, and I'd like to at least 
> get a shared understanding of the scope of this issue, and maybe carve 
> off some parts to make it more manageable. Here are a few statements 
> that I believe capture where we're at; if you disagree, please say so 
> (hopefully without reopening the entire discussion);
> 

Yes.

> include Reason-Phrase, filename-parm, warn-text, as well as the comments 
> in field-content.

Yes.

> specify its use; e.g., regarding what encoding forms are allowable, line 
> length limits, charsets used, folding.

Yes.

> 4. From also deserves a look.

Ok.

> 5. Either the definition of TEXT or CTL may need the C1 control 

TEXT already allows C1 controls (and always did) 
(<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#rule.TEXT>):

   TEXT           = %x20-7E | %x80-FF | LWS
                  ; any OCTET except CTLs, but including LWS

That being said, I'd like it to exclude C1 controls.

> to punt on this, I'd like to at least nail down the parts of it that we 
> know we have to deal with.

Agreed.

BR, Julian

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 3 Apr 2008 21:16:45 +0100

I agree with everything Julian said, except:

> That being said, I'd like it to exclude C1 controls.

If C1 controls _in the form of octets %x80-9F_ are excluded, and HTTP
agents begin to reject TEXT containing those octets, it will be harder
to transition to UTF-8 later.  (In case you'd forgotton, UTF-8 uses
those octet values for normal characters).

In other words, rejecting %0x80-9F pretty much commits the high octets
of TEXT to representing iso-8859-1.  Even if nobody uses it (like now
it seems), there may be agents which don't send it, but reject
%0x80-9F if it's recommended by an HTTP RFC.

Excluding C1 controls _encoded as UTF-8_ is quite reasonable.  But
then, there are lots of other controls one might wish to exclude too -
preferably by saying senders SHOULD NOT send them, but perhaps
receivers shouldn't reject them.

-- Jamie

---

From: Julian Reschke <[email-masked]>
Date: Thu, 03 Apr 2008 22:26:42 +0200

> those octet values for normal characters).

No I didn't. Good point.

> ...

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 08:00:42 +1100

Julian -- sorry, I mean to say that we need to consider excluding C1.

Jamie -- if we later decided to allow UTF-8, we'd of course have to  
figure out how it fit into the overall picture. This sub-issue is just  
about whether we should allow C1 in iso-8859-1, as the spec is  
currently written.

Cheers,

On 04/04/2008, at 7:26 AM, Julian Reschke wrote:

>>> TEXT already allows C1 controls (and always did) (<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p1-messaging-latest.html#rule.TEXT 
>>> >):
>>>

---

From: Jamie Lokier <[email-masked]>
Date: Thu, 3 Apr 2008 22:34:57 +0100

> about whether we should allow C1 in iso-8859-1, as the spec is  

I see your point, but I think the reality at the moment is it's just
permission to transport high valued octets, with %x20-7F being ASCII.
I don't think anyone seriously treats the high values as iso-8859-1
(except by accident).

I'm thinking the current spec, plus current practice (assumed, not
thoroughly investigated), has a nice loophole to move it to UTF-8
without breaking anything.

The key thing is "without breaking anything".  Even if there are a few
implementations which treat the upper octets as iso-8859-1, they will
continue working if we start sending UTF-8.

-- Jamie

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 09:58:30 +1100

Yes, that's one path we can take, but we need to make that decision.

Separate from that, we need to determine whether omitting the C1  
controls when using iso-8859-1 was an oversight, or purposeful;  
there's more than once character in that set.

Cheers,

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 3 Apr 2008 16:40:43 -0700

> Yes, that's one path we can take, but we need to make that decision.

Not really.  I'll reiterate again -- it makes absolutely no sense
whatsoever to be micromanaging the unused details of octet encoding
when the specification parts that define that very thing are already
on the hook to be removed/rewritten in accordance with the draft
partitioning.  This is not the low hanging fruit we are looking for.

The parsing algorithm will not say anything about C1 controls because
no known implementation of HTTP checks for C1 controls.  HTTP cares
about ASCII ":" and CRLF.  This means that the message parser will
carve the message into envelope, headers, and body before we can even
think about other encodings.  We are then left with four places in
which encodings can be defined: request/response line, defined header
fields, the grammar for extension header fields, and the message body.

Each of those can be dealt with in isolation, when we've reached the
point where we can make decisions based on what has been implemented.

It is important to understand that there are no generic field-content
parsers for HTTP, so specifying generic requirements that are not
backed up by implementations is a bad idea.  The specification can be
dramatically simplified (and made more future-compatible with UTF-8)
by removing the unused requirements that are not found in
implementations.  Adding more generic field encoding requirements,
which are neither needed nor backed by current practice, is a bad idea.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 11:04:06 +1100

On 04/04/2008, at 10:40 AM, Roy T. Fielding wrote:

> partitioning.

Roy, you often talk about things that will happen as part of the  
partitioning, but it's not clear what's involved. Do you have a  
roadmap of what you'd like to happen? If we're going to attempt  
substantial rewrites of different sections, we need to start that  
process soon.

Without some idea of that, it's going to be difficult to make any  
forward progress if we have to block any substantive issues on future  
rewrites that we may or may not do.

> no known implementation of HTTP checks for C1 controls.

That doesn't follow. If there are security or interoperability  
implications of C1 controls in text, it certainly deserves  
consideration. Also, while the message parsing machinery doesn't touch  
TEXT, there are other parts of implementations that do -- e.g.,  
command-line tools, Web forms for configuring servers and proxies,  
configuration files, and so forth. Just because it's payload doesn't  
mean that it doesn't have implementation impact.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 11:51:58 +1100

> newly defined headers, and/or (b) in places where TEXT is now.

This will remain in i74, now Character Encodings in TEXT.

> 2. We intend to remove the "blanket" RFC2047 encoding associated  
> with TEXT and (if kept) move it to the definitions of the individual  
> rules, so that it's clear where such encoding may occur. Candidates  
> for this include Reason-Phrase, filename-parm, warn-text, as well as  
> the comments in field-content.

new issue: <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/111>

> 3. If RFC2047 encoding is used / referenced, we need to more  
> carefully specify its use; e.g., regarding what encoding forms are  
> allowable, line length limits, charsets used, folding.

Expanding #63 (now: RFC2047 encoded words) to include other aspects  
(63 and 111 can probably be addressed at the same time).

> 4. From also deserves a look.

Looking at p2, 10.3, I see we've already updated this to reference  
RFC2822. Does anyone feel we need to have a separate issue for this?

> 5. Either the definition of TEXT or CTL may need the C1 control  

Also in i74.

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 3 Apr 2008 18:09:28 -0700

> substantial rewrites of different sections, we need to start that  

My roadmap is to make each part independent and complete.  It should  
only
affect the requirements that are vaguely specified as applying to HTTP
as a whole (most notably in part 1 where the message is parsed, where
connections are managed, and the spaghetti cross-references and  
redundant
requirements in caching).  Absolutely no protocol changes will occur as
a result -- only removal of overspecification.

> forward progress if we have to block any substantive issues on  
> future rewrites that we may or may not do.

This is only the second issue (out of 90+) that I have asked to be
deferred.  Are we making progress right now by driving for consensus
on an absolutely pointless and UNIMPLEMENTED issue?  Even if you
derive an answer, it is inherently bogus and can't be published
because we still must have two implementations of every requirement.
That's why I keep asking for demonstrated need -- if we can't  
demonstrate
that a new requirement is needed, then there is no reason to discuss
the text of a solution.  It is false progress and it prevents me from
allocating my own time more wisely.

> implications
> of C1 controls in text, it certainly deserves consideration.

There aren't any for HTTP.

> Also, while the message parsing machinery doesn't touch TEXT, there  
> are other parts of implementations that do -- e.g., command-line  
> tools, Web forms for configuring servers and proxies, configuration  
> files, and so forth. Just because it's payload doesn't mean that it  
> doesn't have implementation impact.

It does have implementation impact in the four fields that I described,
each of which are defined in a different location.  Two of those  
locations
(body and defined header fields) are specified already.  Reason  
Phrase is
solidly iso-8859-1 and already ignored.  That leaves extension-field,
which can say whatever we like without impacting standard status because
it is specifically defined for extensions.  This will be obvious after
the message parsing algorithm in Part 1 is properly specified and
verified against current practice, which is why I asked that the issue
be deferred.  I'll have time to work on it after April 13, when I get
back from ApacheCon in Amsterdam where I am giving two presentations
next week.

Meanwhile, if you really want to make progress on *this* issue, then the
way forward is to start using the wiki to collect experience reports
on what is and is not implemented in practice.  Further discussion is  
*not*
making progress.  In particular, ask the working group to help  
demonstrate
implementation practice instead of just offering opinions.  Encourage
folks to try using SetHeader in Apache, its equivalent in IIS, Squid,
and others.  Try doing the same with javascript or extensions in  
browsers.
We might find that raw UTF-8 is already possible for extension fields.
Or it may be that OCTETs are the only things that matter and no existing
character encoding truly applies.  In any case, it is easier to find
compliant implementations if we reduce the number of existing  
requirements
rather than add to them.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 4 Apr 2008 12:27:15 +1100

Roy,

I want to make progress -- whether on this issue or others doesn't  
matter much. If you have suggestions for doing so -- such as you've  
given below -- they're very welcome. In particular, if there are  
issues that you (or anyone else) think we'd profit from focusing on,  
I'd love to hear it; I've repeatedly asked for input on this, and  
haven't received much.

Using the wiki to collect experience is a good idea, if people are  
willing to participate. Anyone with a tools.ietf.org password* can  
create and modify pages on it; all that I ask is that you coordinate  
with me before modifying the front page.

WRT this issue -- as should be clear by recent traffic, I'm trying to  
carve off the parts of the issue that we can agree on now, so that we  
don't have to rehash the entire discussion in a few months. The parts  
that you contend aren't ready for discussion -- i.e., character  
encoding and C1 controls -- are in i71, which is the part we're  
deferring.

* See: http://www3.tools.ietf.org/newlogin

On 04/04/2008, at 12:09 PM, Roy T. Fielding wrote:

> on what is and is not implemented in practice.  Further discussion  
> is *not*

---

From: Henrik Nordstrom <[email-masked]>
Date: Sun, 06 Apr 2008 00:52:33 +0200

> about whether we should allow C1 in iso-8859-1, as the spec is  

My standpoint is that the aim should be to try to get away from
iso-8859-1 and try to get over to UTF-8 where applicable.

Any steps taken which restricts general syntax in a way that UTF-8 would
not fit is a step backwards on that, and in the wrong direction I think.
Any new things added need to fit within the general syntax definition,
which today is based on TEXT.

We could split that, making TEXT octets based allowing any octets except
for CTLs but including LWS (current definition) and another definition
for ISO-8859-1 and use this explicitly where we see that staying with
ISO-8859-1 is the only and the only possible way for the duration of
HTTP/1.x, but I don't see much point in that.

Regards
Henrik

---

From: "Brian Smith" <[email-masked]>
Date: Mon, 7 Apr 2008 08:51:11 -0700

> I'd love to hear it; I've repeatedly asked for input on this, and  

I am really interested in seeing how the issues with conditional
requests on content-negotiated resources will get resolved (i22, i37,
i38, i39, i58, i69, i71, i89, i101, i107, i109, i110). Content
negotiation is being used everywhere (Vary: Content-Encoding), but ETags
for content-negotiated resources are not processed uniformly in existing
implementations. In particular, ETag handling for mod_deflate varies
considerably even between some minor ("bug-fix") updates of Apache. As
more applications start using conditional requests (especially PUT),
these differences make interoperability difficult. In order to resolve
these issues, it looks like the WG will have to declare that many (most?
all?) deployed implementations are wrong. It is a bad idea to defer
these issues any longer because they are important but probably
contentious; resolving these issues would be a good test to see if a
HTTPbis will be able to move forward on a reasonable schedule. 

Regards,
Brian

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 8 Apr 2008 10:13:21 +1000

Thanks for the input.

Many (but not all) of those have some amount of dependency upon i109.  
I've been writing a straw-man proposal for it in the background, and  
should post it soon (please don't let that stop you if you have ideas).

I don't know that I agree we'll have to declae so many implementations  
as wrong, but let's see where we go.

Cheers,