Subject: Re: Unknown text/* subtypes [i20]

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Feb 2008 15:03:52 +0100

Hi.

With change <http://www3.tools.ietf.org/wg/httpbis/trac/changeset/209>, 
I have removed the character set defaulting, as proposed in 
<http://www3.tools.ietf.org/wg/httpbis/trac/ticket/20#comment:4>.

Section 3.1.1 (formerly 2.1.1) is gone, Section 3.3.1 (formerly 2.3.1) 
now says:

"3.3.1 Canonicalization and Text Defaults

Internet media types are registered with a canonical form. An 
entity-body transferred via HTTP messages MUST be represented in the 
appropriate canonical form prior to its transmission except for "text" 
types, as defined in the next paragraph.

When in canonical form, media subtypes of the "text" type use CRLF as 
the text line break. HTTP relaxes this requirement and allows the 
transport of text media with plain CR or LF alone representing a line 
break when it is done consistently for an entire entity-body. HTTP 
applications MUST accept CRLF, bare CR, and bare LF as being 
representative of a line break in text media received via HTTP. In 
addition, if the text is represented in a character set that does not 
use octets 13 and 10 for CR and LF respectively, as is the case for some 
multi-byte character sets, HTTP allows the use of whatever octet 
sequences are defined by that character set to represent the equivalent 
of CR and LF for line breaks. This flexibility regarding line breaks 
applies only to text media in the entity-body; a bare CR or LF MUST NOT 
be substituted for CRLF within any of the HTTP control structures (such 
as header fields and multipart boundaries).

If an entity-body is encoded with a content-coding, the underlying data 
MUST be in a form defined above prior to being encoded.

HTTP/1.1 recipients MUST respect the charset label provided by the 
sender; and those user agents that have a provision to "guess" a charset 
MUST use the charset from the content-type field if they support that 
charset, rather than the recipient's preference, when initially 
displaying a document." -- 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-latest.html#canonicalization.and.text.defaults>

BTW: it the subsection title still correct?

I also added a reminder to the Security Considerations to talk about the 
implications of character set sniffing (proposals welcome), and noted 
the change in Appendix C.2:

"Remove character set defaulting for text media types. (Section 3.3.1)" 
-- 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-latest.html#rfc.section.C.2>

Feedback appreciated, Julian

---

From: Geoffrey Sneddon <[email-masked]>
Date: Tue, 12 Feb 2008 16:26:43 +0000

On 12 Feb 2008, at 14:03, Julian Reschke wrote:

> sender; and those user agents that have a provision to "guess" a  
> charset MUST use the charset from the content-type field if they  
> support that charset, rather than the recipient's preference, when  
> initially displaying a document.

Does this mean using US-ASCII for text/xml without an explicit  
charset? I read it as meaning so (as by omitting it you have  
implicitly sent a US-ASCII charset label).

If you really want to require such a thing it is worth noting that it  
is extremely unlikely that any major HTTP implementation will actually  
abide by what 2616bis requires (therefore making the major  
implementations non-conforming). Do you really want to write a spec.  
with  couple of academic/experimental implementations, and nothing  
else? Surely it'd be more useful to specify HTTP in such a way that  
the major implementations can actually abide by the specification  
(while meeting market demands)?

--
Geoffrey Sneddon
<http://gsnedders.com/>

---

From: Albert Lunde <[email-masked]>
Date: Tue, 12 Feb 2008 11:39:49 -0500

> <http://www3.tools.ietf.org/wg/httpbis/trac/ticket/20#comment:4>.

It seems like, given history, you should explictly say somewhere 
that the default in previous versions of the spec has
been removed, and that there is no specified default,
(not even US-ASCII inherited from MIME).

-- 
    Albert Lunde  [email-masked]
                  [email-masked]  (new address for personal mail)
                  [email-masked] (old address)

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Feb 2008 17:42:04 +0100

Geoffrey Sneddon wrote:
> Does this mean using US-ASCII for text/xml without an explicit charset? 

It means whatever the other specifications say about that (e.g., RFC2046 
and RFC3023).

> I read it as meaning so (as by omitting it you have implicitly sent a 
> US-ASCII charset label).

Yes. So HTTP/1.1 doesn't specify a default anymore, but there are 
defaults specified somewhere else.

> If you really want to require such a thing it is worth noting that it is 
> extremely unlikely that any major HTTP implementation will actually 
> with  couple of academic/experimental implementations, and nothing else? 
> Surely it'd be more useful to specify HTTP in such a way that the major 
> implementations can actually abide by the specification (while meeting 
> market demands)?

Did you follow the long discussion leading to this change?

The reason we are making a change is that user agents indeed do *not* do 
what RFC2616 said. Thus we're removing that specific requirement. IMHO.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Feb 2008 18:09:48 +0100

> that the default in previous versions of the spec has

As mentioned earlier (:-), that's in 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-latest.html#changes.from.rfc.2616>

> (not even US-ASCII inherited from MIME).

Are you saying the defaults defined in RFC2046 and RFC3023 do not apply? 
Or should not apply?

My takeaway from the discussions was that we (those specifying HTTP) 
want to stop interfering with other specs, so, in particular, we are 
*not* overriding whatever these specs say.

So, if we don't like what RFC3023 says...:

"Note: There is an inconsistency between this specification and 
HTTP/1.1, which uses ISO-8859-1[ISO8859] as the default for a historical 
reason.  Since XML is a new format, a new default should be chosen for 
better I18N.  US-ASCII was chosen, since it is the intersection of UTF-8 
and ISO-8859-1 and since it is already used by MIME.)" -- 
<http://tools.ietf.org/html/rfc3023#section-3.1>

...we should work within the IETF process to change that.

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 12 Feb 2008 18:23:57 +0100

Julian Reschke wrote:
 
>| When in canonical form, media subtypes of the "text" type use
>| CRLF as the text line break. HTTP relaxes this requirement and
>| allows the transport of text media with plain CR or LF alone 
>| representing a line break when it is done consistently for an
>| entire entity-body. 

I'm not sure about this, it was found to be strange enough for a
dishonourable note in the future net-utf8 RFC.  I think what is 
really going on is something like this:

| HTTP does not depend on this canonical lineend in "text" types,
| and therefore does not require it in the content.

>| HTTP applications MUST accept CRLF, bare CR, and bare LF as being 
>| representative of a line break in text media received via HTTP.
>| In addition, if the text is represented in a character set that
>| does not use octets 13 and 10 for CR and LF respectively, as is
>| the case for some multi-byte character sets, HTTP allows the use
>| of whatever octet sequences are defined by that character set to 
>| represent the equivalent of CR and LF for line breaks.

I think that's beside the point.  AFAIK XML permits U+0085 NEL, and
text/xml exists (but maybe I confused XML 1.1 with XML 1.0 here).
If the charset uses octets 0D and 0A for U+000D and U+000A does not
necessarily affect octet 85 used as U+0085 in some legacy charsets.

HTTP does not really "allow" whatever represents a line break, it
simply does not "care" (within bodies or chunks).  How applications
interpret content is their business.  As far as HTTP is concerned
applications cannot trust that text/* comes with a canonical CRLF.

What really matters for HTTP is the header (and anything else not
belonging to the content).  And *there* CRLF is of course REQUIRED.

> HTTP/1.1 recipients MUST respect the charset label provided by
> the sender

Please justify this MUST strictly following RFC 2119, or replace
it by a SHOULD.  Many HTTP servers (even including IANA and W3C)
get some content types and their charsets where applicable wrong.  

> those user agents that have a provision to "guess" a charset 
> MUST use the charset from the content-type field

Please justify also this MUST using RFC 2119 terms.  Let's better
face it as it is, many HTTP servers are liars in practice, and do
not deserve too much respect.

 Frank

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Feb 2008 18:29:44 +0100

Frank,

you are raising lots of good questions, but in *this* discussion,
let's please focus on the removal of the text charset defaulting.

I totally agree that the line end stuff probably *also* needs work, but 
if we try to do all at once, nothing will ever get done.

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 12 Feb 2008 09:37:23 -0800

On Feb 12, 2008, at 8:42 AM, Julian Reschke wrote:
>>> the sender; and those user agents that have a provision to  
>>> "guess" a charset MUST use the charset from the content-type  
>>> field if they support that charset, rather than the recipient's  
>>> preference, when initially displaying a document.
>> charset?
> It means whatever the other specifications say about that (e.g.,  
> RFC2046 and RFC3023).

And that's why there is no consensus for this change.  -1

>> I read it as meaning so (as by omitting it you have implicitly  
>> sent a US-ASCII charset label).
>

Which have been wrong for 20 years, even for email.  The default was  
changed for HTTP
because that is how all HTTP implementations worked at that time.  A  
new default may
be required, but it can't make compliant HTTP/1.1 senders non- 
compliant.  It has
to match what HTTP/1.1 interoperates with today.

>> If you really want to require such a thing it is worth noting that  
>> it is extremely unlikely that any major HTTP implementation will  
>> actually abide by what 2616bis requires (therefore making the  
>> major implementations non-conforming). Do you really want to write  
>> a spec. with  couple of academic/experimental implementations, and  
>> nothing else? Surely it'd be more useful to specify HTTP in such a  
>> way that the major implementations can actually abide by the  
>> specification (while meeting market demands)?
>
> The reason we are making a change is that user agents indeed do  
> *not* do what RFC2616 said. Thus we're removing that specific  
> requirement. IMHO.

First, that's not true.  Four popular browser implementations do not  
obey
the standard.  They still don't obey the change you made.  They do  
not amount
to the measure of all user agents.

I was waiting for specific text to be suggested, but we can just fix it
in the draft.

....Roy

---

From: Geoffrey Sneddon <[email-masked]>
Date: Tue, 12 Feb 2008 17:39:51 +0000

On 12 Feb 2008, at 16:42, Julian Reschke wrote:

>> actually abide by what 2616bis requires (therefore making the major  
>> implementations non-conforming). Do you really want to write a  
>> spec. with  couple of academic/experimental implementations, and  

Yes, I did.

> *not* do what RFC2616 said. Thus we're removing that specific  

My point is that the issue with a MUST level requirement is that  
because of what other specifications require (in the text/xml case,  
RFC3023) it is still impossible to satisfy HTTP's requirements,  
therefore user agents still do *not* (and will not) do what HTTP says.  
The four most popular browsers aren't going to change their behaviour  
regarding this. The only sensible thing to do, for the sake of  
interoperability, is to match what they do, which we still don't do.

--
Geoffrey Sneddon
<http://gsnedders.com/>

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Feb 2008 18:49:32 +0100

> And that's why there is no consensus for this change.  -1

I understood we had consensus as our WG chair had directed us to make 
the change in the spec. Also, in 
<http://lists.w3.org/Archives/Public/ietf-http-wg/2008JanMar/0105.html> 
you seemed to say that you're ok with the change as long as we describe 
the vulnerability.

> because that is how all HTTP implementations worked at that time.  A new 
> default may
> be required, but it can't make compliant HTTP/1.1 senders 
> non-compliant.  It has

I have no problem with that. My understanding was that the 
implementations actually do *not* implement this, but maybe I was 
focused too much to text/xml.

>>> is extremely unlikely that any major HTTP implementation will 
>> The reason we are making a change is that user agents indeed do *not* 
>> do what RFC2616 said. Thus we're removing that specific requirement. 
>> IMHO.
> 
> First, that's not true.  Four popular browser implementations do not obey
> the standard.  They still don't obey the change you made.  They do not 

I'm aware of that, but at least this change would mean that they are 
violating one specification less than before.

> amount

Seems more like "back to the drawing board". My understanding was we 
*did* agree on removing the default -- see 
<http://www3.tools.ietf.org/wg/httpbis/trac/ticket/20#comment:4> -- but 
apparently there has been some misunderstanding related to that.

BR, Julian

---

From: Robert Sayre <[email-masked]>
Date: Tue, 12 Feb 2008 13:01:14 -0500

On Feb 12, 2008, at 12:49 PM, Julian Reschke wrote:

>> not

I agree that browsers do not amount to the measure of all user agents.  
However, their HTTP behavior will appear in many applications, not  
just browsers. MSHTML, WebKit, and Gecko have been embedded in many  
types of apps. Some browsers are relying on OS networking libraries as  
well--Safari relies on CFNetwork and some Linux browsers use libcurl.

- Rob

---

From: Julian Reschke <[email-masked]>
Date: Tue, 12 Feb 2008 19:01:44 +0100

Geoffrey Sneddon wrote:
> My point is that the issue with a MUST level requirement is that because 
> of what other specifications require (in the text/xml case, RFC3023) it 
> is still impossible to satisfy HTTP's requirements, therefore user 
> agents still do *not* (and will not) do what HTTP says. The four most 

I think RFC3023 is making the requirement *because* of the default in 
RFC2616. So it was RFC2616 overruling RFC2046, in turn being overruled 
by RFC3023.

> popular browsers aren't going to change their behaviour regarding this. 
> The only sensible thing to do, for the sake of interoperability, is to 
> match what they do, which we still don't do.

As Roy just stated, there are many other clients we need to consider in 
addition to the "four most popular browsers". What may seem to be the 
best behavior for them may be unacceptable to other user agents (and 
this includes many things that are not browsers at all).

What we need to discuss now is whether we want HTTP/1.1 to continue 
overruling RFC2046, whether we want to start overruling RFC3023 as well, 
and so on.

My preference clearly is to get out of that business, and let the MIME 
specs rule, in particular as those aforementioned browsers did not do 
what HTTP/1.1 says anyway.

So is there anything we *can* specify that matches reality?

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 12 Feb 2008 19:21:00 +0100

> let's please focus on the removal of the text charset defaulting.

Okay.  Admittedly I'm not following all "procedural" details like
the issue tracker, trusting that Mark, Roy, Lisa, you, and others
know what they want and how to achieve it.  I took your proposal
as final word on text/* and line breaks.

If that's not the case, yes, I clearly support to get rid of Latin,
and to invent *no* new rules, let alone anything incompatible with
MIME or text/xml, in this round (= 2616bis).

> I totally agree that the line end stuff probably *also* needs
> work, but if we try to do all at once
[...]

Maybe note this as issue if that's not yet the case.

 Frank

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 12 Feb 2008 13:12:44 -0800

On Feb 12, 2008, at 10:01 AM, Robert Sayre wrote:
>> I'm aware of that, but at least this change would mean that they  
>> are violating one specification less than before.
>
> I agree that browsers do not amount to the measure of all user  
> agents. However, their HTTP behavior will appear in many  
> applications, not just browsers. MSHTML, WebKit, and Gecko have  
> been embedded in many types of apps. Some browsers are relying on  
> OS networking libraries as well--Safari relies on CFNetwork and  
> some Linux browsers use libcurl.

Of course, but they aren't all user agents.  In order to make a  
normative
change to how messages are interpreted (i.e., the entire point of  
this spec)
without changing the HTTP version number, all implementations that were
compliant before must remain compliant after the change.

The charset issue came about originally because of bugs in one important
browser, Mosaic, that was unable to parse parameters on the Content- 
Type.
This was not such a big deal at the time because the entire Web required
iso-8859-1 and Mosaic only supported that charset anyway.  So, the  
solution
imposed (by interoperability with the real world) was that HTTP/1.x  
would
default the charset to iso-8859-1 instead of US-ASCII.  That is one of
the many reasons why HTTP is not MIME-compliant and never has been.
HTTP IS NOT EMAIL.  We have different requirements, different  
performance
limitations, and different implementors.

Here we are, 14 years later, and we have a new set of interoperability
requirements.  Browsers fixed their original charset limitations, a new
somewhat-universal charset is finally supported by somewhat-universal
font availability, and the Web consists of dozens of different charsets,
most of which are left unlabeled because there is no commonly accepted
way of indicating charsets in filename metadata (and no real need to
anyway, since user agents will either sniff the content anyway or just
assume everything is in the fixed local charset known by the tool).

Servers, OTOH, send text/* content with the assumption that it will be
treated as iso-8859-1 (or at least some safe superset of US-ASCII).
Servers don't sniff content because they can't -- it is impossible to
look at every byte of a page while handling 7,000 reqs/sec, let alone
the 20,000 reqs/sec that a decently tuned server can handle.  In  
addition,
some servers (particularly when serving dynamic content) will add a
charset parameter to unlabeled text/html content based upon how they  
have
been configured to scan for cross-site scripting.  They do so  
specifically
because of known bugs in browsers that sniff the content for bizarre
charsets that bypass the resource's security assumptions and
cause the browser's user to fall victim to stupid XSS attacks.

None of these implementations assume that a missing charset means
US-ASCII.  We cannot "pass the buck" to MIME because we are still
not MIME-compliant and never will be (see Content-Encoding).

So, what is interoperable now?

The answer is that iso-8859-1 is still the most interoperable default
*with* the addition of safe sniffing only when the charset is left
unlabeled or when charset="iso-8859-1".  By safe sniffing, I mean
specifically excluding any charset-switching in mid-content
for which the text media type's delimiter set (e.g., <"':> in HTML)
would be mapped to different octets than they are in US-ASCII.
In other words, it is safe to sniff for charsets in the first ten
or so characters, and also to switch to other US-ASCII supersets
after reading something like the <meta http-equiv="content-type" ...>,
but it is definitely unsafe to continue sniffing for charset changes
after that point unless they are restricted to US-ASCII supersets.

That allows HTTP/1.1 compliant serving today to remain compliant
after the change, and addresses all of the interoperability issues
in regard to mislabeled content without ignoring the fact that the
main reason they are mislabeled today is to work around existing
bugs.  For all other cases, the charset can and should be labeled
correctly.

....Roy

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 12 Feb 2008 23:19:16 +0100

> treated as iso-8859-1 (or at least some safe superset of US-ASCII).

That could be US-ASCII itself, windows-1252, UTF-8, and a bunch of
similar windows-xxxx, iso-8859-x, or other "unknown-ascii" supersets
(not counting UTF-1, UTF-7, or weirder charsets for obvious reasons).

> not MIME-compliant and never will be (see Content-Encoding).

All sound ASCII supersets have one thing in common, the 128 US-ASCII
octets, meaning U+0000 up to U+007F.     

> iso-8859-1 is still the most interoperable default *with* the
> addition of safe sniffing only when the charset is left unlabeled
> or when charset="iso-8859-1".

Any of these US-ASCII supersets could do as default.  The problems
of an explicit iso-8859-1 where that is not true cannot get worse,
and unfortunately also not better, with another default.

> after reading something like the <meta http-equiv="content-type"

An US-ASCII, windows-1252, or UTF-8 default would not change that.
And US-ASCII is the best approximation of "unknown-ascii" we have
at the moment.

If you decide that it's not good enough the "charset list" already
discussed to register "unknown-ascii" in addition to the existing
"unknown-8bit" some months ago.  And we could make sure that this
default pseudo-charset by definition won't cover UTF-1, UTF-7, or
similar abominations.  

But I think an US-ASCII default does precisely what you want, and
I fail to see how this could break existing HTTP implementations.

 Frank

---

From: Geoffrey Sneddon <[email-masked]>
Date: Tue, 12 Feb 2008 23:23:21 +0000

> after that point unless they are restricted to US-ASCII supersets.

ISO-8859-1 isn't actually the most interoperable default: a huge  
number of documents (and not just HTML documents, but also a large  
number of feeds) rely on ISO-8859-1 being treated as Windows-1252.  
What is probably the best default is windows-1252 with sniffing (the  
exact details of which are inevitably reliant on the exact format  
being used, as what is used in HTML obviously isn't suitable for XML,  
for example). I don't think it's worthwhile attempting to define what  
type of sniffing (e.g., your "safe" sniffing) can be used, as it is  
very much context dependant (and if there's one thing we've learnt  
from this, let it be that context is very important), and in some  
cases it may be ideal to throw out what you already have. However, in  
defence of "safe" sniffing, HTML5 requires a partial US-ASCII superset  
(to sniff it from meta), and XML 1.0 implicitly requires a superset of  
the encoding being used in Appendix F (when there is no BOM).

--
Geoffrey Sneddon
<http://gsnedders.com/>

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 12 Feb 2008 20:59:22 -0800

On 12/02/2008, at 9:37 AM, Roy T. Fielding wrote:

> I was waiting for specific text to be suggested, but we can just fix  
> it in the draft.

Roy, if you disagree with consensus on this issue, please suggest  
specific text to replace Julian's work.

In the future, when you don't agree with emerging consensus, I'd  
appreciate it if you tell us as soon as is practical.

Regards,

--
Mark Nottingham     http://www.mnot.net/

---

From: Robert Sayre <[email-masked]>
Date: Wed, 13 Feb 2008 17:01:04 -0500

On Feb 12, 2008, at 4:12 PM, Roy T. Fielding wrote:
>
> the Web consists of dozens of different charsets,

Fully agree.

> treated as iso-8859-1 (or at least some safe superset of US-ASCII).

Somewhat disagree. I think many servers assume that UAs will sniff,  
and deal with the issue for them.

> cause the browser's user to fall victim to stupid XSS attacks.

I know some cases of this attack, but I would appreciate more detailed  
references on these if you have them.

> bugs.  For all other cases, the charset can and should be labeled

I agree with your conclusion, but I'm fuzzy on the spec text it would  
lead to. Have specific wording in mind?

- Rob

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 13 Feb 2008 19:39:57 -0800

> specific text to replace Julian's work.

It isn't consensus until the people who have to change their
implementations agree to do so.  The change was applied in a way
that I did not anticipate, which made it a new requirement on
previously conforming implementations rather than a relaxation
of the existing requirements.  The issue did not require that much.

    http://www3.tools.ietf.org/wg/httpbis/trac/ticket/20

Here is the change that Julian made according to the issue:

    http://www3.tools.ietf.org/wg/httpbis/trac/changeset/209

[2.1.1 is deleted; the last para of 2.3.1 is replaced with

    HTTP/1.1 recipients MUST respect the charset label provided by the
    sender; and those user agents that have a provision to "guess" a  
charset
    MUST use the charset from the content-type field if they support  
that
    charset, rather than the recipient's preference, when initially  
displaying
    a document.
]

Here is what it said in p3 before that change:

2.1.1.  Missing Charset

    Some HTTP/1.0 software has interpreted a Content-Type header without
    charset parameter incorrectly to mean "recipient should guess."
    Senders wishing to defeat this behavior MAY include a charset
    parameter even when the charset is ISO-8859-1 ([ISO-8859-1]) and
    SHOULD do so when it is known that it will not confuse the  
recipient.

    Unfortunately, some older HTTP/1.0 clients did not deal properly  
with
    an explicit charset parameter.  HTTP/1.1 recipients MUST respect the
    charset label provided by the sender; and those user agents that  
have
    a provision to "guess" a charset MUST use the charset from the
    content-type field if they support that charset, rather than the
    recipient's preference, when initially displaying a document.  See
    Section 2.3.1.
...

2.3.1.  Canonicalization and Text Defaults

    Internet media types are registered with a canonical form.  An
    entity-body transferred via HTTP messages MUST be represented in the
    appropriate canonical form prior to its transmission except for
    "text" types, as defined in the next paragraph.

    When in canonical form, media subtypes of the "text" type use  
CRLF as
    the text line break.  HTTP relaxes this requirement and allows the
    transport of text media with plain CR or LF alone representing a  
line
    break when it is done consistently for an entire entity-body.  HTTP
    applications MUST accept CRLF, bare CR, and bare LF as being
    representative of a line break in text media received via HTTP.  In
    addition, if the text is represented in a character set that does  
not
    use octets 13 and 10 for CR and LF respectively, as is the case for
    some multi-byte character sets, HTTP allows the use of whatever  
octet
    sequences are defined by that character set to represent the
    equivalent of CR and LF for line breaks.  This flexibility regarding
    line breaks applies only to text media in the entity-body; a bare CR
    or LF MUST NOT be substituted for CRLF within any of the HTTP  
control
    structures (such as header fields and multipart boundaries).

    If an entity-body is encoded with a content-coding, the underlying
    data MUST be in a form defined above prior to being encoded.

    The "charset" parameter is used with some media types to define the
    character set (Section 2.1) of the data.  When no explicit charset
    parameter is provided by the sender, media subtypes of the "text"
    type are defined to have a default charset value of "ISO-8859-1"  
when
    received via HTTP.  Data in character sets other than  
"ISO-8859-1" or
    its subsets MUST be labeled with an appropriate charset value.  See
    Section 2.1.1 for compatibility problems.

================

And here is what I suggest for a rewrite, merging both of the above
sections under Media Types and inverting the "fantasy island"
requirements of the original text to what is permitted in HTTP
beyond the registration defaults of MIME.

2.3.1.  Canonicalization and Text Media Types

    Internet media types are registered with a canonical form and
    defaults for the optional parameter values.  An ideal HTTP
    entity-body would contain data formatted strictly according to that
    canonical form.  However, HTTP does not require the sender to verify
    that an entity-body is in canonical form prior to transfer.   
Instead,
    an HTTP recipient MUST be prepared to accept and properly interpret
    several variances in the format of textual types, as described  
below,
    and treat other variances as errors.

    The "charset" parameter (Section 2.1) is used with some media types
    to indicate the character encoding of the data.  When a media  
type is
    registered with a default charset value of "US-ASCII", it MAY be  
used
    to label data transmitted via HTTP in the "iso-8859-1" charset (a
    superset of US-ASCII) without including an explicit charset  
parameter
    on the media type.  In addition, when a media type registered with a
    default charset value of "US-ASCII" is received via HTTP without a
    charset parameter or with a charset value of "iso-8859-1", the
    recipient MAY inspect the data for indications of a different
    character encoding and interpret the data accordingly if the  
encoding
    is a superset of US-ASCII or if the encoding can be determined  
within
    the first 16 octets of data and interpreted consistently thereafter.

       Note: The first variance is due to a significant portion of early
       HTTP user agents not parsing media type parameters and instead
       relying on a then-common default encoding of iso-8859-1.  As a
       result, early server implementations avoided the use of charset
       parameters and user agents evolved to "sniff" for new character
       encodings as the Web expanded beyond iso-8859-1 content.  The
       second variance is due to a certain popular user agent that
       employed an unsafe encoding detection and switching algorithm
       within documents that might contain user-provided data (see
       Section security.sniffing), the most common workaround for which
       is to supply a specific charset parameter even when the actual
       character encoding is unknown.

    When in canonical form, media subtypes of the "text" type use  
CRLF as
    the text line break.  However, it is also commonplace for such types
    to be transmitted in HTTP with CR or LF alone indicating a line
    break and occasional for such types to be transmitted with a
    character encoding that requires some other set of octet sequence(s)
    to indicate a line break.  HTTP recipients MUST accept and properly
    interpret CRLF, bare CR, and bare LF as indicating a line break when
    encountered within an entity-body received via HTTP that is labeled
    as a text type and provided in a character encoding that allows CRLF
    to indicate a line break.

       Note: Line breaks are specified in MIME with the expectation that
       they are enforced during email message composition, when it is
       scalable to ensure that every octet is placed in canonical form,
       and with the anticipation that a message may be transmitted or
       processed using line-oriented protocols.  HTTP message  
generation,
       in contrast, is usually performed at high speed, encloses data
       that cannot be modified without also altering its metadata, and
       is processed using length-delimited protocols.

=====================

> appreciate it if you tell us as soon as is practical.

This is as soon as practical.  The last discussion of it took place
the day before I got hit by the bronchitis fever, and I did disagree
with the proposal at that time.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 14 Feb 2008 00:06:45 -0800

> of the existing requirements.  The issue did not require that much.

Julian applied the recorded intent faithfully (there was little room  
for interpretation; see <http://www.w3.org/mid/[email-masked] 
 >), and no-one -- including yourself, unless I'm missing e-mail --  
objected to it beforehand.

Your point about consensus is well-taken; it is dependent on many  
things, and can of course change over time. However, we should avoid  
revisiting topics when possible.

> with the proposal at that time.

Sorry to hear that, and I'm glad you're feeling better. I was  

> I was waiting for specific text to be suggested, but we can just fix  

which was difficult to interpret, given that specific changes had been  
on the table for some time.

Thanks for providing a proposal. I'd suggest that people review it  
carefully, because it's a much more substantial change, from a lines- 
of-text standpoint.

Regards,

P.S. Would people find it useful if we listed the current design  
issues under discussion somewhere (e.g., on the wiki, and/or in a  
weekly summary e-mail)?

---

From: Yutaka OIWA <[email-masked]>
Date: Thu, 14 Feb 2008 19:35:18 +0900

Sorry that I was off from this list for a month that was very busy...

> beyond the registration defaults of MIME.

+1: I prefer this alternative for minimizing new security concerns.

Julian's text (at least implicitly) allows clients to guess any
character encoding when no charset value is provided, which is
unacceptable for HTTP/1.1 because many existing applications
immediately become vulnerable due to this change (and existence of
UTF-7).  I strongly suggest that this must be treated as an
incompatible normative change, and it is hard to realize even if the
version number is raised to HTTP/2.0.

Roy's suggestion (and my previous one) is moderate in this scope,
acceptable within version 1.1 in reality, and almost consistent to
on-going HTML5 updates which will be actually implemented in real user
agents.

Possible my comment to Roy's proposal is

  - Auto-detection under explicit "iso-8859-1" charset label is dirty.
    I suggest either drop it or state it as "if required for backward
    compatibility".  I do not want to give server owners a permission
    for blindly adding ISO-8859-1 charset label without consulting the
    real contents any more in future.  I think it will not be
    supported under HTML5 rules.

  - We may need some clarification on the definition of whether "the
    encoding is a superset of US-ASCII".  There are several well-used
    character encodings which are marginal on this property.

-- 
Yutaka OIWA, Ph.D.                                       Research Scientist
                            Research Center for Information Security (RCIS)
    National Institute of Advanced Industrial Science and Technology (AIST)
                      Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[995DD3E1] fp[3C21 17D0 D953 77D3 02D7 4FEC 4754 40C1 995D D3E1]

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Feb 2008 14:48:20 +0100

> of the existing requirements.  The issue did not require that much.

And I don't think we realized that we may be *adding* requirements; the 
goals were (IMHO) to reduce inconsistency of specs and to allow what UAs 
indeed do today. It seems the conclusion is that in this case these 
goals are contradictory.

>    http://www3.tools.ietf.org/wg/httpbis/trac/changeset/209

...which was what the issue resolution asked for; so the mistake 
happened earlier.

In the meantime I have backed out the change 
(<http://www3.tools.ietf.org/wg/httpbis/trac/changeset/211>), and I'd 
propose to re-open the issue.

>    that an entity-body is in canonical form prior to transfer.  Instead,
>    several variances in the format of textual types, as described below,

Good introduction.

>    to indicate the character encoding of the data.  When a media type is
>    registered with a default charset value of "US-ASCII", it MAY be used
>    superset of US-ASCII) without including an explicit charset parameter
>    character encoding and interpret the data accordingly if the encoding
>    is a superset of US-ASCII or if the encoding can be determined within

Q: so if a text type defines a different default, such as "UTF-8", we 
disallow defaulting to ISO-8859-1 and sniffing?

>       character encoding is unknown.

Q: so specifying ISO-8859-1 was a countermeasure to broken content 
sniffing? Is that version of that UA still common enough to have this 
exception?

>       processed using line-oriented protocols.  HTTP message generation,

Ok.

Sounds good to me, except for:

- Q: do we need to state that we overrule RFC3023 (WRT default charset 
for text/xml?)

- we still need the security consideration WRT sniffing.

BR, Julian

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Feb 2008 14:58:21 +0100

> character encoding when no charset value is provided, which is

Just for the record: it wasn't "my" text, but the text suggested by our 
working group chair, thinking we actually *did* reach consensus (IMHO 
rightfully, looking at the mailing list archive).

Except for strictly editorial issues, the editors do not have the 
mandate to change specification text without the WG telling them to do so.

> version number is raised to HTTP/2.0.

I would feel much better if somebody would finally write down *exactly* 
what the UTF-7 vulnerability is...

>   - Auto-detection under explicit "iso-8859-1" charset label is dirty.

Yes, it means overriding what should be authoritative metadata.

>     supported under HTML5 rules.

But it seems to me this is the whole point of it, isn't it?

>     character encodings which are marginal on this property.

What does being "marginal on this property" mean?

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Thu, 14 Feb 2008 18:18:03 +0100

Roy T. Fielding wrote:

 [p3 version] 
>| Some HTTP/1.0 software has interpreted a Content-Type header without
>| charset parameter incorrectly to mean "recipient should guess."
>| Senders wishing to defeat this behavior MAY include a charset
>| parameter even when the charset is ISO-8859-1 ([ISO-8859-1]) and
>| SHOULD do so when it is known that it will not confuse the  
>| recipient.

When senders wish to defeat that s/behavior/misbehavior/ they
s/MAY/have to/ include a charset parameter.  It is no OPTION
after a decision to try it.   

The optional part could be s/Senders wishing/Senders MAY wish/,
but a recommendation is clearer:  "Senbers SHOULD avoid this
misbehavior by including a charset parameter." (period)

Strike the "even when" Latin-1 blurb, as that's precisely the
cause of this historical mess.  Explicit Latin-1 really *is*
Latin-1, and where that is not the case (e.g. windows-1252)
it is broken.  Specifying varying degrees of brokenness is a
dubious idea, maybe put it in a note:

"Note:  Historically Latin-1 ([ISO-8859-1]) was a predominant
charset, and some senders explicitly announced this charset
even when it was incorrect."  The conclusions for readers and
implementors are obvious, there be dragons.

>| and SHOULD do so when it is known that it will not confuse
>| the recipient.

Strike that, nobody knows what confuses recipients.  Some old
Mosaic browsers have to bite, a reasonable lines is "if an UA
cannot do Host: header fields it is hopeless".  

>| Unfortunately, some older HTTP/1.0 clients did not deal
>| properly with an explicit charset parameter.

Yes, but it doesn't affect what servers do in this millennium,
unless you want it as justification for an explicit "MAY omit
Latin-1" in 2616bis.  Going back to the start this could be 
the excuse where senders violate (see above) "Senders SHOULD
avoid this misbehavior by including a charset parameter."

Noting the good excuses to violate a SHOULD makes sense.  But
in this case limited to HTTP/1.0 and some "hopeless" browsers
I think 2616bis can get away without convoluted explanations.

>| HTTP/1.1 recipients MUST respect the charset label provided
>| by the sender; and those user agents that have a provision to
>| "guess" a charset MUST use the charset from the content-type
>| field if they support that charset, rather than the
>| recipient's preference, when initially displaying a document.

I don't see remotely why recipients "MUST" do this, it is just
information, with a proposed historical note explaining a case
where the information "Latin-1" could be wrong.

How clients display documents is not the business of HTTP, that
task is minimally two protocol layers above HTTP.  Let's strike
this paragraph, the word *display* triggered my bogon detector.

>| Internet media types are registered with a canonical form.  An
>| entity-body transferred via HTTP messages MUST be represented
>| in the appropriate canonical form prior to its transmission
>| except for "text" types, as defined in the next paragraph.

s/except for/with the possible exception of/  After all HTTP
still "allows" to use CRLF for canonical line ends in text/*.

>| HTTP relaxes this requirement and allows the transport of
>| text media with plain CR or LF alone representing a line
>| break when it is done consistently for an entire entity-body.

s/relaxes/does not depend on/ and then just say "with other
line break indicators including but not limited to bare LF".

The "bare CR" case is for a now historical platform, and IMO
2616bis doesn't need to talk about "bare CR" explicitly (?)  

>| HTTP applications MUST accept CRLF, bare CR, and bare LF
>| as being representative of a line break in text media
>| received via HTTP.

I think what you really want is "MUST NOT modify other line
break conventions on the fly", as opposed to non-binary FTP.

>| if the text is represented in a character set that does
>| not use octets 13 and 10 for CR and LF respectively, as
>| is the case for some multi-byte character sets

What about NL in text/xml Latin-1, a charset offering CRLF ?
Or FWIW in UTF-1 ?  A note about one representative case
where the octets 0A does not mean LF should suffice to make
the point, proposal:

"Note that octet 10 (decimal) does not necessarily mean LF
 (u+000A) in various charsets, e.g., u+010A in UTF-16."

Trim the rest, keeping something in the direction of:

>| This flexibility regarding line breaks applies only to
>| text media in the entity-body; a bare CR or LF MUST NOT
>| be substituted for CRLF within any of the HTTP control
>| structures (such as header fields and multipart boundaries).
[...]

Back to the subject of this thread:

>| When no explicit charset parameter is provided by the
>| sender, media subtypes of the "text" type are defined to
>| have a default charset value of "ISO-8859-1" when received
>| via HTTP.

s/are defined to have a default/used to have a default/.  Add:
"This HTTP/1.0 work around for historic browsers choking on
 an explicit charset ISO-8859-1 is not more needed, senders
 SHOULD (see 2.1.1) label ISO-8859-1 explicitly."

>| Data in character sets other than "ISO-8859-1" or its 
>| subsets MUST be labeled with an appropriate charset value.
>| See Section 2.1.1 for compatibility problems.

ACK.  Potential issues in your version:

>: When a media type is registered with a default charset value
>: of "US-ASCII", it MAY be used to label data transmitted via
>: HTTP in the "iso-8859-1" charset (a superset of US-ASCII)
>: without including an explicit charset parameter on the media
>: type.

For 2616bis that should be no valid option (MAY), it should be
a *violation* of a new SHOULD for the stated historical reason.
Going from MAY to SHOULD NOT is possible, nothing breaks.

>: In addition, when a media type registered with a default
>: charset value of "US-ASCII" is received via HTTP without a
>: charset parameter or with a charset value of "iso-8859-1",
>: the recipient MAY inspect the data for indications of a
>: different character encoding
[...]

That is convoluted.  Certainly it "MAY" try to determine the
charset by sniffing if there is no charset, arguably it "must"
(lower case) do this for the (non-HTTP) purpose of displaying
a document.  And it "MAY" do this whenever it wishes, the case
of an erroneous iso-8859-1 IMO does not justify a HTTP "MAY".

As far as HTTP is concerned an explicit charset means what it
says, including charset="iso-8859-1".  Where that is incorrect
it is an ordinary bug on the side of the sender.  Limit this
oddity to a note (as proposed above)

>: if the encoding can be determined within the first 16 octets
>: of data and interpreted consistently thereafter.

Please no arbitrary magic numbers like "16" in a standard, let
alone in a standard where the complete "sniffing" business is
off topic.

>: Note: The first variance is due to a significant portion of
>: early HTTP user agents not parsing media type parameters and
>: instead relying on a then-common default encoding of iso-8859-1.
>: As a result, early server implementations avoided the use of
>: charset parameters and user agents evolved to "sniff" for new
>: character encodings as the Web expanded beyond iso-8859-1
>: content.

Yes, and (as you noted in another article) servers have no time 
for any sniffing on their side for dynamical content.  But that
does not justify a "variance" going as far as an option (MAY),
violating a SHOULD NOT is good enough for this historical case.

I don't see why 2616bis should try to overrule text/xml defaults
with a MAY, as HTTP certainly does not try to tell clients what
a say image/x-icon might be, and how to display it.

>: The second variance is due to a certain popular user agent that
>: employed an unsafe encoding detection and switching algorithm
>: within documents that might contain user-provided data (see
>: Section security.sniffing), the most common workaround for
>: which is to supply a specific charset parameter even when the
>: actual character encoding is unknown.

No.  Plausible reasons why servers might intentionally lie with
"iso-8859-1" do not belong in an Internet standard.  If an UA is
broken it needs to be fixed.  Servers could also try their luck
with the registered "unknown-8bit" instead of lying, this is out
of scope for HTTP.

 Frank

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 14 Feb 2008 11:16:34 -0800

On Feb 14, 2008, at 5:48 AM, Julian Reschke wrote:
> And I don't think we realized that we may be *adding* requirements;  
> the goals were (IMHO) to reduce inconsistency of specs and to allow  
> what UAs indeed do today. It seems the conclusion is that in this  
> case these goals are contradictory.
>

I didn't mean to imply otherwise -- I seriously have no recollection,  
nor
saved messages (I normally save any message that calls for a change), of
the specific diff instructions included in that issue and that you  
applied.
I didn't even know the issue was considered as having a specific  
resolution
until after the changeset was received.  [We really should have a  
separate
status for that in trac.]

All that must have occurred during my personal black-out period.  I  
don't
have any problem with applying the change as suggested -- I have a  
problem
with any claim to calling it consensus (even an emerging consensus is  
a bit
of a stretch).  Silence is not consent.  Even had I been aware of the  
diff
instruction, the effect of the result of the change is not visible until
we look at the remaining words.  We removed all of the exceptions and  
then
were left with MUST be MIME.

> In the meantime I have backed out the change (<http:// 
> www3.tools.ietf.org/wg/httpbis/trac/changeset/211>), and I'd  

Thanks.

> Q: so if a text type defines a different default, such as "UTF-8",  
> we disallow defaulting to ISO-8859-1 and sniffing?

Yes.  Error recovery is a different issue (more text on my to-do list)

> sniffing? Is that version of that UA still common enough to have  
> this exception?

MSIE 3-6?  Yes.  Apache received three more XSS reports last month,  
all due
to the same broken browser.  They resurfaced because I removed the  
default
charset setting last year.  I wouldn't be surprised if some other  
browsers
have since copied the same buggy behavior (you know how that works),  
but we
only get reports on one that I can recall.

> - Q: do we need to state that we overrule RFC3023 (WRT default  
> charset for text/xml?)

No.  RFC 3023 defines a couple media types for email and is therefore
incapable of going beyond MIME's requirements on text/* (even though
most MIME clients nowadays do their own sniffing).

> - we still need the security consideration WRT sniffing.

Yep, ran out of time.

....Roy

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 14 Feb 2008 11:42:35 -0800

> status for that in trac.]

That would be nice, but the current version of Trac doesn't support  
this; custom workflows will be available in .11, which I'm hoping will  
be final and on the IETF tools site soon.

Also, while doing it in Trac is convenient for me and the editors, it  
doesn't really improve visibility for the rest of the WG. It's  
important that discussion and decisions be made on the list, not in  
the tools.

So, I'm going to try doing the following:
   1) linking the issues that are being considered for closure on our  
wiki page <http://www3.tools.ietf.org/wg/httpbis/trac/wiki/WikiStart>,  
and
   2) recording *mature* proposals in the issues list, and
   3) highlighting potential closures to design issues with a distinct  
Subject line, and
   4) sending a weekly status e-mail to the list.

Feedback as to whether any of these help, as well as additional  
suggestions, welcome.

---

From: Julian Reschke <[email-masked]>
Date: Thu, 14 Feb 2008 21:17:04 +0100

Mark Nottingham wrote:
> doesn't really improve visibility for the rest of the WG. It's important 
> that discussion and decisions be made on the list, not in the tools.
> 
> wiki page <http://www3.tools.ietf.org/wg/httpbis/trac/wiki/WikiStart>, and

All of this sounds good.

In particular 4), it might help the WG concentrate on issues we want to 
close.

BR, Julian

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 14 Feb 2008 14:20:56 -0800

On Feb 14, 2008, at 11:42 AM, Mark Nottingham wrote:

>   1) linking the issues that are being considered for closure on  
> our wiki page <http://www3.tools.ietf.org/wg/httpbis/trac/wiki/ 
> WikiStart>, and
>   3) highlighting potential closures to design issues with a  
> distinct Subject line, and

I am sure they will help, though it's not as if we have run into any
significant problems so far (one revert is no big deal).

I don't think (1) is necessary if (4) contains links, but no objection.

....Roy

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 15 Feb 2008 01:50:59 +0100

[...]
> Feedback as to whether any of these help, as well as additional  

Fine.  Some related questions:

- Is the Wiki supposed to be a tool for the editors + Chairs, or
  should all here try to get an account or whatever it takes for
  write access ?  [ Better keep it for the editors + Chair ;-) ]

- What's going to happen with the old "outsourced" errata ?  One
  2617 erratum has a major impact on various Internet protocols,
  compare <http://article.gmane.org/gmane.ietf.sasl/3007>.

- Julian proposed to split the question of "default Latin-1" 
  from issues related to "line break" conventions (over HTTP),
  is the latter already tracked separately ?

- several folks demanded to kill UTF-7 a.s.a.p. at odd places not
  limited to HTTP and "HTML 5".  Can we have a clear decision that
  this should be done generally, once and for all, allowing HTTP
  and "HTML5" to reference a document doing this as they see fit ?

 Frank

---

From: "Roy T. Fielding" <[email-masked]>
Date: Thu, 14 Feb 2008 19:01:57 -0800

It looks like 1/2 of your response is about small changes to the
text that is being deleted, and another 1/4 about the bits left
after the last change, and only the last 1/4 about my proposed
rewrite.  That is really confusing.

I'm just going to skip ahead to the comments on my proposed text...

> On Feb 14, 2008, at 9:18 AM, Frank Ellermann wrote:
>> Roy T. Fielding wrote:
>>

That would change the protocol such that all currently compliant
HTTP senders that transmit text messages in "iso-8859-1" without
a charset parameter would be violating a SHOULD requirement.
My proposal states the fact that such messages do occur in practice
and alters the MIME requirement for HTTP to accommodate them.

> of an erroneous iso-8859-1 IMO does not justify a HTTP "MAY".

If browsers are willing to implement that, fine.

> alone in a standard where the complete "sniffing" business is

It is more important that it works (or that we find out it doesn't).

> violating a SHOULD NOT is good enough for this historical case.

Sorry, that decision was made in 1994 and is now way out of scope.

> a say image/x-icon might be, and how to display it.

Then you don't know (or don't care) what the MIME specs say.  I do.
It was an intentional decision based on the needs of different
protocols.  The other alternative would be to define a separate
media type registration system, which was considered more harmful
than simply stating the differences and noting the requirements
for translating an HTTP-compliant message to a MIME-compliant
message.

> with the registered "unknown-8bit" instead of lying, this is out

Then get back to us when you have fixed that user agent.  Sending
any charset that is invalid/unknown to that user agent will fail to
trigger the one safe path that allows us to workaround its stupid
bugs.  All I am trying to explain in that paragraph is why the
theory of "servers should just leave the charset empty" is never
going to happen in the foreseeable future.  Otherwise, I am happy
to mark the issue as WONTFIX and let the browsers deal with their
own bugs.

....Roy

---

From: "Frank Ellermann" <[email-masked]>
Date: Fri, 15 Feb 2008 17:32:08 +0100

> rewrite.  That is really confusing.

Well, I was confused, I found your long "p3" text, and seeing that
it went on and on thought that's your proposal, commenting inline
as good as possible.  And about an hour later I arrived at the cut
where you started your real/new proposal.

After that I marked the first part as "p3" keeping ">|" as quote
indicator, and used ">:" for your real/new proposal.  Whatever
"p3" actually is, it is what you wrote, and took about 3/4 of my
reply, yielding a complete proposal based on "p3".

The last 1/4 was what you said above.

> a charset parameter would be violating a SHOULD requirement.

Yes, that is the point of getting rid of "default Latin-1", which
was quite popular in this long "unknown text/* subtypes" thread,
and before.   IIRC Martin proposed to get rid of it back in 2006,
admittedly that was about 2617bis, and at that time I screamed.

The HTML5 WG apparently wants to replace Latin-1 by windows-1252
to some degree, I didn't check the details, only what I saw when
trying to figure out interesting points in the "HTNL5 diff" draft.

> My proposal states the fact that such messages do occur in

Sure, but that's not more desirable.  As you explained earlier it
used to be a kind of hack 15 years ago, and one of the early RFCs 
already said "we'll do Unicode a.s.a.p.".

 
> It is more important that it works (or that we find out it 

IMO it is a design principle, stay away from magic numbers unless
you really must have them, and then folks will ask "why 16 if 512
is a much better buffer size ?"  Which could be ignored as stupid
question - unless it turns out that 512 is really better.

HTTP in its role as "TP" doesn't need to sniff, so why specify it
at all here ?  Whatever browsers do, it has also has to work with
other protocols (or other URI schemes to cover the file: case).

>> (as you noted in another article) servers have no time for any
>> sniffing on their side for dynamical content.  But that does 
>> not justify a "variance" going as far as an option (MAY),
 
> Sorry, that decision was made in 1994 and is now way out of scope.

It's 2008 now, the old browsers choking on an explicit charset did
HTTP/1.0 without a Host: header field, they are gone.  I tried to
use "IBM Webexplorer" in this millennium a few times, they really
are hopeless.  

A decision in 1994, before UTF-8 existed, is not necessarily good
enough today.  UTF-7 was published 1994, it was cute, but we all
agree to deprecate it somehow for today.  

> Then you don't know (or don't care) what the MIME specs say.

I know that image/vnd.microsoft.icon is a registered MIME type, and
I think that 2616bis doesn't need to talk about it.

>> Plausible reasons why servers might intentionally lie with
 
> Then get back to us when you have fixed that user agent.

If we are talking about IE6 among others, I get monthly fixes for
security issues in this beast.  If something in IE6 is so horribly
wrong that it affects what 2616bis will say 2009 I'd like to know
precisely what it is.

What you propose boils down to "iso-8859-1" means "dunno", and it
is required to disable some unclear (for me) bug in IE versions.
Likewise no charset means "iso-8859-1" and it is required for old
Mosaic derivates back in 1994.  

And servers wishing to announce a real Latin-1 text can pick what
they like better, but actually no client (apart from IE and Mosaic
with their own problems) can actually believe them.

Do you see how *odd* that sounds, when I extract it from your 2119
prose ?  Is "iso8859-1" = "dunno" really what you want in 2616bis ?

If HTTP servers wish to use this hack, why can't they limit this
effort depending on the User-Agent ?  

 Frank

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 19 Feb 2008 16:58:13 +1100

>  write access ?  [ Better keep it for the editors + Chair ;-) ]

Right now, anyone who wants to get a tools account can comment on open  
issues and work on the wiki. Having said that, I do see it primarily  
as a tool for the chair and editors, as well as for WG members to keep  
up with what we're doing. I'm not particularly against folks adding to  
what's there, as long as it's in-scope and useful, but discussion and  
decisions need to happen on the list, so I'm not sure what the benefit  
would be. What did you have in mind?

>  compare <http://article.gmane.org/gmane.ietf.sasl/3007>.

Good question. The 2617 errata still stands at Scott's URI <http://skrb.org/ietf/http_errata.html 
 >, but it hasn't been transferred to our list, since it's not in our  
scope (yet, at least).

>  is the latter already tracked separately ?

Not IIRC; I've been on the road for a couple of weeks, so have fallen  
a bit behind on the issues list. I'll get it up there ASAP.

>  and "HTML5" to reference a document doing this as they see fit ?

My initial feeling is that apps-discuss might be a good place to bring  
that up; while we could publish a document that makes recommendations,  
if it really is bigger than HTTP, it probably deserves a bigger venue.

Cheers,

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 19 Feb 2008 09:53:31 +0100

Mark Nottingham wrote:

> What did you have in mind?

Now I forgot it, maybe it was about the RFC 2617 erratum wrt
MD5-sess.  Or about extracting a "P3"-based proposal from the
"unknown text/* type" thread, to get rid of default Latin-1.  

 Frank

---

From: "Brian Smith" <[email-masked]>
Date: Tue, 26 Feb 2008 18:14:41 -0800

Mark Nottingham wrote:
> > should all here try to get an account or whatever it takes 
> > for write access ?

> Right now, anyone who wants to get a tools account can 
> comment on open issues and work on the wiki. Having said 
> that, I do see it primarily as a tool for the chair and 
> editors, as well as for WG members to keep up with what we're 
> doing. I'm not particularly against folks adding to what's 
> there, as long as it's in-scope and useful, but discussion 
> and decisions need to happen on the list, so I'm not sure 
> what the benefit would be. What did you have in mind?

In the course of discussions on i69, several other issues were raised:

* What is the definition of Content-* headers?

* Why does PUT have a specific requirement regarding Content-* headers
while POST and other entity-carrying requests don't?

* If multiple resources can be created in a single (POST/PUT) request,
then the descriptions of these methods and of the 201 response code
should explicitly state that, and should be rewritten to use plural
instead of singular language.

Also, I just posted two more issues to the mailing list:

* The "Allow" header field is wrongly classified as an entity header.
* Unrecognized header fields should not be assumed to be entity headers.

These are not issues that the WG is currently working on (according to
the hope page of the wiki), and so I don't expect any immediate response
from the editors. But I don't want these issues to get lost or forgotten
either. I'd like to see them added to the issue tracker, and I am
willing to do it myself, but I've avoided doing because it seems to be
reserved for the editors' use.

- Brian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Feb 2008 12:12:53 +0100

> reserved for the editors' use.

Actually, for design issues, the editors do not have any special 
privileges with respect to the issue tracker (except for being able to 
close issues when a agreed-upon change was applied).

I do agree that getting this into the issue tracker needs to be 
optimized; how about using the Wiki to draft a problem statement (and 
potentially a proposed change)? (Of course the information still needs 
to go to the mailing list so that it's properly archived and IPR is 
taken care of).

BR, Julian

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 28 Feb 2008 08:38:10 +1100

Yes. Sorry for the delay; I've had them (and others) flagged, but my  
family has been sick for the last week and a half, so I've fallen  
behind. I should be getting things in today.

Cheers,