Subject: Re: Unknown text/* subtypes

From: Julian Reschke <[email-masked]>
Date: Fri, 04 Jan 2008 17:33:36 +0100

Martin Duerst wrote:
> At 00:45 07/12/19, Frank Ellermann wrote:
>> Julian Reschke wrote:
>>
>>> there's also RFC2616
>> Yes, that's an ugly legacy exception...  
>>
>>> <http://www.w3.org/Protocols/HTTP/1.1/rfc2616bis/issues/#i20>
>> ...maybe 2616bis can drop this oddity in favour of a simple
>> "unknown text is ASCII" rule.
> 
> The new version of the HTTP spec, 2616bis, should definitely
> drop the iso-8859-1 default, but NOT in favor of "unknown text is ASCII".
> It should just say that there is no default.

As far as I understand, we currently have RFC2046, RFC2616 and RFC3023 
making conflicting requirements:

RFC2046: the default for text/* is US-ASCII 
(<http://tools.ietf.org/html/rfc2046#section-4.1.2>)

RFC2616: the default for text/* received over HTTP is ISO8859-1 
(<http://tools.ietf.org/html/rfc2616#section-3.7.1>)

RFC3023: the default for text/xml is US-ASCII, even when received over 
HTTP (<http://tools.ietf.org/html/rfc3023#section-3.1>)

This is a mess, and as far as I can tell, it would be good if at least 
HTTP would get out of it.

So it seems that we need to decide on two separate questions:

1) Do we want HTTP to override RFC2046's defaults at all?

2) If we do want to continue that, what do we want to mandate?

Right now, browsers (just tested Opera/Safari/Mozilla/IE7) ignore all 
three RFCs for at least text/xml (they all look at the content).

If our answer to 1) is "no", the content will still be broken, but at 
least it's not HTTP's fault anymore.

Otherwise we can state "in absence of charset parameter recipient MAY do 
charset sniffing (BOM, XML decl, HTML meta tag, ...), which would 
probably match what's actually implemented.

> There is a big difference between these two, especially
> for document formats that contain internal 'charset' information.
> A default of US-ASCII makes document-internal 'charset' information
> useless (because the external information wins). No default means
> that the recipient will look at the internal information.

Yep.

>> HTTP oddities shouldn't affect
>> MIME registrations, there's no string "2616" in BCP13.
> 
> One reason for the problems with text/xml was that the
> original MIME default of US-ASCII was enforced. This made
> it impossible to serve XML documents with internal 'charset'
> information only as text/xml.

Hmm. So why did RFC3023 then mandate US-ASCII again? In general, is it 
acceptable *at all* to override charset defaults defined in RFC2046?

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Tue, 8 Jan 2008 05:04:04 +0100

Julian Reschke wrote:

 [I've removed the types list, feel free to reinsert it]
> 1) Do we want HTTP to override RFC2046's defaults at all?

No.  Overriding it with UTF-8 would make sense (later, not
in 2616bis).  Let's go back to the 2046 defaults for now. 

> browsers (just tested Opera/Safari/Mozilla/IE7) ignore all 

Of course, authors might know what they do, besides browsers
also have to work with the content behind file and ftp URLs.
And it's tricky to get this right for HTTP server admins...

> we can state "in absence of charset parameter recipient MAY
> do charset sniffing (BOM, XML decl, HTML meta tag, ...),
> which would probably match what's actually implemented.

...HTTP offers a sound mechanism, what browsers do when that
mechanism is not used could be "out of scope" for this WG.
Let's say default ASCII, better guesses are no HTTP problem.

 Frank

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 9 Jan 2008 14:16:42 +1100

Personally -- I agree; the only sane thing to do here seems to be to  
remove HTTP defaulting.

The simplest thing seems to be to remove this text;

> When no explicit charset parameter is provided by the sender, media  
> subtypes of the "text" type are defined to have a default charset  
> value of "ISO-8859-1" when received via HTTP.

BUT, note the following text:

> Data in character sets other than "ISO-8859-1" or its subsets MUST  
> be labeled with an appropriate charset value.

Depending on how you read the context, this would need to be restated  
as something like:

"Media subtypes of the "text" type MUST be labeled with an appropriate  
charset value."

As I think I've said before, requiring this often leads to  
mislabelling, because (for example) a Web server administrator will  
set an unrealistic policy like "all of our content is UTF-8",  
configure headers to suit, forgetting some legacy content on the site  
that's in a different encoding.

My preference would be to soften this to a SHOULD, so that in cases  
where it's administratively difficult for people to set a charset  
value, conflicting statements aren't made. I'd rather have the  
metadata be explicitly missing than wrong.

--
Mark Nottingham     http://www.mnot.net/

---

From: "Robert Brewer" <[email-masked]>
Date: Wed, 9 Jan 2008 09:48:24 -0800

> metadata be explicitly missing than wrong.

Except that administrative difficulties lead to complaints which lead to
fixed software where nothing else seems to. As a server implementor, I'm
tired of guessing charsets on request bodies [1]. I'd much rather see a
MUST so the conversation fails in a way that clearly fixes blame, even
though I may choose to work around it.

Robert Brewer
CherryPy Team
[email-masked]

[1] http://www.cherrypy.org/browser/trunk/cherrypy/lib/encoding.py#L25

---

From: "Javier Godoy" <[email-masked]>
Date: Wed, 9 Jan 2008 16:51:58 -0200

----- Original Message ----- 
From: "Robert Brewer" <[email-masked]>
To: "Mark Nottingham" <[email-masked]>; "Frank Ellermann" 
<[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, January 09, 2008 3:48 PM
Subject: RE: Unknown text/* subtypes [i20]

Robert Brewer wrote:

>> note the following text:

>From RFC 2119:
 SHOULD   This word, or the adjective "RECOMMENDED", mean that there
   may exist valid reasons in particular circumstances to ignore a
   particular item, but the full implications must be understood and
   carefully weighed before choosing a different course.

In this case, the "full implications" of a client not sending a charset 
parameter would be that the server
may do a wrong guess (AFAIK, it is not REQUIRED to do any guess at all).

This situation has a counterpart on the recipient-side, as Julian said on 
2008-01-04
> (...) we can state "in absence of charset parameter recipient MAY do

+1

>From RFC 2119:
 MAY   This word, or the adjective "OPTIONAL", mean that an item is
   truly optional.  One vendor may choose to include the item because a
   particular marketplace requires it or because the vendor feels that
   it enhances the product while another vendor may omit the same item.

I think it adequately describes the situation you refer (as complaints lead 
to implementing OPTIONAL features, which would have not been implemented 
otherwise).

Regards,

Javier
.

---

From: Geoffrey Sneddon <[email-masked]>
Date: Thu, 10 Jan 2008 16:26:14 +0000

On 4 Jan 2008, at 16:33, Julian Reschke wrote:

> RFC2616: the default for text/* received over HTTP is ISO8859-1 (<http://tools.ietf.org/html/rfc2616#section-3.7.1 
> >)

Of note is the fact that real implementations tend to use Windows-1252  
(a superset of ISO-8859-1's graphical characters) in place of  
ISO-8859-1, as content relies upon this (though they mostly try some  
sort of encoding sniffing first).

> Otherwise we can state "in absence of charset parameter recipient  
> MAY do charset sniffing (BOM, XML decl, HTML meta tag, ...), which  
> would probably match what's actually implemented.

It isn't "in absence of a charset parameter" though, at least in the  
case text/xml (I've come across feeds served as "text/ 
xml;charset=ANSI" that have a charset of GB2312 — I strongly doubt  
that text/xml is the only MIME type to be affected like this).

--
Geoffrey Sneddon
<http://gsnedders.com/>

---

From: Ian Hickson <[email-masked]>
Date: Sun, 13 Jan 2008 05:47:27 +0000 (UTC)

On Fri, 28 Dec 2007, Frank Ellermann wrote:
> 
> Years later (after 2616bis) it might be possible to upgrade "default 
> ASCII" to UTF-8, Latin-1 was a dead end.  As soon as we're back to 
> "default ASCII" just let RFC 2277 finish it off.

FWIW, a number of specs are already overriding both MIME and HTTP when it 
comes to character encodings. For example HTML4 says to not default to any 
encoding at all [1], CSS defaults to a complicated heuristic [2], HTML5 as 
currently proposed defaults to an even more complicated heuristic [3], and 
so on.

In the "real world" the implementations are following the heuristics 
described in CSS2.1 and HTML5 (or something close to them), and those 
differ for text/css and text/html, so it would seem pointless for HTTP to 
try to define something here: it would just get ignored.

IMHO the best option is for HTTP to stay out of the discussion altogether 
and let the lower level specs (MIME) and the higher level specs (XML, 
HTML, CSS, etc, defining the formats) figure it out amongst themselves.

-- Footnotes --

[1] http://www.w3.org/TR/html4/charset.html#h-5.2.2
This text explicitly says that HTTP's default is useless. It then 
recomments behaviour that is even more useless, but that's another 
problem altogether...

[2] http://www.w3.org/TR/CSS21/syndata.html#charset

[3] http://www.whatwg.org/specs/web-apps/current-work/multipage/section-parsing.html#determining

Cheers,
-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'

---

From: "Larry Masinter" <[email-masked]>
Date: Sun, 13 Jan 2008 08:53:19 -0800

"If we couldn't fix it then, why do you imagine you can fix it now?"

I'm arguing for documenting current practice, making some recommendations
for safe behavior, and moving on.

We certainly *wanted* to change the default charset for HTTP when working on
2026 but couldn't find a way around the impasse between backward
compatability, client sniffing, server misconfiguration et al.

I think the main thing to do is to document the actual situation
sufficiently such that new HTTP implementations don't break things:

1) servers (senders): don't make up a charset if you don't know what it is
(this is a good rule for any kind of descriptive information, isn't it?). 
2) clients (receivers): servers (senders) are unfortunately often
misconfigured and will label things with the wrong charset. (This is often
because lots of software uses 'mime type' when what's wanted is usually
'content type' and the parameters get lost). But guessing blindly and
ignoring what the server sent seems like a bad idea, and even has security
implications. So "beware".
3) everybody: even if you agree about charset, accept other end-of-line
terminations (not just CRLF which MIME required.)

(It's really receiver & sender, not client & server, since the rules should
apply for file upload as well as anything else.)

Larry

---

From: Eric Prud'hommeaux <[email-masked]>
Date: Sun, 13 Jan 2008 13:40:12 -0500

* Ian Hickson <[email-masked]> [2008-01-13 05:47+0000]
> HTML, CSS, etc, defining the formats) figure it out amongst themselves.

I think this is consistent with Martin's proposal that HTTP1.1bis not
set a default encoding
  http://www.w3.org/2008/01/rdf-media-types#noDefault
(noting that Frank Ellerman believed the default should be us-ascii for
 the same effect)
  http://www.w3.org/2008/01/rdf-media-types#defAscii

What we still need, however, is an update to 2046 that reflects
current practice (and eases the discovery process for folks
registering non-ascii text/ media types). Let's geek out the
changes to we'd like to see.

• CRLF rules:
[[
  The canonical form of any MIME "text" subtype MUST always represent
  a line break as a CRLF sequence.  Similarly, any occurrence of CRLF
  in MIME "text" MUST represent a line break.  Use of CR and LF
  outside of line break sequences is also forbidden.
]] — RFC2046 §4.1.1 ¶1 http://www.rfc.net/rfc2046.html#s4.1.1.
is not respected by HTTP1.1, nor is it respected in general when
shipping text/xml.

Does anyone rely on any vestige of this rule (e.g. mail clients, MTAs,
web servers, proxies or clients)? I would like to think that MIME
shouldn't care about recognizing new lines in the text block.

If it can't go away, can it be relaxed in accordance with HTTP 1.1
[[
  The line terminator for message-header fields is the sequence CRLF.
  However, we recommend that applications, when parsing such headers,
  recognize a single LF as a line terminator and ignore the leading
  CR.
]] — RFC2616 §19.3 ¶3 http://www.rfc.net/rfc2616.html#s19.3
or XML 1.1 (which includes NEXT LINE (NEL) and LINE SEPARATOR):
[[
   1. the two-character sequence #xD #xA

   2. the two-character sequence #xD #x85

   3. the single character #x85

   4. the single character #x2028

   5. any #xD character that is not immediately followed by #xA or
      #x85.
]] — XML 1.1 §2.11 ¶2 http://www.w3.org/TR/xml11/#sec-line-ends

The XML 1.1 rule interacts with character encoding because, while most
character encodings line up with ascii on CR and LF, clearly none do
on #x85 and #x2028

• character encoding:
[[
Unlike some other parameter values, the values of the charset
parameter are NOT case sensitive.  The default character set, which
must be assumed in the absence of a charset parameter, is US-ASCII.

The specification for any future subtypes of "text" must specify
whether or not they will also utilize a "charset" parameter, and may
possibly restrict its values as well.  For other subtypes of "text"
than "text/plain", the semantics of the "charset" parameter should be
defined to be identical to those specified here for "text/plain",
i.e., the body consists entirely of characters in the given charset.
In particular, definers of future "text" subtypes should pay close
attention to the implications of multioctet character sets for their
subtype definitions.

The charset parameter for subtypes of "text" gives a name of a
character set, as "character set" is defined in RFC 2045.  The rules
regarding line breaks detailed in the previous section must also be
observed -- a character set whose definition does not conform to these
rules cannot be used in a MIME "text" subtype.
]] — RFC2046 §4.1.2 ¶2-4 http://www.rfc.net/rfc2046.html#s4.1.2.

When should the "default" character set apply?
  • no charset parameter
  • no charset parameter, no fixed encoding for the media type
  • no charset, no fixed encoding, no internal encoding declaration

The current text specifies the first, while HTML and CSS count on the
third. From the use case of "best effort rendering", we are already in
a state where users who are better-informed than their web or mail
clients manually set the encoding so they can see the right
characters. The following heuristics may meet or exceed the user
experience with today's data while advancing the state of the art to
enable better rendering with future data:
[[
Unlike some other parameter values, the values of the charset
parameter are NOT case sensitive. The first of the following
determinants that apply will identify the character set:

  1. charset parameter

  2. fixed encoding registered with the media type, if known

  3. encoding algorithm registered with the media type, if known

  4. UFT-8 if the document conforms to the UTF-8 encoding pattern

  5. ISO-8859-1 if all the octets are in [\r\n\x20-\x7e]

  6. application preference
]]

@@charset constraints — can it have faux line feeds?

@@bidi? Martin, what do you think?

@@lowest common demoninator:
  RFC2046 §4.1.2 ¶22 http://www.rfc.net/rfc2046.html#s4.1.2.
Is it better to encourage the world to write "UTF-8" or "US-ASCII"
for ascii subset? tension between lcd and one common encoding.

@@Content-Transfer-Encoding: Base64
  Content-Type: text/wibbly
How does TE affect this? I suspect it's completely orthogonal.

-- 
-eric

office: +1.617.258.5741 NE43-344, MIT, Cambridge, MA 02144 USA
mobile: +1.617.599.3509

([email-masked])
Feel free to forward this message to any list for any purpose other than
email address distribution.

---

From: Ned Freed <[email-masked]>
Date: Sun, 13 Jan 2008 16:26:01 -0800 (PST)

> changes to we'd like to see.

You might, and I emphasize might, be able to get this changed to protocol
specific restriction. (The MIME specifications specify both an email-specific
extension as well as some more generally useful facilities.) There is no chance
of this rule being lifted in general.

> web servers, proxies or clients)?

Not only does email depend on this, conformance to this has been dramatically
strengthened, not weakened, in subsequest revisions of the email protocol
specification. Specifically, RFC 821 was essentially silent on what bare CR and
LF mean, but 2821 and 2821bis (now in last call) both say that bare CR and LF
MUST NOT be sent and if received MUST NOT be treated as CRLF.

This, incidentially, is not the way I personally think things should have been
done. I like the "ignore bare CR treat LF like CRLF" approach. But my personal
opinion isn't especially relevant - I mention it only to avoid "shoot the
messenger" sorts of responses.

> I would like to think that MIME

I'm sorry, but that's fanciful in the extreme.

> ]] — RFC2616 §19.3 ¶3 http://www.rfc.net/rfc2616.html#s19.3

Again, I personally think this is the way to go. But that's not what
has happened.

>   6. application preference

Again, there is absolutely no chance this will fly for email so it cannot be 
written with this degree of generality. And if this is made protocol specific
the specifics of any protocol other than email don't belong in a RFC 2046
revision.

> for ascii subset? tension between lcd and one common encoding.

Marking something as utf-8 when it is in fact restricted to the us-ascii subset
has been known to cause problems. I think change in this area is unlikely.

				Ned

---

From: Ian Hickson <[email-masked]>
Date: Tue, 15 Jan 2008 07:08:44 +0000 (UTC)

On Tue, 15 Jan 2008, Frank Ellermann wrote:
> >
> > For example HTML4 says to not default to any encoding at all [1]
> [...]
> 
> Yes, but HTTP has to work for plain text

RFC 2046 handles that.

> pre-HTML 4

HTML5 defines processing for all versions of HTML, so that's taken care of 
too.

> etc

etc like what?

> and I think HTTP needs its own idea of what is allowed in a HTTP header.

Sure, but that's unrelated to text/*.

---

From: "Larry Masinter" <[email-masked]>
Date: Thu, 17 Jan 2008 10:57:29 -0800

(bcc: http-wg because this is about the ietf-types proposal for RDF types)

I think it would be good to separate out the discussion of HTTP & browser
behavior from the MIME registrations of RDF representations. 

The simple solution for Media Types for RDF languages is -- if you really
want to use text/*, then always use a charset parameter, even if you'd
rather not. 

Don't try to change MIME behavior or update the default or otherwise mess
with it. You can say, using current MIME, what you want to say. Perhaps the
default to US-ASCII is annoying, the content-type string is longer than you
would like, but changing MIME is unnecessary.

Secondly, don't use use the '+' syntax, as in: text/rdf+n3, text/rdf+turtle
since it doesn't fit into the paradigm established for +xml; this was noted
during the discussion of registration of +zip types.

For an alternative, I imagine there are lots of possibilities. Personally, I
prefer being more explicit, e.g.,

Label N3 documents as:
     text/w3c.rdf.n3;charset=utf-8

label Turtle files as:
     text/w3c.rdf.turtle;charset=utf-8

Larry

---

From: Eric Prud'hommeaux <[email-masked]>
Date: Thu, 17 Jan 2008 18:48:58 -0500

* Ned Freed <[email-masked]> [2008-01-13 16:26-0800]
> MUST NOT be sent and if received MUST NOT be treated as CRLF.

I guess we can measure compliance by what causes the clients to wrap
lines. Attached is a set of documents in different encodings, each
served with text/x-unknown and no charset. I expet this is a violation
of 2046, but I wanted to see what clients actually do.

These documents are also available on the web:

http://www.w3.org/2008/01/text/CRLF.txu		    CRs and LFs
http://www.w3.org/2008/01/text/latin-1-diac.txu	    díâçrìtïcàls in latin-1
http://www.w3.org/2008/01/text/utf-8-diac.txu	    díâçrìtïcàls in utf-8
http://www.w3.org/2008/01/text/utf-8-cyrillic.txu   сириллик in utf-8
http://www.w3.org/2008/01/text/utf-8-kanji.txu	    漢字 in utf-8
http://www.w3.org/2008/01/text/shift-jis-kanji.txu  漢字 in shift_jis

and their behavior in various web clients is documented at
  http://esw.w3.org/topic/ConformanceNotes

> I'm sorry, but that's fanciful in the extreme.

Let me try provide a concrete argument against my proposal:

"Mail, News and HTTP clients all need to render text/ stuff on the
screen, and a shared understanding of CRLF is essential to the user
experience."

> Again, I personally think this is the way to go. But that's not what

It's not what happened spec-wise, but I'm using this to see what happened
in the implementations.

> the specifics of any protocol other than email don't belong in a RFC 2046

fair point. HTTP1.1bis folks may want to consider this.

> has been known to cause problems. I think change in this area is unlikely.

tx for your thoughts and attention on this.

> 				Ned

@@is a "no-change revision" is changing things if it
documents current practice contrary to the old spec.
-- 
-eric

office: +1.617.258.5741 32-G528, MIT, Cambridge, MA 02144 USA
mobile: +1.617.599.3509

([email-masked])

---

From: Eric Prud'hommeaux <[email-masked]>
Date: Thu, 17 Jan 2008 19:13:34 -0500

* Eric Prud'hommeaux <[email-masked]> [2008-01-17 18:48-0500]
>        Attached is a set of documents in different encodings, each

-- 
-eric

([email-masked])

---

From: Julian Reschke <[email-masked]>
Date: Sun, 20 Jan 2008 12:28:41 +0100

> "If we couldn't fix it then, why do you imagine you can fix it now?"

Depends on the definition of "fixing" :-)

Given almost 10 additional years of experience, and observing what 
software actually does today, we really have sufficient reason to 
improve what the spec says.

+1

> compatability, client sniffing, server misconfiguration et al.

So let's assume we remove 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-01.html#rfc.section.2.3.1.p.4>:

"The "charset" parameter is used with some media types to define the 
character set (Section 2.1) of the data. When no explicit charset 
parameter is provided by the sender, media subtypes of the "text" type 
are defined to have a default charset value of "ISO-8859-1" when 
received via HTTP. Data in character sets other than "ISO-8859-1" or its 
subsets MUST be labeled with an appropriate charset value. See Section 
2.1.1 for compatibility problems."

Would that break anything in practice today?

> (this is a good rule for any kind of descriptive information, isn't it?). 

Right.

> implications. So "beware".

Right; maybe also point to <http://www.w3.org/2001/tag/doc/mime-respect>.

> terminations (not just CRLF which MIME required.)

<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-01.html#rfc.section.2.3.1.p.2> 
currently says:

"When in canonical form, media subtypes of the "text" type use CRLF as 
the text line break. HTTP relaxes this requirement and allows the 
transport of text media with plain CR or LF alone representing a line 
break when it is done consistently for an entire entity-body. HTTP 
applications MUST accept CRLF, bare CR, and bare LF as being 
representative of a line break in text media received via HTTP. In 
addition, if the text is represented in a character set that does not 
use octets 13 and 10 for CR and LF respectively, as is the case for some 
multi-byte character sets, HTTP allows the use of whatever octet 
sequences are defined by that character set to represent the equivalent 
of CR and LF for line breaks. This flexibility regarding line breaks 
applies only to text media in the entity-body; a bare CR or LF MUST NOT 
be substituted for CRLF within any of the HTTP control structures (such 
as header fields and multipart boundaries)."

Does this need fixing?

> apply for file upload as well as anything else.)

Correct.

So my proposal would be:

- drop paragraph 4 (ISO-8859-1),

- add a note covering Larry's points 1) and 2), and

- mention this is a normative change in 
<http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-01.html#changes.from.rfc.2616>.

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Sun, 20 Jan 2008 16:45:19 +0100

Julian Reschke wrote:

 [Removing the default Latin-1] 
> Would that break anything in practice today?

The only "victims" could be text/html for HTML 2 (maybe 3.2)
relying on this default.  That won't break HTTP, it did its
job, get object X from A to B (in the case of a GET).  What
the HTTP client later does with X is its business.  Stuff
like <meta http-equiv="Content-Type" ... is not really HTTP,
and we're not trying to add a "missing" meta to HTML 2 here.

As far as HTML 2 was "broken" it was fixed 1997 by RFC 2070,
nothing to do 2008 in HTTP.  Please correct me if I miss an
obvious clue.

> "When in canonical form, media subtypes of the "text" type
>  use CRLF as the text line break.

Is "canonical" a shorthand for "MIME compatible" ?  

>  HTTP relaxes this requirement and allows the transport of
>  text media with plain CR or LF alone representing a line 
>  break when it is done consistently for an entire entity-body.

I wonder why HTTP cares about lineends in text media types,
it has no line buffer with 1000 characters like SMTP.  HTTP
only needs to spot the begin and the end of the body, or the
similar task for chunks where that's used.

>  HTTP applications MUST accept CRLF, bare CR, and bare LF as
>  being representative of a line break in text media received
>  via HTTP.

Why should HTTP care at all about lineends in a body, it's not
supposed to fix them into local lineends on the fly, or is it ?
  
>  HTTP allows the use of whatever octet sequences are defined 
>  by that character set to represent the equivalent of CR and
>  LF for line breaks.

In other words it doesn't care, with both UTF-16, both UTF-32,
and various ECDICs there are many charsets with other ideas,
additional Unicode points to indicate lineends, all irrelevant
for HTTP as the task to get content X from A to B.

> Does this need fixing?

Maybe it needs trimming, I'm not sure.

 Frank

---

From: Albert Lunde <[email-masked]>
Date: Sun, 20 Jan 2008 16:23:27 -0500

> supposed to fix them into local lineends on the fly, or is it ?

At least one of the original motivations for HTTP's relaxed treatment
of end-of-line markers in text types was as a server optimization: 
so that servers could just send out the bytes in whatever was the 
local text format, _without_ having to convert or canonize the text.

(More sopisticated servers are free to do more, but this formulation
allowed a simpler server on various platforms.)

-- 
    Albert Lunde  [email-masked]
                  [email-masked]  (new address for personal mail)
                  [email-masked] (old address)

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 22 Jan 2008 23:21:30 +1100

On 20/01/2008, at 10:28 PM, Julian Reschke wrote:

> <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-01.html#rfc.section.2.3.1.p.2 
> > currently says:
>
> "When in canonical form, media subtypes of the "text" type use CRLF  
> as the text line break. HTTP relaxes this requirement and allows the  
> transport of text media with plain CR or LF alone representing a  
> line break when it is done consistently for an entire entity-body.  
> HTTP applications MUST accept CRLF, bare CR, and bare LF as being  
> addition, if the text is represented in a character set that does  
> not use octets 13 and 10 for CR and LF respectively, as is the case  
> for some multi-byte character sets, HTTP allows the use of whatever  
> octet sequences are defined by that character set to represent the  
> equivalent of CR and LF for line breaks. This flexibility regarding  
> line breaks applies only to text media in the entity-body; a bare CR  
> or LF MUST NOT be substituted for CRLF within any of the HTTP  
> control structures (such as header fields and multipart boundaries)."
>
> Does this need fixing?

If it does, that can be a separate issue (this text is lifted directly  
from 2616).

> - mention this is a normative change in <http://greenbytes.de/tech/webdav/draft-ietf-httpbis-p3-payload-01.html#changes.from.rfc.2616 
> >.

To be clear, we're talking about removing <http://tools.ietf.org/id/draft-ietf-httpbis-p3-payload-01.txt 
 >, section 2.3.1, the entire forth paragraph (i.e., the last one in  
that section). This includes removing both the defaulting and the MUST- 
level requirement for labeling text/* in a charset other than  
ISO-8859-1.

What should happen to section 2.1.1? Given the changes above, the only  
still-relevant part of it seems to be:
> HTTP/1.1 recipients MUST respect the charset label provided by the  
> sender; and those user agents that have a provision to "guess" a  
> charset MUST use the charset from the content-type field if they  
> support that charset, rather than the recipient's preference, when  
> initially displaying a document.
The most straightforward thing to do may be to extract the text above  
and put it at the end of 2.3.1, removing the rest of 2.1.1.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 22 Jan 2008 13:39:00 +0100

+1

> To be clear, we're talking about removing 
> <http://tools.ietf.org/id/draft-ietf-httpbis-p3-payload-01.txt>, section 
> 2.3.1, the entire forth paragraph (i.e., the last one in that section). 
> This includes removing both the defaulting and the MUST-level 
> requirement for labeling text/* in a charset other than ISO-8859-1.

Right.

> and put it at the end of 2.3.1, removing the rest of 2.1.1.

Sounds good to me.

BR, Julian

---

From: Yutaka Oiwa <[email-masked]>
Date: Wed, 23 Jan 2008 11:26:15 +0900

> requirement for labeling text/* in a charset other than ISO-8859-1.

In general, I agree for dropping "ISO-8859-1" default for text/* content types,
however, for "text/html" I have a specific concern with that.

As mentioned by many other people, many current browsers ignore HTTP/1.1
specification and implement charset auto-detection and <meta http-equiv>
tag detection.  This has caused several cross-site scripting
vulnerabilities.

The way of the attack is to insert an ASCII byte sequence which looks
like UTF-7 escaped string at some earlier point of the documents (where
browsers uses for character set detection), and insert a UTF-7-encoded
<script> tag in the documents.

The most effective countermeasure to this attack is declaring charset
in the HTTP header.  However, there are some issues about that:

  * It is not always possible to declare charsets in HTTP headers,
    especially for static contents.

  * Charsets are somewhat "open" standard, at least from the viewpoint of
    HTTP WG and W3C. It is not possible to ban future problematic
    charsets (e.g. UTF-7) from being defined.

  * Charset auto-detection and <meta http-equiv="content-type"> charset
    detection interfere each other. However, it is almost impossible to
    specify the detailed behavior of charset detection algorithms.

  * Existing ASCII-based applications should be kept safe for backward
    compatibility, at least in the specification level.

There are number of ways to solve this, and my current preference is
to add the following restrictions regarding charset auto-detection:

  * If charset is declared in the header, it MUST be honored. (current
    requirement in 2.1.1 may be copied).

  * If charset is not declared in the header, clients MAY guess the
    charset of the payload by any means (e.g. by examining the payload
    octets, using special attributions defined for content-types, or
    using the client-defined defaults).  However, if the payload is
    composed solely by octets representing ASCII printable characters and
    HTML-defined control characters (CR, LF, HT, VT and SP), it MUST be
    treated as if it is in ASCII or equivalent character sets. If the
    payload contains other octets, the behavior of clients is
    implementation-dependent.

By the above specification, the client is disallowed to guess charset
which is not ASCII upper-compatible (such as UTF-7).  The true intention
of this specification is to make detection of <meta> tags much reliable.
If UTF-7 and future ASCII-incompatible charsets are excluded, Web
authors can put <meta> declaration in the very top of HTML documents and
expect that it will be respected by the browser (as required by W3C spec).
We can further force such detection mandatory, but I feel it is overkill
for HTTP.

(I have dropped ISO-8859-1 backward compatibility to the
  implementation-defined level.  I have once written a proposal
  including full ISO-8859-1 compatibility, but it had become much complicated
  and unrealistic.  I hope this does not make any real problems.)

-- 
Yutaka OIWA, Ph.D.                                       Research Scientist
                             Research Center for Information Security (RCIS)
     National Institute of Advanced Industrial Science and Technology (AIST)
                       Mail addresses: <[email-masked]>, <[email-masked]>
OpenPGP: id[995DD3E1] fp[3C21 17D0 D953 77D3 02D7 4FEC 4754 40C1 995D D3E1]

---

From: "Roy T. Fielding" <[email-masked]>
Date: Tue, 22 Jan 2008 18:54:20 -0800

On Jan 22, 2008, at 6:26 PM, Yutaka Oiwa wrote:
> which is not ASCII upper-compatible (such as UTF-7).

I think it would be easier to simply say that (i.e., "The charset
guessing algorithm MUST exclude 7-bit character encodings other
than US-ASCII.  In particular, UTF-7 MUST NOT be guessed.")

....Roy

---

From: Yutaka Oiwa <[email-masked]>
Date: Wed, 23 Jan 2008 12:05:02 +0900

> than US-ASCII.  In particular, UTF-7 MUST NOT be guessed.")

 From Asian point of view, it is almost unacceptable to exclude all
ISO-2022-* charsets which use ESC as an escape character.
It is not historic, is better than 8-bit charsets in some context
(because these explicitly declare charset using ISO-2022 defined sequences),
and is ASCII upper-compatible by the above definition.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Jan 2008 10:23:21 +0100

Yutaka Oiwa wrote:
>> <http://tools.ietf.org/id/draft-ietf-httpbis-p3-payload-01.txt>, 
>> section 2.3.1, the entire forth paragraph (i.e., the last one in that 
>> section). This includes removing both the defaulting and the 
>> MUST-level requirement for labeling text/* in a charset other than 
>> ISO-8859-1.
> 
> ...

My understanding was that we want to get out of the business of 
specifying anything here, and leave it both to MIME in general and the 
specific media type registrations.

Thus, if character set sniffing is desirable for text/html, it should be 
specified in the HTML spec.

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 23 Jan 2008 13:46:00 +0100

> than US-ASCII.  In particular, UTF-7 MUST NOT be guessed.")

I'm not sure about other 7-bit character encodings, all I'm aware
of (excl. UTF-7) are harmless wrt security.  Deprecating UTF-7 for
use agents using HTTP (among other protocols) is IMO not the job
of HTTPbis.  It is also not the job of "net-utf8" as proposed on
the "apps" list recently:

<http://permalink.gmane.org/gmane.ietf.apps-discuss/946>

IMO it's generally a good idea to deprecate UTF-7 and Unicode-1-1,
and as far as I know one of the authors (Mark) and other experts
(Addison) would also support to deprecate UTF-7.  How about that
"general" solution ?

 Frank

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 23 Jan 2008 16:56:19 +0100

> (Addison) would also support to deprecate UTF-7.  How about that

What does deprecate mean? If support for UTF-7 can't be removed than  
deprecating it will hardly matter. (I'm not sure whether support can or  
can not be removed, but I'd expect there to be content to rely on it.)  
Roy's suggestion of not sniffing for it seems like better advice to  
implementors than a notion of it being deprecated.

-- 
Anne van Kesteren
<http://annevankesteren.nl/>
<http://www.opera.com/>

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Jan 2008 17:59:13 +0100

> implementors than a notion of it being deprecated.

Understood and agreed -- but where does that advice belong into? HTML5 
or HTTPbis? I would have thought the former...

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 23 Jan 2008 18:02:43 +0100

Anne van Kesteren wrote:
 
> What does deprecate mean?

My best guess is what I wrote on the "apps" list, it means updating
the reference in the registry to a document explaining why UTF-7 is
generally considered as bad idea today (with limited legacy usage
in e-mail).  To be sure we could ask on the "charset" list.
 
> If support for UTF-7 can't be removed than deprecating it will
> hardly matter.

You're not forced to support all registered charsets today, do you
support say pc-multilingual-850+euro (a.k.a. cp00858) or any older
incarnations of "cp850" ?  Likely you don't, and IMO "deprecating"
UTF-7 +/- Unicode-1-1 just offers you a reference to justify your
decision to drop it from your list of supported charsets.

Likely you also don't support UTF-1, BOCU-1, SCSU, or UTF-EBCDIC,
what's special with UTF-7 ?

As far as the IETF is concerned only UTF-8 is a MUST (including
US-ASCII is a nobrainer outside of "mobileok" validators ;-), as
far as XML is concerned you also need all UTF-16s (I'm too lazy
to check what XML says about UTF-32), and anything else is your
decision.  Not covering windows-1252 would be of course odd, and
I think you need Latin-1 for HTML versions before HTML I18N, but
UTF-7 isn't required (IIRC IMAP requires its own variant, that's
not the UTF-7 we're talking about).

> Roy's suggestion of not sniffing for it seems like better advice
> to implementors than a notion of it being deprecated.

His advice could be also put in a document deprecating it.  It's
not clear what HTTP has to do with it, UTF-7 is not "better" when
you find it in documents behind ftp: or file: URLs, or is it ?

 Frank

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 23 Jan 2008 18:06:25 +0100

> or HTTPbis? I would have thought the former...

I agree that it makes the most sense for HTML5 to define how text/html  
processing works. Just like CSS 2.1 defines text/css processing (although  
I suspect that the RFC is not yet updated).

---

From: David Morris <[email-masked]>
Date: Wed, 23 Jan 2008 09:08:26 -0800 (PST)

> or HTTPbis? I would have thought the former...

It seems to me that if there is a known security exposure for applications
built on HTTP, then the httpbis document should at the minimum note the
issue and provide a reference to the details. Seems like appropriate
content for the security section.

Dave Morris

---

From: "Anne van Kesteren" <[email-masked]>
Date: Wed, 23 Jan 2008 18:12:32 +0100

> what's special with UTF-7 ?

Character encodings are sort of beyond what I invest my time in, but I'd  
expect us to support encodings that are in use (apart from UTF-32 which we  
support for no good reason) and not necessarily those that made it to a  
list.

> [...] Not covering windows-1252 would be of course odd, and

You seem to have the impression that browser treat different versions of  
HTML differently. It has been pointed out several times that this is not  
the case and I will tell you again that this is so. Browsers don't care  
about HTML versions.

I would expect support for UTF-7 to depend on existing content. Again, I  
haven't done research in this area.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Jan 2008 18:17:22 +0100

> content for the security section.

My understanding was that that security risk is not specific to content 
transported over HTTP at all -- so I'd rather not talk about it in 
*this* document.

BR, Julian

---

From: James Graham <[email-masked]>
Date: Wed, 23 Jan 2008 17:35:30 +0000

> implementors than a notion of it being deprecated.
FWIW HTML 5 says "User agents must not support the CESU-8, UTF-7, BOCU-1 and 
SCSU encodings" [1]

It has a similar constraint that authors must not use those encodings. As you 
say, this means nothing if support is required for web-compatibility.

[1] http://www.whatwg.org/specs/web-apps/current-work/#character0

-- 
"Eternity's a terrible thought. I mean, where's it all going to end?"
  -- Tom Stoppard, Rosencrantz and Guildenstern are Dead

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 23 Jan 2008 10:34:53 -0800

On Jan 23, 2008, at 9:17 AM, Julian Reschke wrote:
> My understanding was that that security risk is not specific to  
> content transported over HTTP at all -- so I'd rather not talk  
> about it in *this* document.

Because the only known way to avoid the security holes in existing
browsers that sniff UTF-7 is to add a charset parameter even when
the exact charset is not known to the server.  That is specific to
HTTP and is a known problem due to browser's ignoring the existing
requirements of HTTP that this thread intends to remove.

....Roy

---

From: Julian Reschke <[email-masked]>
Date: Wed, 23 Jan 2008 20:38:18 +0100

> requirements of HTTP that this thread intends to remove.

Hm.

1) MIME says: default for text/* is US-ASCII.
2) RFC2616 says: default for text/* is ISO-8859.
3) Browsers do content sniffing, thus they ignore both 1) and 2).

So if we remove 2), how does this change the situation WRT sniffing?

I'm not totally opposed to mentioning this, but I'd really like to 
understand how the intended change changes the situation...

BR, Julian

---

From: "Frank Ellermann" <[email-masked]>
Date: Wed, 23 Jan 2008 21:09:45 +0100

Julian Reschke wrote:
 
> I'm not totally opposed to mentioning this, but I'd really like
> to understand how the intended change changes the situation...

I think the subject is misleading:

Whatever the default is, MIME ASCII (or the dubious Latin-1), for
authors it's an excuse where they might get away without stating
explicitly within their text/* documents what it is, because the
documents really are ASCII (or Latin-1 for RFC 2616).

For clients the wannabe-default is actually no reliable info, if
they need to know what it is they check it *assuming* ASCII (or
Latin-1) until they get to a point where it's either clear what
it is, or where the assumption is apparently okay, or where they
decide that they have no clue what it is, but certainly not the
wannabe-default.

In other words I think that the UTF-7 observation is a separate
new issue unrelated to default ASCII / Latin-1 / UTF-8, because
UTF-7 could be (mis-)interpreted as subset of of these defaults.

And if that deserves a paragraph with MUSTard in the security
considerations, then this doesn't depend on what the default is.

 Frank

---

From: "Roy T. Fielding" <[email-masked]>
Date: Wed, 23 Jan 2008 14:33:22 -0800

> So if we remove 2), how does this change the situation WRT sniffing?

It doesn't.  It eventually allows for servers to be compliant with
the new requirement (yes, removing a default is setting a new
requirement because right now sniffing is non-compliant).

> understand how the intended change changes the situation...

If a server sends generated representations in any text type today
without adding a charset parameter, they will be accused of having
"XSS security holes" which are mostly just a byproduct of stupid
browsers that guess the UTF-7 charset.  That situation will
not change until the browsers remove guessing of UTF-7 (and any other
charsets of the same ilk).  In the past, servers could point to this
section and say "this blatant non-compliant behavior on the part of
the browser is responsible for any loss as a result of the content",
whereas removing that paragraph leaves responsibility in doubt.

Therefore, if the change is made as suggested, a corresponding
entry in the Security Considerations must define this problem as
a vulnerability in browsers and exclude such charsets from the
guessing algorithm.  If not, then we are better served by not
changing the paragraph because then at least the compliant behavior
is safe (even if no browser chooses to be compliant).

....Roy

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 24 Jan 2008 11:50:21 +1300

forgive me if I'm barking up the wrong tree, but why is it the 
responsibility of HTTP (remembering what the 2nd 'T' stands for) to be 
concerned about the content of any entity being transferred? 

That would be like the postal service having to be concerned about the 
contents of envelopes, rather than just the addresses on the outside.

Security issues in these cases seem to be related to what happens to 
content once it is received by a browser.  HTTP is already out of play 
at that stage.

There are a zillion things that browsers do that aren't specified in 
HTTP.  Like rendering HTML for starters.

Apart from the content descriptions (i.e. Content-Type, charset etc) 
advertised by content providers and interpreted by content consumers, as 
long as HTTP isn't changing these, it's up to the server to provide it, 
and the client to accept it or otherwise, but the transport only has the 
responsibility to faithfully copy this data from one end of the 
communication to the other.  It can't be required to police it.

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 24 Jan 2008 17:57:12 +1100

On 24/01/2008, at 9:50 AM, Adrien de Croy wrote:

> responsibility of HTTP (remembering what the 2nd 'T' stands for) to  
> be concerned about the content of any entity being transferred?
> That would be like the postal service having to be concerned about  
> the contents of envelopes, rather than just the addresses on the  
> outside.

Oh, but they do, at least last time I was in the US...

> content once it is received by a browser.  HTTP is already out of  
> play at that stage.
>
> advertised by content providers and interpreted by content  
> consumers, as long as HTTP isn't changing these, it's up to the  
> server to provide it, and the client to accept it or otherwise, but  
> the transport only has the responsibility to faithfully copy this  
> data from one end of the communication to the other.  It can't be  
> required to police it.

Are you saying that you're against adding a sentence or two to  
Security Considerations about this issue? So far, I've seen pretty  
strong support for doing so from a variety of people.

Cheers,

---

From: Adrien de Croy <[email-masked]>
Date: Thu, 24 Jan 2008 21:21:13 +1300

> strong support for doing so from a variety of people.

I'm not against warning people about security issues, however I think 
HTTP needs to take a stand with respect to where the boundaries of its 
responsibility lie.  E.g. by making comment on security issues unrelated 
to the protocol itself, it should take pains to avoid being seen to be 
accepting any responsibility for what happens to content whilst being 
processed by a user agent.

Such warnings would perhaps be best put in another place, i.e guidelines 
for content processing of UAs.  Such a document certainly I would 
support being referenced by the HTTP RFC.

If this is the only issue, then it could be expedient to cover it in 
HTTP RFC, but I'd make it clear that it's nothing to do with HTTP 
itself.  If it's not the only issue then where to draw the line?

My POV is one of a proxy developer.  For UA or server developers, they 
are forced anyway to deal with these issues as part of other non HTTP 
(protocol) functions in their products.  UAs already heuristically 
surmise content types and character sets even regardless of (what should 
be MIME-) headers (in many cases done to cope with poorly configured 
servers).  Servers likewise have a human somewhere telling them or 
setting rules for content type and possibly char set for the resources 
they serve.

Proxies don't have this luxury.  They can only realistically forward 
entity headers untouched.  I'd hate this charset issue to become a 
requirement for proxies to sanitize entity headers.

Regards

Adrien

---

From: "Larry Masinter" <[email-masked]>
Date: Thu, 24 Jan 2008 07:14:36 -0800

What do you think about a separate document for "HTTP for Web Browsers and
Servers" that talks about charset and EOL for HTML and text types, warnings,
login dialogs, etc.?

Target BCP instead of (Draft) Standard.

It would be a way of removing some of these topics from the HTTP standard
while keeping them as IETF official specifications.

Larry

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of Adrien de Croy
> Sent: Thursday, January 24, 2008 12:21 AM
> To: Mark Nottingham
> Cc: HTTP Working Group
> Subject: Re: security impact of dropping charset default
> 
> 
> 
> Mark Nottingham wrote:
> >

---

From: Yves Lafon <[email-masked]>
Date: Thu, 24 Jan 2008 11:30:28 -0500 (EST)

On Thu, 24 Jan 2008, Mark Nottingham wrote:

> Are you saying that you're against adding a sentence or two to Security 
> Considerations about this issue? So far, I've seen pretty strong support for 
> doing so from a variety of people.

It would be a nice addition to describe the issue in general, not only for 
HTML content, when UA are into the "content sniffing" business. It fits 
well in the security section of HTTP.

The specific case of HTML needs also to be explained, but has its place in 
a document reserved for browser implementors. I am pretty sure there is 
already one that can be extended that way.

-- 
Baroula que barouleras, au tiéu toujou t'entourneras.

         ~~Yves

---

From: Geoffrey Sneddon <[email-masked]>
Date: Thu, 24 Jan 2008 16:32:16 +0000

> Servers" that talks about charset and EOL for HTML and text types,  

For HTML I feel there's little point as that work is already going on  
within the HTML WG, though a general thing for when there aren't more  
specific standards would be nice.

--
Geoffrey Sneddon
<http://gsnedders.com/>

---

From: "Larry Masinter" <[email-masked]>
Date: Fri, 25 Jan 2008 04:28:57 -0800

> specific standards would be nice.

Yes, the IETF specification for Web Browers and Servers could reference
The HTML working group document, at least for those topics covered.

 Larry

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 30 Jan 2008 20:48:04 +1100

I don't hear anyone arguing that this should generate extra  
requirements for proxies (and I have trouble believing that any  
interop problem that's limited to the consumers would be best fixed by  
intermediaries).

Let's call this issue closed as outlined, including the addition of  
some media type-generic text in security considerations regarding  
content sniffing, on the grounds that it's a problem partially caused  
by HTTP's previous meddling in this area. If later in the process* we  
find that we have a surfeit of implementation advice, we can explore  
Larry's idea of a BCP implementation guide.

Make sense?

Cheers,

* That may happen at any time; it's just a separate discussion.

On 24/01/2008, at 7:21 PM, Adrien de Croy wrote:

> I'm not against warning people about security issues, however I  
> think HTTP needs to take a stand with respect to where the  
> boundaries of its responsibility lie.  E.g. by making comment on  
> security issues unrelated to the protocol itself, it should take  
> pains to avoid being seen to be accepting any responsibility for  
> what happens to content whilst being processed by a user agent.
>
> Such warnings would perhaps be best put in another place, i.e  
> guidelines for content processing of UAs.  Such a document certainly  
> I would support being referenced by the HTTP RFC.
>
> My POV is one of a proxy developer.  For UA or server developers,  
> they are forced anyway to deal with these issues as part of other  
> non HTTP (protocol) functions in their products.  UAs already  
> heuristically surmise content types and character sets even  
> regardless of (what should be MIME-) headers (in many cases done to  
> cope with poorly configured servers).  Servers likewise have a human  
> somewhere telling them or setting rules for content type and  
> possibly char set for the resources they serve.
>

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 30 Jan 2008 20:49:34 +1100

Do you think that it would be able to take a substantial amount of the  
material in 2616 into a separate document?

My concern is that writing such a beast from scratch is a pretty big  
undertaking, when we already have BIS and the security property docs  
on our plate. Are you volunteering? :)

On 25/01/2008, at 2:14 AM, Larry Masinter wrote:

>> guidelines
>> they