Subject: PaceMustBeWellFormed

From: Mark Pilgrim <[email-masked]>
Date: Tue, 6 Jul 2004 14:42:50 -0400

http://intertwingly.net/wiki/pie/PaceMustBeWellFormed

== Abstract ==

[MarkPilgrim] Clarify the rules for determining well-formedness of an
Atom feed served over HTTP, covering issues of character encoding and
MIME types.  Also, client requirements for handling non-well-formed
feeds.

== Status ==

Open

== Rationale ==

RFC 3023 defines rules for determining the character encoding of a
feed (or any other XML document served over HTTP).  The default
configuration for most web servers is to serve ".xml" files as
"text/xml" with no charset parameter.  According to RFC 3023, all of
these feeds MUST be parsed as "us-ascii".  This leaves
UnprivilegedUsers without a way to publish Atom feeds in any other
encoding except us-ascii.  Furthermore, the Atom-enabled applications
that the UnprivilegedUsers are running may not even have enough
privileges to determine that they are, in fact, unprivileged in this
way.

Therefore, this proposal recommends that all Atom-enabled publishers
"assume the worst" and only emit ASCII-compatible XML.

== Proposal ==

Insert as section 6:

6. Client processing requirements

Atom feeds served over HTTP MUST be well-formed XML 1.0, as defined in
Section 2.1 of the XML specification
<http://www.w3.org/TR/REC-xml/#sec-well-formed>.  Furthermore, the
concept of XML well-formedness relies on first determining the
character encoding of the XML document.  RFC 3023 defines how to
determine the character encoding of XML documents served over HTTP.

6.1 Determining the character encoding of an Atom feed

The rules for determining the character encoding of an Atom feed are
the same as determining the character encoding of any XML document
served over HTTP.  The rules are wholely defined by RFC 3023, but they
are summarized here because there has been widespread confusion over
how RFC 3023 should be interpreted:

 1. When serving an Atom feed, it is RECOMMENDED that publishers
include the charset parameter along with the media type in the
Content-type HTTP header.  If the charset parameter is present,
clients MUST parse the Atom feed in that charset, ignoring any charset
declared in the encoding attribute of the XML declaration.
 1. Publishers SHOULD serve all Atom feeds with the media type
"application/atom+xml" (registered in Section 8 of this document). 
Clients MUST treat "application/atom+xml" as "application/xml" and
determine the character encoding as per RFC 3023 or its successor.
 1. If a publisher wishes to serve an Atom feed over HTTP, but for
some reason they are unable to use the "application/atom+xml" media
type, the publisher SHOULD use "application/xml", and clients MUST
determine the character encoding as per RFC 3023 or its successor.
 1. If a publisher is unable to serve their Atom feed with a
Content-Type of "application/atom+xml" or "application/xml", they MAY
use "text/xml".  According to RFC 3023, XML documents served as
"text/xml" with no charset parameter have a character encoding of
"us-ascii".
  1. When serving an Atom feed as "text/xml", publishers MUST escape
all non-US-ASCII characters as
[http://www.w3.org/TR/REC-xml/#sec-references character references]. 
For example, '&#xf8;' for the character 'ø'.
  1. When retrieving an Atom feed served with a Content-type of
"text/xml", clients MUST parse it with a "us-ascii" encoding.  If such
a feed contains non-US-ASCII characters, and clients MUST reject it as
non-well-formed.
 1. Publishers MUST NOT serve Atom feeds with a media type other than
"application/atom+xml" (registered in this Section 8 of document) or
one of the XML media types defined in RFC 3023 or its successor.  In
particular, "text/plain" is never an appropriate media type for an
Atom feed.  When retrieving an Atom feed served with a non-XML media
type, clients MUST reject it as non-well-formed.

6.2 Handling well-formedness errors

After determining the character encoding by the rules in section 6.1
of this document, clients MUST use a conforming XML parser to parse an
Atom feed.  In particular, clients MUST stop processing at the first
well-formedness error, although they MAY display any information they
have parsed before the first well-formedness error.

Here is a non-comprehensive list of things clients have been known to
do after encountering a well-formedness error, which this document
specifically prohibits:

 * Clients MUST NOT reparse the feed in any other character encoding.
 * Clients MUST NOT "tidy" the feed to attempt to fix mismatched start
and end tags.
 * Clients MUST NOT guess at the meaning of undefined entities,
including entities defined in the HTML specification.

== Impacts ==

This proposal has significant impact for both publishers and clients. 
Publishers must be aware of their web server configuration and ensure
that Atom feeds are served with the appropriate media type, or, if
that is not possible, that all non-US-ASCII characters are properly
escaped.  Clients must ensure that they properly implement RFC 3023,
which few tools currently support.

-----

CategoryProposals

-- 
Cheers,
-Mark

---

From: Danny Ayers <[email-masked]>
Date: Wed, 07 Jul 2004 10:23:46 +0200

Generally looks very good, thanks, though I'm a little puzzled by the 
following line:

>"assume the worst" and only emit ASCII-compatible XML.

I don't understand what the "worst" is here. Consumers, even those 
running server-side, should be able to deal with non-ASCII XML, I don't 
see why the publisher should have to 'dumb down' if they have the 
capability for producing unescaped Unicode as "application/atom+xml". 
The worst is that *some* publishers lack privileges, I don't see why 
that should impact all publishers.

Cheers,
Danny.

-- 

Raw
http://dannyayers.com

---

From: Mark Pilgrim <[email-masked]>
Date: Wed, 7 Jul 2004 08:12:55 -0400

> >"assume the worst" and only emit ASCII-compatible XML.

I think I explained this better in the "Impacts" section.  Some
publishing programs (such as Movable Type) don't always know *and
can't even discover* whether the feeds they generate will be served as
"text/xml" or "application/xml"-equivalent.  Such programs MUST
generate ASCII-compatible output, otherwise they risk running afoul of
RFC 3023.

This is Martin's worst nightmare [1], that specs would start
recommending ASCII-compatible encoding and eventually everyone would
do that without knowing why.  And he's right, it's the old "cat tied
up during meditation practice" problem [2].  But I don't see any
alternative.  Programs like Movable Type are

(a) our early adopters
(b) publishing static files that are later served by the underlying web server
(c) able to run in extremely unprivileged environments where the end
user is unable to override the default server configuration
(d) generally run on default Apache installs where "text/xml" is still
the default (yes, I know this was changed late last year)

These people *can use Atom*, and they *will use Atom*, and Movable
Type *already supports Atom*.  This proposal spells out the steps they
MUST take to ensure that they are always doing it correctly, even in
unprivileged environments.

Atom is XML, generally served over HTTP.  That is ruled by the
well-formedness provisions and draconian error handling of the XML
specification, and RFC 3023.  I don't make the rules, and (apparently)
I can't change them either.  Failing that, the best I can do is
provide guidance so people can live within the rules.

[1] http://www.imc.org/atom-syntax/mail-archive/msg05928.html
[2] http://www.rider.edu/~suler/zenstory/ritualcat.html

-- 
Cheers,
-Mark

---

From: Danny Ayers <[email-masked]>
Date: Wed, 07 Jul 2004 15:30:48 +0200

>generate ASCII-compatible output, otherwise they risk running afoul of

Right, thanks, I see what you mean now. Still seems a little 
uncomfortable, likely to make some publishers choose the lowest common 
denominator when they don't have to. But is there any significant cost 
in that? Offhand I can't think of any, but this is such a morass.

I wonder if it could be worded in such a way as to encourage 
ASCII-compatible output as the default, but not the *only* possible 
output, and maybe changing the "publishers" in the sentence to 
"developers"?

btw, thanks for the tip [2], I am now on the path to greater 
enlightenment [3]...

Cheers,
Danny.

>[2] http://www.rider.edu/~suler/zenstory/ritualcat.html

[3] http://dannyayers.com/2004/07/frumpy-tied.jpg

-- 

Raw
http://dannyayers.com

---

From: Antone Roundy <[email-masked]>
Date: Wed, 7 Jul 2004 07:40:29 -0600

> "assume the worst" and only emit ASCII-compatible XML.

Therefore, this proposal recommends that all UnpriviledgedUsers "assume 
the worst" and only emit ASCII-compatible XML.

> 4. If a publisher is unable to serve their Atom feed with a 

4. If a publisher is unable to serve their Atom feed with a 
Content-Type of "application/atom+xml" or "application/xml", they MAY 
use "text/xml". The published SHOULD specify a charset parameter to 
indicate the character encoding used by the feed.

5. If the publisher uses "text/xml" and is unable to specify the 
charset parameter, then according to RFC 3023, the a character encoding 
is "us-ascii".

=====

It's unlikely that anyone who can't control the Content-Type won't be 
able to specify the charset parameter, so #4 may never be done, but it 
does improve clarity, and it does specify an acceptable, if 
unpreferred, Content-Type.

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 9 Jul 2004 22:51:47 -0700

-1

This is great advice, but it restates a number of requirements in other 
specifications that are referred to by Atom. That's not good 
specification practice, as it very often results in conflicts of 
interpretation. Furthermore, this places a large number of requirements 
on "clients" that are difficult to test and constrain their behaviours 
in frankly unrealistic ways. Finally, many of the other requirements 
are made upon "publishers" -- an audience that almost assuredly will 
never read the Atom specification.

I strongly suggest this be moved to a primer, another non-normative 
document, or to WG-external supporting documentation.

--
Mark Nottingham     http://www.mnot.net/

---

From: Sam Ruby <[email-masked]>
Date: Sat, 10 Jul 2004 08:33:54 -0400

Mark Nottingham wrote:
> in frankly unrealistic ways. Finally, many of the other requirements are 
> made upon "publishers" -- an audience that almost assuredly will never 
> read the Atom specification.
> 

Lots of good specifications have informative notes when there are 
relevant specifications that have specific requirements that need to be 
aware of.

I'm sure that many of these non-normative notes started out lenghty 
before being boiled down to their essentials by repeated scrubbing.

I'm quite OK with a non-normative note, and brevity whenever it does not 
affect clarity is always a virtue, but I am not OK with leaving the 
effort of finding the relevant specifications as an exercise left for 
the student.  In producing the feedvalidator I have been continually 
frustrated by vague specifications and contradictory and 
non-authoritative clarifications.

So: I strongly encourage that we continue to work towards the placement 
of SOMETHING that attempts to address the problem this PACE proports to 
solve in one of the documents produced by this workgroup.

- Sam Ruby

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 10 Jul 2004 10:01:48 -0700

The relevant question is where to draw the line. I suspect we're going 
to come up with a lot of other implementation advice, and if we do, 
putting it in the spec is going to make it harder to find the normative 
parts of the spec. This is especially true with an RFC, where the 
format is plain text.

So, I agree with Sam; if we put this information in the spec, it should 
be much more compact, non-normative, and in an appendix or otherwise 
clearly delineated from normative content. However, if we collect more 
than just this, we should create a separate document -- a primer, if 
you like -- and move such content to it.

Cheers,

On Jul 10, 2004, at 5:33 AM, Sam Ruby wrote:

> Mark Nottingham wrote:
>> -1
>> This is great advice, but it restates a number of requirements in 
>> other specifications that are referred to by Atom. That's not good 
>> interpretation. Furthermore, this places a large number of 
>> requirements on "clients" that are difficult to test and constrain 
>> their behaviours in frankly unrealistic ways. Finally, many of the 
>> other requirements are made upon "publishers" -- an audience that 
>> almost assuredly will never read the Atom specification.
> relevant specifications that have specific requirements that need to 
> be aware of.
>
> I'm quite OK with a non-normative note, and brevity whenever it does 
> not affect clarity is always a virtue, but I am not OK with leaving 
> the effort of finding the relevant specifications as an exercise left 
> for the student.  In producing the feedvalidator I have been 
> continually frustrated by vague specifications and contradictory and 
> So: I strongly encourage that we continue to work towards the 
> placement of SOMETHING that attempts to address the problem this PACE 
> proports to solve in one of the documents produced by this workgroup.
>
> - Sam Ruby
>

---

From: Sam Ruby <[email-masked]>
Date: Wed, 14 Jul 2004 13:19:38 -0400

(6.1 1) I disagree with the recommendation that charset be provided.  In 
fact, one of the primary benefits I see to recommending application/* 
instead of text/* is that one can omit the charset and obtain a sensible 
result.

(6.1 1) I would, howevever, be supportive of a statement that if a 
charset were provided with any application/* MIME type, THEN it MUST 
match what the encoding that would have been determined had the charset 
been omitted.

(6.1 4) it is unclear that the two subbullets of this item only apply IF 
the charset parameter is omitted.  This should be clarified.

(6.2) I am uncomfortable with the lists of prohibitions.  I would be 
much more comfortable with what I see presented in section 4 of the TAG 
findings on Authoritative Metadata[1].

- Sam Ruby

[1] http://www.w3.org/2001/tag/doc/mime-respect.html#inconsistency

---

From: Mark Pilgrim <[email-masked]>
Date: Wed, 14 Jul 2004 14:09:32 -0400

> instead of text/* is that one can omit the charset and obtain a sensible

http://www.w3.org/International/questions/qa-encoding-alts.html

> match what the encoding that would have been determined had the charset

That is not an RFC-2119-compliant use of the word "MUST".  Conflicting
metadata declarations are not an interoperability problem, because
there are clear precedence rules governing which one wins.  (The
charset in the HTTP headers always wins.)

-- 
Cheers,
-Mark

---

From: Sam Ruby <[email-masked]>
Date: Wed, 14 Jul 2004 14:23:30 -0400

> charset in the HTTP headers always wins.)

It certainly is valid for an RFC produced by the AtomPub working group 
to constrain the permissable values of the charset parameter of the 
Content-Type header MUST be when the application/atom+xml MIME type is 
used in conjunction with either the AtomPub format or protocol.

I would also suggest that we coordinate with RFC 3023 to promote the use 
of similar language in that specification when application/xml is used.

- Sam Ruby

---

From: Mark Pilgrim <[email-masked]>
Date: Wed, 14 Jul 2004 14:25:24 -0400

> used in conjunction with either the AtomPub format or protocol.

Not if you're normatively referencing RFC 2119, it's not.

> of similar language in that specification when application/xml is used.

That is well beyond our charter.

-- 
Cheers,
-Mark

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Wed, 14 Jul 2004 12:56:40 -0700

>http://www.w3.org/International/questions/qa-encoding-alts.html

Not sure what this response means. That page has ??? and XXX in a few 
places, and is quite clearly not finshed (or even that clear in 
places).

>charset in the HTTP headers always wins.)

Correct. If the precedence is given by a MUST, and if what you are 
talking about is not first in the precedence, then a MUST is 
incorrect according to a careful (and possibly legalistic) reading 
RFC 2119.

>That is well beyond our charter.

Fully disagree. RFC 3023 is a pre-existing standard in the IETF; we 
should follow it to the best of our ability, and say why we aren't 
when we don't.

--Paul Hoffman, Director
--Internet Mail Consortium

---

From: Sam Ruby <[email-masked]>
Date: Wed, 14 Jul 2004 16:58:20 -0400

Paul Hoffman / IMC wrote:

> talking about is not first in the precedence, then a MUST is incorrect 
> according to a careful (and possibly legalistic) reading RFC 2119.

What spec defines the precedence rules for a MIME type that we have yet 
to register (i.e., application/atom+xml)?

- Sam Ruby

---

From: Mark Pilgrim <[email-masked]>
Date: Wed, 14 Jul 2004 17:19:34 -0400

> to register (i.e., application/atom+xml)?

Section 8 of http://www.ietf.org/internet-drafts/draft-ietf-atompub-format-00.txt
states:

"""Optional parameters:
      "charset": This parameter has identical semantics to the charset
         parameter of the "application/xml" media type as specified in
         RFC 3023."""

-- 
Cheers,
-Mark

---

From: Mark Pilgrim <[email-masked]>
Date: Wed, 14 Jul 2004 17:27:02 -0400

> places, and is quite clearly not finshed (or even that clear in

True.  Just about the only part they've decided on is the part that
backs up my argument:

"""Since the HTTP Content-Type header has precedence, and is also the
easiest information to retrieve (user-agents do not have to parse the
resource to get it), it is almost always the preferred way to provide
the character encoding for an (X)HTML document"""

I submit that the same is true of an Atom document.  It has multiple
ways to specify the encoding (in-band and out-of-band), and the HTTP
Content-Type header takes precedence, therefore it should be the
preferred way to provide the character encoding for an Atom document.

The same unfinished document has a remarkably complete discussion of
when this is not possible:

"""However, in at least two cases, this is simply not possible:

    * The document author does not have any way to configure the
server to send the proper HTTP Content-Type header
    * The document is not served via HTTP.
"""

These cases also apply equally well to Atom documents.

I admit the link was dropped in without explanation, but I'm not sure
what other conclusion you could draw from that document.  Are you
suggesting that Atom documents are so different from XHTML documents
that a different precendence order should apply, or that a different
practice should be considered "best practice"?

If my wrestling with RFC 3023 has taught me anything, it is that HTTP
has primacy.  If you want to serve Atom feeds over BEEP, then
obviously different rules will apply.  Everyone I know serves them
exclusively over HTTP.

-- 
Cheers,
-Mark

---

From: Sam Ruby <[email-masked]>
Date: Wed, 14 Jul 2004 18:17:18 -0400

Mark Pilgrim wrote:

>> What spec defines the precedence rules for a MIME type that we have
>> yet to register (i.e., application/atom+xml)?
> 
> Section 8 of
> http://www.ietf.org/internet-drafts/draft-ietf-atompub-format-00.txt 
> states:
> 
> """Optional parameters: "charset": This parameter has identical
> semantics to the charset parameter of the "application/xml" media
> type as specified in RFC 3023."""

It seems to me that THIS is the IETF work group has authority to issue 
new revisions of that draft specification.  Since this is under our 
control, surely we can construct language using the keyword MUST in a 
RFC 2119 compliant manner which covers this situation.

What am I missing?

- Sam Ruby

---

From: Paul Hoffman / IMC <[email-masked]>
Date: Wed, 14 Jul 2004 15:43:14 -0700

>preferred way to provide the character encoding for an Atom document.

Agree. But we *have* to say the explicitly in the document if we 
expect anyone to interoperate.

>what other conclusion you could draw from that document.

That it is something that the W3C is thinking about but haven't come 
to a firm conclusion on.

>   Are you

Nope. I am (now) saying explicitly that we have to lay out our rules 
in the spec.

>has primacy.

Sounds good to me. How do others feel?

--Paul Hoffman, Director

---

From: Tim Bray <[email-masked]>
Date: Wed, 14 Jul 2004 16:12:38 -0700

> Sounds good to me. How do others feel?

I feel very unhappy.  Mark produced excellent statistics demonstrating 
powerfully that (in the case of RSS) a very high proportion of websites 
cheerfully ignore 3023, and that, as a consequence of this, a high 
proportion of RSS clients cheerfully ignore 3023 since if they followed 
it, they would have to throw otherwise-OK data away.   Specifically, 
the problem is that 3023 blesses the use of text/xml with no charset 
(which was the Apache default for *.xml for a long time) which in 
nearly every case is wrong, and thus, if you follow the rest of 3023, 
will force software to throw the data on the ground.

Mark has made two intelligent, coherent suggestions for how we might 
deal with this:

1. In the case where the data is delivered as text/html with no 
charset, ignore what RFC3023 says and ascertain if this is a 
well-formed Atom instance using the (very robust) self-describing 
nature of XML as regards character encodings.
2. Because of the broken-ness in 3023 and Apache, always serve 
everything as US-ASCII by using  numeric character references for all 
non-ASCII characters.

Mark subsequently stated he wants to withdraw suggestion #1 but I'm 
dissenting on that, it's a reasonable proposal that we should look at.  
There may be other proposals worthy of our consideration.

In any case, saying "HTTP has primacy" sounds good but should cause 
extreme nervousness in the case that (1) the relevant media-type RFC is 
broken and (2) the media-type is demonstrably being used incorrectly in 
a high proportion of cases.

And I *guarantee* that a high proportion of implementors will, if we 
foolishly say, "follow the 3023 rules" cheerfully ignore us if it will 
result in them having to discard apparently valid Atom feeds.

I do not think that this is anyone else's problem.  It's our problem.  
-Tim

---

From: Sam Ruby <[email-masked]>
Date: Wed, 14 Jul 2004 19:29:56 -0400

> Sounds good to me. How do others feel?

I don't think that is framing the question correctly.  As an example, I 
presume that we are free to say that in order to produce an Atom 
compliant application, the following header MUST be present:

   Content-Type: application/atom+xml; charset=utf-8

In other words, we are free to specify what content-types are valid, and 
we are free to specify what charsets are valid.

Now, I am not going to propose that we limit charset to utf-8.  Instead, 
I am simply saying that we should treat as an error the case where the 
charset in the HTTP header and the encoding that RFC 3023 would have 
determined to be in effect in the absence of a charset parameter are 
different.

If it isn't valid for them to be different, the question of precedence 
never comes up.

I presume that we can control the definition for mime types that we have 
yet to define.  And while we can work with the authors of RFC 3023, I 
don't have any hope that they will change the definition for text/xml, 
but there just *might* be wiggle room to consider a change in 
application/xml.

Mark, does that seem OK, or do you still want to specify the precedence?

- Sam Ruby

---

From: Martin Duerst <[email-masked]>
Date: Thu, 15 Jul 2004 08:00:16 +0900

>>>match what the encoding that would have been determined had the charset

This does not make sense. Assume you blog is generated by some
script, or comes out of a database. The scripting language or
database treats everything as characters, it doesn't have a clue
what encoding will be used when the stuff is served as HTTP.
Then there is a last step that does the encoding, and puts
on a HTTP header.

>It certainly is valid for an RFC produced by the AtomPub working group to 
>constrain the permissable values of the charset parameter of the 

It is possible to do that, but it may be highly preferable not to
create exceptions to the general rule for application/...+xml.

>of similar language in that specification when application/xml is used.

There may be a new draft out by now. There is a mailing list
specifically for discussing RFC 3023 and its successor. It's
ietf-xml-mime.imc.org. (List-Archive:
http://www.imc.org/ietf-xml-mime/mail-archive/,
List-Subscribe: mailto:[email-masked]?body=subscribe).

I suggest that people interested in that discussion join that list.

Regards,    Martin.
Content-Type: text/plain; charset="us-ascii"
X-UIDL: 526780e3b94f30cec999b6d288ca074b

---

From: Walter Underwood <[email-masked]>
Date: Wed, 14 Jul 2004 21:38:01 -0700

> If my wrestling with RFC 3023 has taught me anything, it is that HTTP

That is true. The spec also has no way to handle the following

>       server to send the proper HTTP Content-Type header

This is a part of the HTTP spec which is simply not practiced.
Clearly, document authors need to specify encodings, and most
servers do not provide an easy way to do that.

Atom can specify that they must control it, but the spec won't
change how the web really works.

This is like the ROBOTS meta tag compared to the robots.txt file,
if that analogy makes sense to anyone. Authors are guaranteed to
have control over their own documents. They are not guaranteed
to have control over anything else.

Has anyone ever deployed a transcoding server? I've never seen
on in real use.

wunder
--
Walter Underwood
Principal Architect, Verity

---

From: Julian Reschke <[email-masked]>
Date: Thu, 15 Jul 2004 08:15:33 +0200

Sam Ruby wrote:
> ...
> ...

What kind of change are you looking for for application/xml?

Best regards, Julian

-- 
<green/>bytes GmbH -- http://www.greenbytes.de -- tel:+492512807760

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jul 2004 07:10:32 -0400

Julian Reschke wrote:
> 
> Sam Ruby wrote:
> 
>> ...
>> I presume that we can control the definition for mime types that we 
>> have yet to define.  And while we can work with the authors of RFC 
>> 3023, I don't have any hope that they will change the definition for 
>> text/xml, but there just *might* be wiggle room to consider a change 
>> in application/xml.
>> ...
> 

Consider the following:

   Content-type: application/xml; charset=iso-8859-7

   <?xml version="1.0" encoding="iso-8859-1"?>
   <foo>Iñtërnâtiônàlizætiøn</foo>

Until now, the discussion appears to have been framed in terms of 
precedence rules: which charset/encoding "wins"?  The tradition is to 
say that the above document is well-formed (and possibly even valid) and 
that no error or even warning is proscribed.

First tangent: the tradition is also to defer to HTTP as being the most 
authoritative meta-data in this situation.  The reason appears to be 
based on the promise given to folks that enabled them to write 
transcoders dealing with content-types of "text/*".  As people appear to 
have deployed solutions based on this promise, it is a promise that is 
rather difficult to retract.  I fully understand and appreciate that.

Back to application/xml.  As near as I can tell, no such promise has 
been made for application/xml.  In fact, RFC 3023 [1] is rather clear: 
when the charset parameter is not present, "An XML-unaware MIME 
processor SHOULD make no assumptions about the charset of the XML MIME 
entity."; and RFC 3023 only proscribes the use of 
content-transfer-encoding (a completely transparent pair of 
transformations) when dealing with situations such as 16 bit encodings 
transported over 7-bit transports.

Back to the example above.  While the current specs may say it is 
well-formed, and even proscribe how it should be interpreted, I see a 
document that is seriously confused.  And I see the relevant draconian 
provisions of the XML specification tossing other documents which have 
committed significantly lesser infractions involving so-called smart 
quotes out on their ear.

So, given that documents such as the above do not tend to appear in 
practice (can you give an example?), and have no reason to appear in 
practice (any transcoders that understand application/xml had better 
understand xml prologs), and that the likelihood that the current 
precedence rules will produce the intended result is low (given the way 
that modern web servers are configured, if the above data were served 
statically, I would tend to trust the document prolog more than the HTTP 
headers in this particular circumstance)... given all this, what is my 
recommendation?

My recommendation is that the presence of conflicting charset/encoding 
information in documents served as application/xml be treated as a 
well-formedness error.

Second tangent: such a recommendation does not affect documents served 
simply as "Content-type: application/xml", i.e., without a charset 
explicitly specified on the HTTP header.  RFC 3023[1] sections 8.9 
through 8.12 defer to the rules defined in the XML specification for 
determining the encoding in such circumstances, so inconsistency is not 
possible in this situation.

What does the above recommendation mean to users of formats such as 
application/atom+xml?  Simply that applications such as the 
feedvalidator should flag such situations (it currently does, albeit 
with a warning), and that libraries such as the feedparser should set 
bozo bits or equivalent in such situations (it doesn't currently as the 
above is currently completely legal), and that applications built on 
such libraries should not attempt to resolve such inconsistencies 
without the consent of the user[2].

- Sam Ruby

[1] http://www.ietf.org/rfc/rfc3023.txt
[2] http://www.w3.org/2001/tag/doc/mime-respect.html#inconsistency

---

From: Julian Reschke <[email-masked]>
Date: Thu, 15 Jul 2004 13:44:11 +0200

> information in documents served as application/xml be treated as a 

Thanks for the clarification, Sam. I absolutely agree here; a conflict 
indicates that something (transcoding?) has gone wrong; and it makes a 
lot of sense to throw the result away...

Best regards, Julian

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jul 2004 08:03:11 -0400

> lot of sense to throw the result away...

A second, and unrelated, suggestion that emerged from this discussion is 
that an informative note be added to section 8.5 "Text/xml with Omitted 
Charset" suggesting the use of XML numeric character references as a 
means of expressing non-ASCII characters.

- Sam Ruby

---

From: Sam Ruby <[email-masked]>
Date: Thu, 15 Jul 2004 08:43:30 -0400

> that no error or even warning is proscribed.

Bah.  s/proscribed/prescribed/g.

- Sam Ruby

---

From: Martin Duerst <[email-masked]>
Date: Fri, 16 Jul 2004 17:55:26 +0900

>Has anyone ever deployed a transcoding server? I've never seen

I have. There were a lot of transcoding proxies in the early days
of the Web in Japan. There were some transcoding servers quite
recently in Russia. You have to retreive the same document both
from an Unix machine and a Windows machine or so to find out
that the same document is served with different encodings
(and that you can rely on the HTTP header, but not on the
<meta> tag). Some mobile providers may also do transcoding,
although I don't have any confirmation on this one.

What is more important, there are well-used script languages
where it's very easy to say 'serve this document in charset X',
which implies to add this info in the HTTP header. But the
document itself isn't changed.

Regards,    Martin.

---

From: Dan Brickley <[email-masked]>
Date: Fri, 16 Jul 2004 05:35:45 -0400

* Martin Duerst <[email-masked]> [2004-07-16 17:55+0900]
> although I don't have any confirmation on this one.

http://www.burningdoor.com/steve/archives/000602.html claims that 
transcoding proxies are getting popular again. It cites Opera's 
proxy service (though I don't see direct evidence that it transcodes
charsets), Google's WAP proxy (http://www.google.com/wml), the 
Nokia One Business Server (http://www.nokia.com/nokia/0,8764,43106,00.html) and 
Feedburner (http://www.feedburner.com/fb/a/home). Hmm glancing at these
I wonder whether "transcoding server" is being used in different ways;
sometimes to refer to char encoding conversion and sometimes, more
broadly, to proxies that do other kinds of conversion (eg. HTML to WAP,
RSS to Atom, ...). 

Dan

---

From: Bill de hÓra <[email-masked]>
Date: Fri, 16 Jul 2004 11:57:16 +0100

> broadly, to proxies that do other kinds of conversion (eg. HTML to WAP,

In a past life I worked on 'transcoding' proxy. I can confirm their 
  potential use is far more than character encodings (such as 
manipulating sessions, css, javascript, java classes, soap headers, 
porn filtering, etc).

cheers
Bill

---

From: Martin Duerst <[email-masked]>
Date: Fri, 16 Jul 2004 18:01:39 +0900

At 13:44 04/07/15 +0200, Julian Reschke wrote:

>indicates that something (transcoding?) has gone wrong; and it makes a lot 
>of sense to throw the result away...

Julian - Why do you say transcoding has gone wrong? Because transcoding
from iso-8859-1 to iso-8859-7 isn't a very probable scenario anyway?
Maybe we should use a more probable example, e.g. koi8-R and iso-8859-5?

Regards,    Martin.

---

From: Martin Duerst <[email-masked]>
Date: Fri, 16 Jul 2004 18:25:04 +0900

At 07:10 04/07/15 -0400, Sam Ruby wrote:

>precedence rules: which charset/encoding "wins"?  The tradition is to say 
>that the above document is well-formed (and possibly even valid) and that 
>no error or even warning is proscribed.
>
>authoritative meta-data in this situation.  The reason appears to be based 
>on the promise given to folks that enabled them to write transcoders 
>dealing with content-types of "text/*".  As people appear to have deployed 
>solutions based on this promise, it is a promise that is rather difficult 
>to retract.  I fully understand and appreciate that.
>
>Back to application/xml.  As near as I can tell, no such promise has been 
>made for application/xml.  In fact, RFC 3023 [1] is rather clear: when the 
>charset parameter is not present, "An XML-unaware MIME processor SHOULD 
>make no assumptions about the charset of the XML MIME entity.";

I don't see the difference to text/* here. How are you going to transcode
a text/* document if you don't have any 'charset' information? You simply
can't. RFC 3023 just spells that out.

>document that is seriously confused.

Most probably. The two encodings you have choosen aren't very probable
ones for transcoding. But such a judgement should not be made by an
XML parser.

>And I see the relevant draconian provisions of the XML specification 
>tossing other documents which have committed significantly lesser 
>infractions involving so-called smart quotes out on their ear.
>
>understand xml prologs),

Transcoders usually don't look inside documents.

>and that the likelihood that the current precedence rules will produce the 
>intended result is low (given the way that modern web servers are 
>configured, if the above data were served statically, I would tend to 
>trust the document prolog more than the HTTP headers in this particular 
>circumstance)...

Well, as a human user, you can always look at the results, and
see whether the text makes sense or not.

>given all this, what is my recommendation?
>

Do you want to redefine application/xml in the Atom spec? Or were
you speaking about application/atom+xml? In the later case, would
you want application/xml and application/atom+xml to behave differently?
And how would you implement the above proposal in a off-the-shelf
parser?

>explicitly specified on the HTTP header.  RFC 3023[1] sections 8.9 through 
>8.12 defer to the rules defined in the XML specification for determining 
>the encoding in such circumstances, so inconsistency is not possible in 
>this situation.
>
>application/atom+xml?  Simply that applications such as the feedvalidator 
>should flag such situations (it currently does, albeit with a warning),

Giving a warning in such cases is a good idea. The W3C Markup Validator
does so, too.

>and that libraries such as the feedparser should set bozo bits or 
>equivalent in such situations (it doesn't currently as the above is 
>currently completely legal), and that applications built on such libraries 
>should not attempt to resolve such inconsistencies without the consent of 
>the user[2].

What about the situation (very probable with scripts) where there
is charset information in the header (and that's not UTF-8 or
UTF-16), but no 'encoding' pseudo-attribute in the xml declaration?
According to your definitions above, this would also be an error.

Regards,    Martin.

---

From: Sam Ruby <[email-masked]>
Date: Fri, 16 Jul 2004 22:36:38 -0400

> can't. RFC 3023 just spells that out.

With text/*, the default is US-ASCII.  With application/*, there is no 
default.

> According to your definitions above, this would also be an error.

iso-8859-1 text in xml documents without a BOM and without an XML prolog 
(and for that matter without a charset in the header) is probably the 
single most common encoding error I see in feeds today.

Such an error are flagged by all conforming XML parsers.

The example I picked (iso-8859-7 vs -1), while not typical, can not be 
detected by an XML parser.

- Sam Ruby

---

From: Julian Reschke <[email-masked]>
Date: Sat, 17 Jul 2004 10:56:20 +0200

> Maybe we should use a more probable example, e.g. koi8-R and iso-8859-5?

Correct, that would be a good idea.

Julian