Subject: RFC Series publishes first RFC with non-ASCII characters

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Thu, 14 Sep 2017 14:07:05 -0700

Hello all,

RFC 8187, "Indicating Character Encoding and Language for HTTP Header
Field Parameters", is the first RFC to be published with UTF-8 encoding
and include characters not in the basic ASCII character set. This
document has been, with the author's consent, patience, and support,
used to test the existing tool chain to produce RFCs to see where the
environment has difficulty in handling non-ASCII characters. The RPC is
continues this testing with the PRECIS document cluster (C326), which is
currently in AUTH48.

The RPC and the Tools Team has identified several areas that need to be
modified to support these characters. Some of those areas will
ultimately be handled with the new format tools; others have been
modified as part of the more general work to prepare for the new RFC
format. For the documents being used to test the toolchain, a
significant amount of manual processing is required to publish the
RFCs with non-ASCII characters in the final text. In order to keep
overall processing times down, leave staff enough time to test
the other tools that are being developed as part of the RFC format
project, and allow the editors time to create and/or update new
procedures as the v3 tools are released, no additional non-ASCII
documents outside of RFC 8187 and the PRECIS cluster will be published
until the new format tools are in production.

Many thanks go out to Julian Reschke and Peter Saint-Andre for their
support, the Tools Team for answering a variety of technical questions,
and to the RPC staff who worked to ensure that the RFC
publication process resulted in readable RFCs.

Heather Flanagan, RSE

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 15 Sep 2017 19:16:03 +0900

Heather Flanagan (RFC Series Editor) wrote:

> and include characters not in the basic ASCII character set.

Don't do that.

It is as stupid as allowing programming languages use non ASCII
characters.

At first, it seems to be working. However, ultimately, it makes
maintenance of code/rfc impossible, unless all the people
maintaining the code/rfc can recognize all the characters
in the code/rfc.

> This
> environment has difficulty in handling non-ASCII characters.

That's not a problem. Problem is in human capability not to be able
to recognize all the characters in the world.

Internationalized code/rfc must be written using characters recognized
by all the international people.

Even if someone write a localized code for some locale, it should be
written as:

	#define NBSP '\240'
	...
	putchar(NBSP);

not

	putchar('\240');

to ease maintenance.

							Masataka Ohta

PS

Language C using full ASCII is already a problem because ASCII back
slash character in '\240' is displayed as YEN sign of JIS X 0201
(Japanese variant of ISO 646) on almost all computers (including mine
I'm using now to write this mail) in Japan. It is not a serious
problem in Japan because all the Japanese are taught that YEN sign
is an escape character of C. But many Japanese who can use C do not
know it is actually ASCII back slash.

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Fri, 15 Sep 2017 09:05:26 -0700

> maintaining the code/rfc can recognize all the characters

Non-ASCII characters are not trivial to include in a document, at least
if you want to make sure the document is broadly readable. So, yes, this
is an area fraught with peril. However, quite a bit of time was put into
determining what guidance should be applied so that we can handle those
characters. See https://www.rfc-editor.org/rfc/rfc7997.txt.

> Even if someone write a localized code for some locale, it should be

I don't think we an ignore these characters - they are in use, and we
need to be able to represent them in a more readable fashion than just
Unicode escape sequences.

-Heather

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Fri, 15 Sep 2017 16:18:30 +0000

On 9/14/17, 5:07 PM, "rfc-interest on behalf of Heather Flanagan (RFC Series Editor)" <[email-masked] on behalf of [email-masked]> wrote:

    RFC 8187, "Indicating Character Encoding and Language for HTTP Header
    Field Parameters", is the first RFC to be published with UTF-8 encoding
    and include characters not in the basic ASCII character set. ...
        
    Heather Flanagan, RSE 

Congratulations, Heather, to you, your team, and the authors. This is a big step forward, and I know it took lots of hard work and community involvement to get here.

I think the steps the team took used the right balance of progress and restraint in this important area.

Congratulations again.

	Tony Hansen

---

From: "tom p." <[email-masked]>
Date: Fri, 15 Sep 2017 17:44:12 +0100

Interesting.

The euro symbol displays fine but depending on which utility I use to
display, I do get some blobs to indicate undisplayable characters.  Thus
in the line which starts
   into parameter valu ....
and ends
.... how to encode non-ASCII characters

the 'e' of value is not an ASCII 'e' and my attempts to cut and paste
the line into this e-mail fails after the 'u' of 'values' suggesting
that there is a line terminator in there.

Unless - of course - that letter I presume is 'e' is intended to be a
glyph outside normal ASCII with a line terminator function.

Tom Petch

---

From: Denis Ovsienko <[email-masked]>
Date: Fri, 15 Sep 2017 18:31:45 +0100

---- On Fri, 15 Sep 2017 17:44:12 +0100 tom p. <[email-masked]> wrote ---- 
 > glyph outside normal ASCII with a line terminator function. 

As far as hexdump of https://www.rfc-editor.org/rfc/rfc8187.txt goes the character you are describing is a plain ASCII "e" (0x65):

00001f20  74 69 6f 6e 0a 20 20 20  69 6e 74 6f 20 70 61 72  |tion.   into par|
00001f30  61 6d 65 74 65 72 20 76  61 6c 75 65 73 20 61 6e  |ameter values an|
00001f40  64 20 61 6c 73 6f 20 68  6f 77 20 74 6f 20 65 6e  |d also how to en|

FireFox correctly displays the .txt file in Unicode by default, this likely has to do with the UTF-8 BOM at the beginning of the file (both observations are based on the file rsync'ed from rfc-editor.org).

One thing I vaguely remember in related documents from around 2013 was that Unicode in the document text and in any person/organization names was supposed to have a 2nd ASCII version as a backup, does it still stand? Because in this RFC the Acknowledgements section does not provide a backup spelling for Martin Dürst. Not a big issue, just curious.

Other than that, this is a remarkable milestone in a big work. Hopefully document authors will remember that more [technical] power implies more responsibility.

-- 
    Denis Ovsienko

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 15 Sep 2017 19:39:39 +0200

> I think the steps the team took used the right balance of progress and restraint in this important area.

Indeed! Congratulations!

   Patrik Fältström

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 15 Sep 2017 21:20:51 +0200

On Sep 15, 2017, at 19:31, Denis Ovsienko <[email-masked]> wrote:
> 
> UTF-8 BOM at the beginning of the file

Duuh, indeed, it really has a BOM at the start.

FAIL!

This is utterly disappointing.

Grüße, Carsten

---

From: Denis Ovsienko <[email-masked]>
Date: Fri, 15 Sep 2017 21:50:08 +0100

---- On Fri, 15 Sep 2017 20:20:51 +0100 Carsten Bormann  wrote ---- 
>This is utterly disappointing. 

The presence of BOM in a UTF-8 file exactly follows Section 2 of RFC 7994 (Requirements for Plain-Text RFCs).

-- 

    Denis Ovsienko

---

From: Martin J. Dürst <[email-masked]>
Date: Sat, 16 Sep 2017 09:22:37 +0900

> environment has difficulty in handling non-ASCII characters.

Congratulations to everybody involved! Really great to see this move 
forward!

Regards,   Martin.

P.S.: Looking at the RFC, there were exactly 5 non-ASCII characters (two 
'£'s and one '€' in the examples, and two 'ü's, one in Julian's address 
and one in my name in the acks. Now if I ever need a claim to fame, I 
can claim that my name contributed to the first non-ASCII characters in 
RFCs :-).

P.P.S.: Regarding the content of that RFC, I think may already know, but 
for the record, I'll repeat here that I think it's way outdated. I 
really wish the HTTP WG would take some hints from this RFC work and 
finally get the message that straightforward UTF-8 is the way to go (we 
all knew that ages ago, anyway).

> The RPC is

---

From: Martin J. Dürst <[email-masked]>
Date: Sat, 16 Sep 2017 09:30:31 +0900

> One thing I vaguely remember in related documents from around 2013 was that Unicode in the document text and in any person/organization names was supposed to have a 2nd ASCII version as a backup, d…

I think the ASCII backup rule is for authors, because these need to be 
cross-linked with older RFCs,... For the Acks section, it's not that big 
a deal for Latin characters. Serious publications such as from the ACM 
or the IEEE since ages don't use fallbacks for Latin characters at all 
even for authors. It would be different for Greek, Chinese, Arabic,... 
names.

Regards,   Martin.

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 16 Sep 2017 11:30:31 +1000

[ note reply-to ]

> On 16 Sep 2017, at 10:22 am, Martin J. Dürst <[email-masked]> wrote:
> 
> P.P.S.: Regarding the content of that RFC, I think may already know, but for the record, I'll repeat here that I think it's way outdated. I really wish the HTTP WG would take some hints from this RFC work and finally get the message that straightforward UTF-8 is the way to go (we all knew that ages ago, anyway).

If we had a magic wand and could start over, perhaps. As it is, there is software that assumes a character set for HTTP headers, and putting UTF-8 through it would cause problems. 

Cheers,

--
Mark Nottingham   https://www.mnot.net/

---

From: "Patrik Fältström" <[email-masked]>
Date: Sat, 16 Sep 2017 06:48:17 +0200

> P.P.S.: Regarding the content of that RFC, I think may already know, but for the record, I'll repeat here that I think it's way outdated. I really wish the HTTP WG would take some hints from this R…

+1

RFC 2130 from April 1997:

> This report recommends the use of ISO 10646 as the default Coded
> Character Set, and UTF-8 as the default Character Encoding Scheme in
> the creation of new protocols or new version of old protocols which
> transmit text. These defaults do not deprecate the use of other
> character sets when and where they are needed; they are simply
> intended to provide guidance and a specification for
> interoperability.

;-)

   paf

---

From: Masataka Ohta <[email-masked]>
Date: Sat, 16 Sep 2017 16:32:56 +0900

Heather Flanagan (RFC Series Editor) wrote:

> is an area fraught with peril.

You still don't understand the problem at all.

As I wrote:

 >> That's not a problem.

Broad readability is *NOT A PROBLEM*.

> However, quite a bit of time was put into

That is a misdirected effort to solve a wrong problem.

And, as I wrote:

 > However, ultimately, it makes

You can't solve the following problem:

 > Problem is in human capability not to be able

 > by all the international people.

which means RFCs should be pure ASCII just as programming languages
use pure ASCII.

							Masataka Ohta

---

From: Masataka Ohta <[email-masked]>
Date: Sat, 16 Sep 2017 17:15:35 +0900

There was Latin1 characters included in early MIME drafts.

I told late Prof. Jon Postel, the rfc editor at that time,
that that is a violation of formatting rules and causes a lot
of problems and is against internationalization but he
answered that it was inevitable because some examples in the
drafts need to include non-ASCII characters.

So, I said Jon that MIME rfcs and there drafts can avoid
including Latin1 characters because examples in the drafts
can (and should, of course) be written with MIME format to
represent Latin1 characters by ASCII. .

As a result, MIME rfcs (1341, 1342 etc.) are published in
pure ASCII.

Since then until just recently, all the rfcs are published in
pure ASCII, though I don't know how deeply Jon understood
various problems caused by non-ASCII characters.

And, I can see no reason why rfc8187 must include non-ASCII
characters only to make rfcs a lot less internationalized.

						Masataka Ohta

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 16 Sep 2017 10:42:02 +0200

> The presence of BOM in a UTF-8 file exactly follows Section 2 of RFC 7994 (Requirements for Plain-Text RFCs).

Yes, I missed that sentence buried in that RFC at the time.

Still FAIL, still utterly disappointing.

STD0063
https://tools.ietf.org/html/rfc3629#section-6
(Look for all the sentences starting “A protocol SHOULD forbid use of U+FEFF as a signature...“.)

"Use of a BOM is neither required nor recommended for UTF-8":
http://www.unicode.org/versions/Unicode10.0.0/ch02.pdf

RFC 7994 is a massive regression here.

Grüße, Carsten

---

From: Lyndon Nerenberg <[email-masked]>
Date: Sat, 16 Sep 2017 08:05:06 -0700

> On Sep 16, 2017, at 12:32 AM, Masataka Ohta <[email-masked]> wrote:
> 

Plan 9's C compilers have supported UTF8 in the grammar for decades.  Go and Swift have since their inception.  

And then there's APL.

--lyndon

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sat, 16 Sep 2017 17:32:39 +0200

First, the important thing: this is a great step, congratulations to
the RFC Editor.

On Fri, Sep 15, 2017 at 09:50:08PM +0100,
 Denis Ovsienko <[email-masked]> wrote 

> 7994 (Requirements for Plain-Text RFCs).

But section 2 of RFC 5198 says the opposite and, IMHO, it is right.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sat, 16 Sep 2017 17:45:35 +0200

On Sat, Sep 16, 2017 at 04:32:56PM +0900,
 Masataka Ohta <[email-masked]> wrote 

> which means RFCs should be pure ASCII just as programming languages

Besides the good counter-examples mentioned by Lyndon Nerenberg, I
suggest that you watch this presentation at IETF 95
<https://www.ietf.org/proceedings/95/hrpc.html>
<https://www.youtube.com/watch?v=FnOnZFaK0bQ> (starts at 11:15)

---

From: Denis Ovsienko <[email-masked]>
Date: Sat, 16 Sep 2017 20:58:37 +0100

---- On Sat, 16 Sep 2017 16:32:39 +0100 Stephane Bortzmeyer  wrote ---- 
>But section 2 of RFC 5198 says the opposite and, IMHO, it is right. 

Well, it looks like "Unicode Format for Network Interchange" was not the intended RFC publication format, otherwise RFCs would have DOS newlines (point 2 of the same section).

-- 

    Denis Ovsienko

---

From: Masataka Ohta <[email-masked]>
Date: Sun, 17 Sep 2017 04:59:44 +0900

Lyndon Nerenberg wrote:

> Plan 9's C compilers have supported UTF8 in the grammar for decades.

 >>> environment has difficulty in handling non-ASCII characters.

lack of implementations to support a bad idea is not a problem.

As such, providing such implementations is not a solution.

With reasons, C, as standardized by ISO, use pure ASCII.

If a really international team of programmers must tightly
cooperate using Plan 9 C compiler, code must be written using
limited set of characters recognized by all the members.

						Masataka Ohta

---

From: Benjamin Kaduk <[email-masked]>
Date: Sat, 16 Sep 2017 15:04:41 -0500

Ohta-san,

> You still don't understand the problem at all.

If that is the case (which I neither confirm nor deny), then you should
provide a more detailed explanation of the problem, rather than
just repeating your previous statement.

> limited set of characters recognized by all the members.

Furthermore, you seem to be repeatedly making assertions of how
things must be, without providing any supporting justifciation
for those assertions.  This does not help us have a productive
discussion; it seems rather likely to instead lead to frustration
on all sides, including the passive readership of this list.

-Ben

---

From: Masataka Ohta <[email-masked]>
Date: Sun, 17 Sep 2017 05:21:48 +0900

> I suggest that you watch this presentation at IETF 95

Arabic text in presentation slides, which is illegible for
most participants, is a good example that we must use
ASCII/English only for IETF discussion involving
really international participants, if commonly recognized
characters and languages by them are ASCII/English only.

That's the internationalization.

Still, it is a good idea that a local chapter of ISOC uses
local characters and a local language for local discussions on
IETF activities, which was what I did with ISOC Japan chapter.

That is good localization but is not internationalization.

So?

					Masataka Ohta

---

From: Masataka Ohta <[email-masked]>
Date: Sun, 17 Sep 2017 05:32:02 +0900

>>>> which means RFCs should be pure ASCII just as programming languages

The statement above is on programming language specification.

>>> Plan 9's C compilers have supported UTF8 in the grammar for decades.

The statement above is on compilers.

> just repeating your previous statement.

That is useless.

If you can't understand simple examples, you can't understand
detailed explanations.

And,

is a detailed explanation. Your response on it:

 > for those assertions.

is a supporting evidence that you can't understand detailed
explanations.

"language specification" != "compilers"

						Masataka Ohta

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 17 Sep 2017 08:54:52 +1200

> RFC 7994 is a massive regression here.

As far as I can tell, both RFC 3629 and RFC 5198 refer to byte streams transmitted
in PDUs. RFC 7994 defines a file format, not a protocol byte stream format.
That's quite explicit in section 2:
"Plain-text files for RFCs will use the UTF-8 [RFC3629] character
encoding...
The plain-text file will include a Byte Order Mark (BOM)..."

Now if we were defining a protocol for the transmission of RFCs, or
more generally a protocol for the transmission of UTF-8 documents,
it would be a different story. We could say:

If the document starts with a Byte Order Mark (BOM) this MUST be
removed prior to transmission.

(Quite what FTP or RSYNC should do in such a case is an interesting
discussion.)

   Brian

---

From: "tom p." <[email-masked]>
Date: Sun, 17 Sep 2017 21:48:50 +0100

---- Original Message -----
From: "Denis Ovsienko" <[email-masked]>
Sent: Friday, September 15, 2017 6:31 PM

---- On Fri, 15 Sep 2017 17:44:12 +0100 tom p. <[email-masked]>
wrote ----
 > glyph outside normal ASCII with a line terminator function.

As far as hexdump of https://www.rfc-editor.org/rfc/rfc8187.txt goes the
character you are describing is a plain ASCII "e" (0x65):

00001f20  74 69 6f 6e 0a 20 20 20  69 6e 74 6f 20 70 61 72  |tion.
into par|
00001f30  61 6d 65 74 65 72 20 76  61 6c 75 65 73 20 61 6e  |ameter
values an|
00001f40  64 20 61 6c 73 6f 20 68  6f 77 20 74 6f 20 65 6e  |d also how
to en|

FireFox correctly displays the .txt file in Unicode by default, this
likely has to do with the UTF-8 BOM at the beginning of the file (both
observations are based on the file rsync'ed from rfc-editor.org).

<tp>

Denis

Thanks for that.  I need to dig some more.

My oldest utility does not understand the BOM at the start of the file
and displays it, well tries to.  Other utilities suppress the BOM but
then some give me those blobs, one of which I identified above, others
do not, all pointed at the same file on the same disk, downloaded from
RFC Editor web site (or a proxy thereof which is invisible to me).  I do
not use rsync.

Um, I do not understand this but will dig some more.

Tom Petch

One thing I vaguely remember in related documents from around 2013 was
that Unicode in the document text and in any person/organization names
was supposed to have a 2nd ASCII version as a backup, does it still
stand? Because in this RFC the Acknowledgements section does not provide
a backup spelling for Martin Dürst. Not a big issue, just curious.

Other than that, this is a remarkable milestone in a big work. Hopefully
document authors will remember that more [technical] power implies more
responsibility.

--
    Denis Ovsienko

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 07:59:57 +0200

> But section 2 of RFC 5198 says the opposite and, IMHO, it is right.

The reason for the BOM was so that existing tools will load the file 
correctly in absence of character encoding information.

(AFAIR, the ability to make tools like Notepad "do the right thing" was 
an important step to actually get to the decision to allow non-ASCII 
characters).

And yes, this is only relevant for plain text (as opposed to HTML), 
served from the file system.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 08:04:05 +0200

On 2017-09-16 02:30, Martin J. Dürst wrote:
>> stand? Because in this RFC the Acknowledgements section does not 
>> provide a backup spelling for Martin Dürst. Not a big issue, just 
>> curious.
> 

"Person names may appear in several places within an RFC (e.g., the 
header, Acknowledgements, and References). When a script outside the 
Unicode Latin blocks [UNICODE-CHART] is used for an individual name, an 
author-provided, ASCII-only identifier will appear immediately after the 
non-Latin characters, surrounded by parentheses. This will improve 
general readability of the text." -- 
<https://www.greenbytes.de/tech/webdav/rfc7997.html#rfc.section.3.2.p.1>

Best regards, Julian

---

From: Tim Bray <[email-masked]>
Date: Sun, 17 Sep 2017 23:09:00 -0700

My hearty thanks to all those who dragged this process into the 21st
century. I might go so far as to say 💓 💓 💓!

Seriously, strong work.

On Sep 14, 2017 2:07 PM, "Heather Flanagan (RFC Series Editor)" <
[email-masked]> wrote:

> _______________________________________________
> rfc-interest mailing list
> [email-masked]
> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
>

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 18 Sep 2017 08:43:50 +0200

> The reason for the BOM was so that existing tools will load the file correctly in absence of character encoding information.
> 
> (AFAIR, the ability to make tools like Notepad "do the right thing" was an important step to actually get to the decision to allow non-ASCII characters).
> 
> And yes, this is only relevant for plain text (as opposed to HTML), served from the file system.

Employing the Byte Order Mark (BOM), which is needed in UTF-16 but not in UTF-8, as a file “signature” (magic number) to identify plain text files that use UTF-8 beyond ASCII, is well known to have caused many of the problems in migrating to UTF-8.

The problems come both from tools that otherwise would have no problem upgrading from ASCII to UTF-8, that are now malfunctioning because of those BOMs, and from tools that now suddenly *expect* that all UTF-8 files beyond ASCII have that signature and no longer work when they don’t.  The first set of problems is confounded by tools that are silently inserting BOMs at various stages of processing UTF-8 files (BOM pollution), and by other tools that make any BOM present in a plain text file invisible to casual examination so problems caused by BOM pollution are hard to recognize.

The problems caused by BOM pollution were already well understood at the time when the various standards around UTF-8 were written.  Unicode itself recommends against it.  RFC 3629 has a whole section denouncing it.  RFC 5198 is careful to avoid BOM pollution in network unicode.

So the standards message is clear.  No BOM pollution.

Yet, on the operational side, the IETF has failed for more than a decade to properly serve UTF-8 in its own systems.  Now that we finally provide RFCs with UTF-8 beyond ASCII, we go ahead and embrace BOM pollution as if we didn’t know what we are doing.

This sends the message that BOM pollution is actually OK, even maybe the right thing everybody else should be doing as well, and the standards documents are for preaching on Sundays but to be ignored when it comes to actual practice.  It’s as if we were running all our servers without security because that might be considered operationally more expedient.

Grüße, Carsten

---

From: Tim Bray <[email-masked]>
Date: Sun, 17 Sep 2017 23:50:17 -0700

Carsten is right. Please no more BOMs in UTF-8.

On Sep 17, 2017 11:44 PM, "Carsten Bormann" <[email-masked]> wrote:

> Employing the Byte Order Mark (BOM), which is needed in UTF-16 but not in
> UTF-8, as a file “signature” (magic number) to identify plain text files
> that use UTF-8 beyond ASCII, is well known to have caused many of the
> problems in migrating to UTF-8.
>
> The problems come both from tools that otherwise would have no problem
> upgrading from ASCII to UTF-8, that are now malfunctioning because of those
> BOMs, and from tools that now suddenly *expect* that all UTF-8 files beyond
> ASCII have that signature and no longer work when they don’t.  The first
> set of problems is confounded by tools that are silently inserting BOMs at
> various stages of processing UTF-8 files (BOM pollution), and by other
> tools that make any BOM present in a plain text file invisible to casual
> examination so problems caused by BOM pollution are hard to recognize.
>
> The problems caused by BOM pollution were already well understood at the
> time when the various standards around UTF-8 were written.  Unicode itself
> recommends against it.  RFC 3629 has a whole section denouncing it.  RFC
> 5198 is careful to avoid BOM pollution in network unicode.
>
> Yet, on the operational side, the IETF has failed for more than a decade
> to properly serve UTF-8 in its own systems.  Now that we finally provide
> RFCs with UTF-8 beyond ASCII, we go ahead and embrace BOM pollution as if
> we didn’t know what we are doing.
>
> This sends the message that BOM pollution is actually OK, even maybe the
> right thing everybody else should be doing as well, and the standards
> documents are for preaching on Sundays but to be ignored when it comes to
> actual practice.  It’s as if we were running all our servers without
> security because that might be considered operationally more expedient.
>
> Grüße, Carsten
>
>

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 08:54:19 +0200

> The problems come both from tools that otherwise would have no problem upgrading from ASCII to UTF-8, that are now malfunctioning because of those BOMs, and from tools that now suddenly *expect* th…

It would be helpful if you have examples for these kinds of tools. I'd 
like to understand whether this has happened in practice (and when), or 
whether it's a purely theoretical argument.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 18 Sep 2017 09:37:45 +0200

On Sep 18, 2017, at 08:54, Julian Reschke <[email-masked]> wrote:
> It would be helpful if you have examples for these kinds of tools. I'd like to understand whether this has happened in practice (and when), or whether it's a purely theoretical argument.

Attacking the BOM pollution problem with scientific rigor is a bit outside my domain, but I can give you some anecdotal evidence for the different classes of tools cited above.

A *pollution-intolerant* tool is one that can do UTF-8, but reacts adversely to UTF-8 with BOMs.  Many tools that look for text signatures to do some form of file type detection are foiled by BOMs.  A recent example I happen to remember is Mark Thomson’s I-D template code; this tries to recognize two different variants of markdown by looking at the first three characters and fails if those include a BOM.

A *pollution-expecting* tool is one that requires all UTF-8 files to have a BOM and doesn’t work properly otherwise.  Windows Notepad is rumored to have that problem, but I don’t use Windows, so I can’t verify that.

A *polluting* tool is one that silently adds BOMs.  The anecdotal evidence I have here is hear-say:  When the IETF secretariat briefly switched to serving plain-text as UTF-8, they rolled that back when they heard that people had processing pipelines that combined a polluting with a pollution-intolerant tool, so the pipeline broke.  Apparently, the polluting tool stored any file that was served via HTTP as UTF-8 with a BOM added; I don’t know what the pollution-intolerant tools were.

A *pollution-hiding* tool is one that makes BOM pollution hard to recognize — I think many tools have reacted to increasing BOM pollution by hiding this arcane detail from their users, so there are many examples — pollution-hiding is the result of the usual process of specification soupification.  (As a counter-example, kramdown-rfc provides a warning when it detects BOM pollution in its input, but does process the file; emacs shows the encoding of a BOM-polluted file as “B” [instead of “U” for UTF-8] end preserves the BOM pollution during editing, continuing to treat BOM pollution as another encoding that is alternative to UTF-8 — so it is *pollution-preserving*.  Clang mentions BOM pollution only when it provides a warning or error about the first line: then it explicitly shows the BOM as <U+FEFF>.  And so on — time for a survey paper, I think.)

It is amazing how much programmer time — and, more importantly, user time — must have been wasted by the idea of being “helpful” with the UTF-8 transition through BOM pollution.  This appears to be an instance of the general rule that, when people try to be “helpful” in a protocol transition, you better run as fast as you can...

Grüße, Carsten

---

From: Jari Arkko <[email-masked]>
Date: Mon, 18 Sep 2017 14:00:27 +0300

Indeed. This was an important and very useful milestone. Thanks, all who helped make it happen. And yes, I’m sure there’s plenty of work ahead too.

Jari

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 18 Sep 2017 16:53:55 +0200

I am somewhat unclear about the process leading to this work
so please excuse one question:

Which working group/mailing list would i had
to participate in to influlence this decision making ? 

Thanks
    Toerless

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 16:56:43 +0200

> to participate in to influlence this decision making ?

<https://www.greenbytes.de/tech/webdav/rfc7990.html> might be helpful.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Mon, 18 Sep 2017 16:57:50 +0200

On Mon, Sep 18, 2017 at 04:53:55PM +0200,
 Toerless Eckert <[email-masked]> wrote 

> to participate in to influlence this decision making ? 

I think it was
<https://www.rfc-editor.org/mailman/listinfo/rfc-interest>

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 18 Sep 2017 17:39:31 +0200

Yes, Heather provided that reference earlier on in the process, but
that RFC seems to be like IAB product, and from my understanding (which may be wrong),
there is no open process to participate in IAB work. At least i was simply
ignored once i asked to be allowed to participate in one IAB mailing list
where specific work was done - eg: seems more like "invitation only work". 

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 18 Sep 2017 17:40:45 +0200

Ah, thanks! subscribed ;-)

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 18:03:39 +0200

> where specific work was done - eg: seems more like "invitation only work".

Did you have a look at 
<https://www.greenbytes.de/tech/webdav/rfc7990.html#overview-of-the-decision-making-process> 
specifically?

Best regards, Julian

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 12:50:53 -0500

On 9/18/17 1:43 AM, Carsten Bormann wrote:
> The problems caused by BOM pollution were already well understood at the time when the various standards around UTF-8 were written.  ...  RFC 3629 has a whole section denouncing it.

You refer to section 6? <https://tools.ietf.org/html/rfc3629#section-6> 
Denunciation seems like a pretty severe mischaracterization. I think the 
verb you were looking for is "endorsing."

Let's look at the actual guidance (ignoring for the moment that it's 
talking about protocol design rather than archival documents):

>    o  A protocol SHOULD forbid use of U+FEFF as a signature for those
>       textual protocol elements that the protocol mandates to be always
>       UTF-8, the signature function being totally useless in those
>       cases.

Well, *that's* not us, since we demonstrably have a mix of ASCII and 
UTF-8 documents. (And, on the protocol level, servers that serve up RFCs 
can certainly have a mix of these and other encodings)

>    o  A protocol SHOULD also forbid use of U+FEFF as a signature for
>       those textual protocol elements for which the protocol provides
>       character encoding identification mechanisms, when it is expected
>       that implementations of the protocol will be in a position to
>       always use the mechanisms properly.  This will be the case when
>       the protocol elements are maintained tightly under the control of
>       the implementation from the time of their creation to the time of
>       their (properly labeled) transmission.

Given that these things are replicated all over the place and metadata 
(such as character encoding) are typically not part of that replication, 
this also seems inapplicable.

>    o  A protocol SHOULD NOT forbid use of U+FEFF as a signature for
>       those textual protocol elements for which the protocol does not
>       provide character encoding identification mechanisms, when a ban
>       would be unenforceable, or when it is expected that
>       implementations of the protocol will not be in a position to
>       always use the mechanisms properly.  The latter two cases are
>       likely to occur with larger protocol elements such as MIME
>       entities, especially when implementations of the protocol will
>       obtain such entities from file systems, from protocols that do not
>       have encoding identification mechanisms for payloads (such as FTP)
>       or from other protocols that do not guarantee proper
>       identification of character encoding (such as HTTP).

Oh. OH! There it is. It even calls out FTP and HTTP as the poster 
children for "SHOULD NOT forbid", and I suspect that we're well past 99% 
of all RFC access being over those two protocols (with the vast majority 
being HTTP).

We did a lot of hashing over of this issue on the rfc-design team (and I 
even started out where you are now, advocating against a BOM). While I 
concede that there is no perfect solution, the "pro" column for using 
BOMs ended up being far more compelling than the "con" column.

Even if we find the philosophical arguments to be a draw, this single 
practical argument is what convinced me that not including a BOM would 
be a solid request for failure:

On 11/3/13 12:08 PM, Dave Thaler wrote:

> Reality check...
>
> I just ran a test with two UTF-8 files, one with a BOM and one 
> without. In case you want to try them yourself, they're at
>
> http://research.microsoft.com/~dthaler/Utf8NoBom.txt
>
> http://research.microsoft.com/~dthaler/Utf8WithBom.txt
>
> It includes Latin, Greek, and Cyrillic.
>
>
> I tried opening them with a bunch of utilities, and browsers (opening 
> local files not using HTTP), and used browsershots.org to get 
> screenshots of HTTP access across many browsers and platforms. Note 
> the HTTP server provides no content encoding headers so it's up to the 
> app to detect. I just copied the files to a generic web server, and we 
> may expect others would do the same with their own I-Ds and RFC mirrors.
>
> Results:
>
> 1) Some apps worked fine with both files.  These include things like 
> notepad, outlook, Word, file explorer, Visual Studio 2012
>
> 2) Some apps failed with both files (probably written to be ASCII 
> only). These include things like Windiff, stevie (a vi clone), 
> textpad, and the Links browser (on Ubuntu), and the Konquerer browser 
> (on Ubuntu)
>
> 3) Everything else, including almost all browsers, only displayed the 
> file correctly with the BOM
>
> This included:
>
>   * Windows apps: Wordpad
>   * Windows using local files (no HTTP): IE, Firefox, Chrome
>   * Windows using HTTP: IE, Firefox, Chrome, Navigator
>   * Mac OSX: Safari, Camino
>   * Debian: Opera, Dillo
>   * Ubuntu: Luakit, Iceape
>
>
> Conclusion: If we want people to use UTF-8 RFCs and I-Ds with existing 
> tools and browsers today, any UTF-8 text format needs to include a BOM.

/a

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 20:01:36 +0200

> can certainly have a mix of these and other encodings)

Well, any text file encoded in US-ASCII by definition is also encoded in 
UTF-8.

I believe a better argument is that we do not control the tools that 
people use to read plain text RFCs, and that we found that some 
important ones simply work better with the BOM.

> this also seems inapplicable.

Right.

> BOMs ended up being far more compelling than the "con" column.

That's my recollection as well.

>> tools and browsers today, any UTF-8 text format needs to include a BOM.

And yes, that's a good summary from back then.

Best regards, Julian

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 14:13:05 -0400

On Sep 18, 2017, at 2:01 PM, Julian Reschke <[email-masked]> wrote:
> I believe a better argument is that we do not control the tools that people use to read plain text RFCs, and that we found that some important ones simply work better with the BOM.

If we believe that it is a bug for an application to fail to correctly display a UTF-8 file without a BOM, then the fact that some significant number of apps currently do the wrong thing isn't compelling.  Browsers have very rapid development cycles.   The fact that some browsers Dave Thaler tested three years ago didn't do the right thing is not a good enough reason to use BOMs now.   The question should be "what is the best solution," not "what works best with legacy apps?"

There was a time when a BOM made technical sense, but I think that time is long past.   At this point it's just an onion, and we should stop using it.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 20:25:52 +0200

On 2017-09-18 20:13, Ted Lemon wrote:
> On Sep 18, 2017, at 2:01 PM, Julian Reschke <[email-masked] 
> <mailto:[email-masked]>> wrote:
> If we believe that it is a bug for an application to fail to correctly 
> display a UTF-8 file without a BOM, then the fact that some significant 
> number of apps currently do the wrong thing isn't compelling.  Browsers 
> have very rapid development cycles.   The fact that some browsers Dave 
> Thaler tested three years ago didn't do the right thing is not a good 
> enough reason to use BOMs now.   The question should be "what is the 
> best solution," not "what works best with legacy apps?"

Browsers are not the issue (well, when accessing HTTP resources). They 
all follow the Content-Type response header field (which has the 
character encoding), and we can set it.

The problems are tools that do *not* have the metadata. That is: 
download the RFC and double-click it in Windows.

Best regards, Julian

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 18 Sep 2017 15:07:11 -0400

> On Sep 18, 2017, at 2:25 PM, Julian Reschke <[email-masked]> wrote:
> 
> The problems are tools that do *not* have the metadata. That is: download the RFC and double-click it in Windows.

Legacy systems will continue to run in some non-UTF8 code page for
some time, but with any luck newer systems are moving on.

FWIW, I have exceedingly little trouble with UTF-8.  On all my
systems (Debian, MacOSX, NetBSD and FreeBSD) I have:

   $ env | grep '^L[AC]'
   LANG=C
   LC_CTYPE=en_US.UTF-8

So UTF-8 is expected by default in all data.  The only time I have
to worry about other encodings is the occasional "whois" server that
returns iso-8859-1 instead of UTF-8, and I have to run the results
through iconv(1):

    $ whois -h whois.registro.br registro.br |
      iconv -f iso-8859-1 -t utf-8 |
      perl -lne 'print if m{[^\x00-\x7f]}'
    owner:       Núcleo de Inf. e Coord. do Ponto BR - NIC.BR

It will take decades, but I hope that eventually most systems
will run in a UTF-8 locale by default, and exceptions will become
increasingly rare.

-- 
	Viktor.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 18 Sep 2017 21:17:27 +0200

On Sep 18, 2017, at 19:50, Adam Roach <[email-masked]> wrote:
> 
>>  Note the HTTP server provides no content encoding headers so it's up to the app to detect.

But that is not the way HTTP works.

If you make servers misrepresent the content, browsers will follow what they say.

“No content encoding” (I read: no charset) for text/plain is not possible, as there is a default; it means ASCII.

RFC 2046 4.2.1:

   The default character set, which
   must be assumed in the absence of a charset parameter, is US-ASCII.

So you sent the UTF-8 files declaring them as ASCII; it is no wonder that browsers had problems with that.

Obviously, servers can easily be configured to send all RFCs as UTF-8, as all *are* UTF-8.
We had a brief time (in 2006, IIRC) when that actually was the case with the IETF servers, until a random user with a BOM-polluting application got in the way.  We should have used that as a teachable moment, instead of regressing to ISO 8859-1 for another decade.

Grüße, Carsten

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 14:33:19 -0500

> But that is not the way HTTP works.

I'd request that you be more careful with attribution; those are Dave's 
words. More importantly, context matters: he was discussing a technical 
aspect of how he was trying to game the browsers into acting as if they 
were loading the files from local store when performing bulk testing 
across multiple operating systems.

The overarching point is that Dave performed some pretty extensive tests 
with a large number of applications, and found (of those tested):

 1. Applications that work both with and without a BOM: a significant
    minority
 2. Applications that fail regardless of whether a BOM is present:
    again, a significant minority
 3. Applications that work with a BOM and fail without one: a clear majority
 4. Applications that fail with a BOM and work without one: Zero.

You have some theoretical arguments here that suppose the fourth 
category is the most important one, while applications in the third 
category should be ignored. That's some pretty bizarro-world reasoning, 
given the collected data. To make a compelling argument, I think you 
would need to explain why your notion of aesthetic purity should 
supersede practical impact on users.

/a

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 15:03:03 -0500

On 9/18/17 1:13 PM, Ted Lemon wrote:
> display a UTF-8 file without a BOM

Do we? Can you cite something that makes this claim under color of 
consensus?

/a

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 16:15:43 -0400

On Sep 18, 2017, at 4:03 PM, Adam Roach <[email-masked]> wrote:
> Do we? Can you cite something that makes this claim under color of consensus?

See Page 36, the paragraph after the table: http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf <http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf>

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 16:18:11 -0400

> Do we? Can you cite something that makes this claim under color of consensus?

Also, consider what it takes to make catting a UTF-8 file work in a terminal program.   ISTM that the only way for it to not spew garbage characters is for the terminal program to understand UTF-8 without requiring a BOM.   I suppose you could make the terminal program elide the BOM wherever it's encountered in the output stream (on a character boundary), but this isn't how BOMs are supposed to work.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 22:18:48 +0200

> So you sent the UTF-8 files declaring them as ASCII; it is no wonder that browsers had problems with that.

This was overruled by 
<https://www.greenbytes.de/tech/webdav/rfc2616.html#rfc.section.3.7.1.p.4>, 
making the default ISO-8859-1. That mistake was fixed in RFC 7231, and 
also the clause you cited was obsoleted by 
<https://www.greenbytes.de/tech/webdav/rfc6657.html>.

> Obviously, servers can easily be configured to send all RFCs as UTF-8, as all *are* UTF-8.

Yes.

But that doesn't help when that metadata is not available (such as after 
"save as").

> We had a brief time (in 2006, IIRC) when that actually was the case with the IETF servers, until a random user with a BOM-polluting application got in the way.  We should have used that as a teacha…

Details, please.

Best regards, Julian

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 15:23:24 -0500

On 9/18/17 15:15, Ted Lemon wrote:
> On Sep 18, 2017, at 4:03 PM, Adam Roach <[email-masked] 
> <mailto:[email-masked]>> wrote:
> See Page 36, the paragraph after the table: 
> http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf
>

Sorry, I meant IETF consensus. I'm not going to cite specific examples 
(no point poking a hornet's nest in this conversation), but I believe 
that there are some places where the Unicode consortium and the IETF 
have reached mutually incompatible conclusions on certain issues.

/a

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 22:25:28 +0200

<http://www.unicode.org/versions/Unicode5.0.0/ch02.pdf#page=30>:

> The endian order entry for UTF-8 in Table 2-4 is marked N/A because UTF-8 code units
> are 8 bits in size, and the usual machine issues of endian order for larger code units do not
> apply. The serialized order of the bytes must not depart from the order defined by the UTF-
> 8 encoding form. Use of a BOM is neither required nor recommended for UTF-8, but may
> be encountered in contexts where UTF-8 data is converted from other encoding forms that
> use a BOM or where the BOM is used as a UTF-8 signature. See the “Byte Order Mark”
> subsection in Section 16.8, Specials, for more information.

Is this the paragraph you're referring to?

Best regards, Julian

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 15:26:08 -0500

On 9/18/17 15:18, Ted Lemon wrote:
> Also, consider what it takes to make catting a UTF-8 file work in a 
> terminal program.   ISTM that the only way for it to not spew garbage 
> characters is for the terminal program to understand UTF-8 without 
> requiring a BOM.   I suppose you could make the terminal program elide 
> the BOM wherever it's encountered in the output stream (on a character 
> boundary), but this isn't how BOMs are supposed to work.
>

Given that the formal name for the thing we're calling a BOM is "ZERO 
WIDTH NO-BREAK SPACE" -- which is a pretty accurate description for how 
it is intended to be rendered in UTF-8 -- I'm not sure what kind of 
visual damage you've envisioning here.

/a

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 16:28:26 -0400

Yes.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 16:31:52 -0400

On Sep 18, 2017, at 4:26 PM, Adam Roach <[email-masked]> wrote:
> Given that the formal name for the thing we're calling a BOM is "ZERO WIDTH NO-BREAK SPACE" -- which is a pretty accurate description for how it is intended to be rendered in UTF-8 -- I'm not sure what kind of visual damage you've envisioning here.
> 

When you copy and paste the text, does it copy the ZWNBS or not?

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 22:35:09 +0200

On 2017-09-18 22:28, Ted Lemon wrote:
> On Sep 18, 2017, at 4:25 PM, Julian Reschke <[email-masked] 

I think you're reading something into this that's no there.

Concrete example: a system text editor that defaults to the system 
encoding (for historic reasons). Are you saying it's buggy when it 
continues to do so?

(And yes, my *preference* is an UTF-8 world as well, but sometimes it's 
hard to get there)

Best regards, Julian

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 15:35:26 -0500

On 9/18/17 15:31, Ted Lemon wrote:
> On Sep 18, 2017, at 4:26 PM, Adam Roach <[email-masked] 
>> WIDTH NO-BREAK SPACE" -- which is a pretty accurate description for 
>> how it is intended to be rendered in UTF-8 -- I'm not sure what kind 
>> of visual damage you've envisioning here.
>>

As with all clipboard-related issues, I'm sure the answer to that varies 
and is heavily dependent on the tuple of {source application, 
destination application, operating system}.

/a

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 16:39:12 -0400

On Sep 18, 2017, at 4:35 PM, Julian Reschke <[email-masked]> wrote:
> Concrete example: a system text editor that defaults to the system encoding (for historic reasons). Are you saying it's buggy when it continues to do so?

No, the system that doesn't have UTF-8 as its default encoding is buggy.   Now, at this present time.   If you are still running Windows XP and you see some artifacts when interacting with the modern Internet, that's not our problem.

The paragraph I quoted simply says that UTF-8 BOM isn't useful or recommended.   So it's agreeing with Carsten, and I guess with me.   If we disagree with the consortium on this, that's okay, but we ought to have a really good reason for disagreeing.

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 16:12:46 -0500

On 9/18/17 15:39, Ted Lemon wrote:
> The paragraph I quoted simply says that UTF-8 BOM isn't useful or 
> recommended.   So it's agreeing with Carsten, and I guess with me.   
> If we disagree with the consortium on this, that's okay, but we ought 
> to have a really good reason for disagreeing.

I think "putting it in is demonstrably necessary for proper rendering in 
a lot of real-world applications" is a Really Good Reason.

So far, in all the arguments *against* BOMs, I have not yet seen a 
concrete "here is what will break with BOMs (but which works fine 
without them)" argument.  I understand -- and, to some degree, 
sympathize with -- the arguments that putting BOMs in UTF-8 documents is 
distasteful in principle. But until someone actually posts a 
step-by-step set of "steps to reproduce the problem," these concerns 
remain theoretical [1].

In this case, I would give real problems significantly more weight when 
balanced against imagined ones.

/a

____
[1] Based on your query about clipboard behavior, I have no doubt that 
you're trying -- and, if this is your best avenue of investigation, 
almost certainly failing -- to come up with such an example. In my 
experience, OSes tend to handle these kinds of things quite gracefully.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 18 Sep 2017 23:21:04 +0200

On 2017-09-18 22:39, Ted Lemon wrote:
> On Sep 18, 2017, at 4:35 PM, Julian Reschke <[email-masked] 
> No, the system that doesn't have UTF-8 as its default encoding is buggy.

Is it the RFC Editor's task to change this?

>    Now, at this present time.   If you are still running Windows XP and 
> you see some artifacts when interacting with the modern Internet, that's 
> not our problem.

Once again, dealing with "the internet" is not a problem.

What's a problem is handling these files once they are stored locally.

> recommended.   So it's agreeing with Carsten, and I guess with me.   If 
> we disagree with the consortium on this, that's okay, but we ought to 
> have a really good reason for disagreeing.

What it says that it's not needed for distinguishing between the various 
Unicode encoding schemes (of which, IIUC, we want to get rid of all but 
UTF-8 anyway).

IMHO it doesn't address distinguishing between Unicode encoding schemes 
and *other* encoding schemes. For which I believe we have evidence that 
it *is* useful in practice in some cases. (And yes, I hear you Martin: 
UTF-8 can be reliably sniffed; but the applications we tested with didn't).

I'm not saying the the decision to use the BOM was the correct one. It 
was reached after a long public discussion though.

Best regards, Julian

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 17:48:36 -0400

Look, i don't want to get into a long exchange about this.   I agree with Carsten: keeping the BOM is a past-looking choice, not a forward-looking choice, and it does cause problems.   It's absurd to think that the people who will be reading these documents in their downloaded form will not be technically competent to sort this out if issues arise.   But honestly, I don't care that much—it's crazy to use the ASCII or UTF-8 version of the document anyway, and if you do, you should expect to lose.

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 16:51:55 -0500

On 9/18/17 16:48, Ted Lemon wrote:
> Look, i don't want to get into a long exchange about this.   I agree with Carsten: keeping the BOM is a past-looking choice, not a forward-looking choice, and it does cause problems.

I don't want to get in a protracted argument either, but I find your 
assertion that "it does cause problems" to be lacking concrete examples. 
Can you provide an existence proof of one such example? If you're right, 
it shouldn't be too hard.

/a

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 17:59:19 -0400

On Sep 18, 2017, at 5:51 PM, Adam Roach <[email-masked]> wrote:
> I don't want to get in a protracted argument either, but I find your assertion that "it does cause problems" to be lacking concrete examples. Can you provide an existence proof of one such example? If you're right, it shouldn't be too hard.

I'm taking Carsten at his word: he described some significant problems.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 19 Sep 2017 00:10:04 +0200

On Sep 18, 2017, at 23:12, Adam Roach <[email-masked]> wrote:
> 
> I think "putting it in is demonstrably necessary for proper rendering in a lot of real-world applications" is a Really Good Reason.

That would have been a Really Good Reason to leave security off on the IETF infrastructure.
Security creates so many problems; it’s best to leave it off.
And security for IETF services does not really matter much in practice.

Still, we are doing Security to lead the pack.
It’s all about sending the right signals.

Oh, and we have had countless IPv6 issues, a Really Good Reason to leave that off, too.

What is so disappointing about this discussion is that it’s occurring in 2017.

In 1997, there might have been a point.

On September 3, 1967, road driving in Sweden was switched to driving on the right side.
There was no transition period; people had to run their systems (cars) using the new rules right away.
(It is maybe educating that the popular vote about this change in 1955 was 15.5 % for, 82.9 % against making the change.)

Soupification approaches like BOM pollution are the road to eternal transition.
It feels like we are going to be driving on both sides of the street for the foreseeable future, forcing all right-side drivers to watch out all the time for the remaining left-side drivers, putting in lots of infrastructure to deal with the inevitable collisions, and just generally driving very slowly.

The transition to UTF-8 is pretty much done out there in the real world.
Still, as with Security and IPv6, IETF MUST be leading, not lagging in this space.
It’s all about sending the right signals.

I intend this to be my last message on this thread (but I’m making no promises).

Grüße, Carsten

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 17:13:01 -0500

On 9/18/17 16:59, Ted Lemon wrote:
> On Sep 18, 2017, at 5:51 PM, Adam Roach <[email-masked] 
>> assertion that "it does cause problems" to be lacking concrete 
>> examples. Can you provide an existence proof of one such example? If 
>> you're right, it shouldn't be too hard.
>

No, he described some significant (and not entirely unjustified) 
concerns. His description of the actual example I ask for, he provided 
nearly nothing.

To wit: he described exactly one tool -- written by Martin Thomson -- 
making the wrong guess between two variants of markup. That's not 
nothing, but I expect that even Martin would concede that it was a 
pretty ugly and fragile hack to start with (e.g., both file formats 
would legally allow files to start with two linefeeds, and cause the 
tool to fail in the same way as a BOM does).

/a

---

From: Ted Hardie <[email-masked]>
Date: Mon, 18 Sep 2017 15:15:53 -0700

On Mon, Sep 18, 2017 at 2:51 PM, Adam Roach <[email-masked]> wrote:

>> Look, i don't want to get into a long exchange about this.   I agree with
>> Carsten: keeping the BOM is a past-looking choice, not a forward-looking
>> choice, and it does cause problems.
>>
>
>
> Can you provide an existence proof of one such example? If you're right, it
> shouldn't be too hard.
>
> /a
>
>
As a simple test case, some BCPs are displayed by concatenating the
relevant RFCs.  As RFC 3629, Section 6 notes, that concatenation will take
U+FEFF and convert it from being a signature to being a non-breaking
space.  Can we test our tools to confirm that we are stripping it correctly
in this case?  If we're not doing it, I doubt anyone else is....

Most of the text in RFC 3629 seems still on the mark, not that I'm an
expert, but if we take it to be true, it has some advice.  That says that
you should forbid the use of U+FEFF as a signature when you have character
encoding identification mechanisms, and that you can retain it otherwise.
What do we think the case is here?

regards,

Ted

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 17:17:29 -0500

If you want to reduce this to analogies, your proposal is much closer to 
the situation we had in the IETF a couple of decades ago when it came to 
NATs: we preferred architecturally pristine approaches that shattered 
into useless, broken shards on contact with reality.

/a

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 17:18:23 -0500

On 9/18/17 17:15, Ted Hardie wrote:
> expert, but if we take it to be true, it has some advice.  That says 
> that you should forbid the use of U+FEFF as a signature when you have 
> character encoding identification mechanisms, and that you can retain 
> it otherwise.  What do we think the case is here? 

Given that these frequently end up on filesystems without metadata 
indicating their character encoding, I think it's pretty clearly the 
second case.

/a

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 18:19:38 -0400

On Sep 18, 2017, at 6:15 PM, Ted Hardie <[email-masked]> wrote:
> Most of the text in RFC 3629 seems still on the mark, not that I'm an expert, but if we take it to be true, it has some advice.  That says that you should forbid the use of U+FEFF as a signature when you have character encoding identification mechanisms, and that you can retain it otherwise.  What do we think the case is here?  

Well, presumably we are disabling FTP downloads since they aren't authenticatable, so in practice the http server can identify the text as text/utf8, which would mean that the RFC3629 advice to follow would be the first clause, not the second.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 18:20:14 -0400

On Sep 18, 2017, at 6:18 PM, Adam Roach <[email-masked]> wrote:
> Given that these frequently end up on filesystems without metadata indicating their character encoding, I think it's pretty clearly the second case.

That would be a bug in the thing that did the download, wouldn't it?   IOW, not our problem?

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 17:24:36 -0500

On 9/18/17 17:20, Ted Lemon wrote:
> On Sep 18, 2017, at 6:18 PM, Adam Roach <[email-masked] 

Unless you know something about NTFS, ext4, HFS, and exFAT that I don't, 
this sort of information isn't generally part of file metadata at all.

/a

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 19 Sep 2017 00:57:18 +0200

On Sep 19, 2017, at 00:17, Adam Roach <[email-masked]> wrote:
> 
> If you want to reduce this to analogies, your proposal is much closer to the situation we had in the IETF a couple of decades ago when it came to NATs: we preferred architecturally pristine approaches that shattered into useless, broken shards on contact with reality.

Nice volley.  Now, BOM pollution and NATs have in common that they are unclean hacks with lots of unpleasant consequences.

But that’s where the analogy ends.

NATs were invented to solve a real-world economic problem:  The attempt by ISPs to create price differentiation by charging for the number of IP addresses provided.  Nothing in hell could have stopped the powerful economic incentive to work around that attempt.

BOM pollution maybe has the incentive of allowing one to drag out the transition towards UTF-8, but that has happened already, so what’s the point.

I sincerely hope that the NAT history isn’t dragged out each time we need an apology for unclean hacks with lots of unpleasant consequences.

Grüße, Carsten

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 18:57:39 -0400

On Sep 18, 2017, at 6:24 PM, Adam Roach <[email-masked]> wrote:
> Unless you know something about NTFS, ext4, HFS, and exFAT that I don't, this sort of information isn't generally part of file metadata at all.

If you download a file in your web browser and save it to disk, the thing responsible for deciding whether or not to apply the BOM is the thing that did the download, not the server from which it was downloaded.   The server already identified the file encoding type: utf8 (not text/utf8, sorry about that).   If the thing that did the download does the wrong thing, that's not our problem.

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 18:03:19 -0500

> I sincerely hope that the NAT history isn’t dragged out each time we need an apology for unclean hacks with lots of unpleasant consequences.

I, for one, plan to only drag it out whenever I see someone who proposes 
that sticking our collective head in the sand regarding deployment 
realities is a wise choice. I hope that allays your stated concern.

/a

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 19 Sep 2017 09:09:33 +1000

On 19 Sep. 2017 08:59, "Ted Lemon" <[email-masked]> wrote:

If you download a file in your web browser and save it to disk, the thing
responsible for deciding whether or not to apply the BOM is the thing that
did the download, not the server from which it was downloaded.   The server
already identified the file encoding type: utf8 (not text/utf8, sorry about
that).   If the thing that did the download does the wrong thing, that's
not our problem.

I'm away from my computer so I can't easily check, but does `wget -r`
provide some sort of option to convert downloaded files in this way?  I
wouldn't be surprised about newline conversions, but I don't know about
BOMs.

Cheers
--
Matthew Kerwin

---

From: Adam Roach <[email-masked]>
Date: Mon, 18 Sep 2017 18:16:10 -0500

> If you download a file in your web browser and save it to disk, the thing responsible for deciding whether or not to apply the BOM is the thing that did the download, not the server from which it w…

I think we're talking at cross purposes here.

Today, as we speak, I have a copy of the RFC repository on my hard 
drive. (To be precise, I have it on most of the hard drives of the 
various machines that I use). For my current workflow, I *think* all of 
them got there via rsync, although it's possible that some of them are 
still using an old wget-based setup. It's kind of immaterial how they 
got there, because a careful examination of them would show the same 
result between the two methods (and any others I could think of, 
including FTP mirroring and manually downloading via web browsers): it's 
a sequence of bytes, with a ".txt" file extension; identical, regardless 
of which tool downloaded them. There is nothing else about the file to 
indicate its encoding.[1]

Okay. So, now, I open up the local file browser to that file on my hard 
drive, and double-click on an RFC. An application is launched. Let's say 
that application is Wordpad. How does it know which character encoding 
to use for this file?

/a

____
[1] If this is one of the Macs, and the download tool were really 
Mac-centric, it might have included a resource fork with some additional 
metadata, but (AFIAK), even the resource fork does not include character 
encoding. Other operating systems have similar constructs, but I'm less 
familiar with them.

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 18 Sep 2017 17:49:12 -0700

> On Sep 18, 2017, at 3:19 PM, Ted Lemon <[email-masked]> wrote:
> 
> Well, presumably we are disabling FTP downloads since they aren't authenticatable,

*Nobody* on the planet has read, let alone implemented, RFC 4217?

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 21:08:20 -0400

> I think we're talking at cross purposes here.

Yes, we are.   You just gave me an example where you know how to rsync the entire IETF document repository, and have set your system up to do it automatically, and yet you are apparently running ancient software that does not default to utf8, and you want us to fix it when you're perfectly capable of doing so yourself!   :)

Seriously, this isn't a use case that I think justifies your position.   You're asking us to do the wrong thing for your convenience.   What you need to do is show that we are asking to do the wrong thing; the convenience-for-geeks argument doesn't show that we are asking to do the wrong thing.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 21:10:29 -0400

> *Nobody* on the planet has read, let alone implemented, RFC 4217?

   - Although it is entirely an implementation decision, it is
     recommended that certificates used for server authentication of the
     TLS session contain the server identification information in a
     similar manner to those used for http servers (see [RFC-2818 <https://tools.ietf.org/html/rfc2818>]).

Whatevs, Saruman.

:)

(In case that was too flippant, my point is that RFC 4217 is really what we now call opportunistic security, and any kind of actual authentication of the endpoint is addressed in the document with a handwave.)

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 19 Sep 2017 13:24:49 +1200

On 19/09/2017 11:16, Adam Roach wrote:
...

> that application is Wordpad. How does it know which character encoding 

So, I made a version of rfc8187.txt without a BOM. Here's what I see
on Windows 7. (FYI, there are no fails when the BOM is present.)

Notepad: fail. All the non-ASCII characters display wrongly.

Wordpad: fail. (I have an ancient Wordpad and a more modern one.
They both fail.)

Libreoffice: fail. I understand that Libreoffice generates a BOM
when writing plain text files.

MS Word: asks me to select an encoding. When I select UTF-8, it works OK.

Firefox: Works OK if View/Text Encoding is set to Unicode

I couldn't find any options in the first three products to get round this.

It would be interesting to hear similar facts from people using other systems.
(My Python3 code for removing the BOM is below, or it seems there is
a utility called iconv that can do it.)

    Brian

rnum=input('RFC number:')
file1 = open("rfc"+rnum+".txt", "rb")
chunk=file1.read(1024)
if chunk[0:3] == b'\xef\xbb\xbf':
    chunk=chunk[3:]
    file2=open("rfc"+rnum+"X.txt", "wb")
    while chunk:
        file2.write(chunk)
        chunk=file1.read(1024)
    file2.close()
    print("Rewritten without BOM as","rfc"+rnum+"X.txt")
else:
    print("No BOM found")
file1.close()

---

From: Ted Lemon <[email-masked]>
Date: Mon, 18 Sep 2017 22:07:00 -0400

> a utility called iconv that can do it.)

On Windows 10, Notepad fails to recognize the EOL character.   Wordpad borks the formatting but at least displays something legible, but assumes that the file is ISO-8859-1, so there's a garbage character before each unicode character.   Can't get Edge to load a file—there's no UI for that.   God, what a straitjacket.   Sigh.

On the Mac, I couldn't get any application to do the wrong thing.

The real question is, what happens when Edge or Explorer downloads a file with an encoding of utf8, that doesn't include a BOM.

---

From: "tom p." <[email-masked]>
Date: Tue, 19 Sep 2017 10:53:41 +0100

----- Original Message -----
From: "Brian E Carpenter" <[email-masked]>
To: "IETF" <[email-masked]>
Sent: Tuesday, September 19, 2017 2:24 AM

> Notepad: fail. All the non-ASCII characters display wrongly.

agree but my (elderly?) Notepad with BOM present displays the BOM as

i double dot   >>  inverted question mark

no surpise there.

Wordpro and Word Viewer older versions likewise display the BOM when it
is present.

newer Word Viewer with no BOM asks me what encoding to use and displays
ok

> Wordpad: fail. (I have an ancient Wordpad and a more modern one.

agree but I note than when I save a file in UTF using Wordpad it uses
UTF16 not UTF8 AFAICT.

Tom Petch

---

From: "tom p." <[email-masked]>
Date: Tue, 19 Sep 2017 11:08:58 +0100

----- Original Message -----
From: "Ted Lemon" <[email-masked]>
Sent: Monday, September 18, 2017 9:31 PM

<tp>

It is bound to depend on which of the hundreds of different utilities
you use.

With recent Wordpad,
select all  - copy - new document - paste - save as text
produces a document that displays ok (and is not in UTF16 which Wordpad
sometimes uses)

Copying just the initial lines of the document produces one fails to
display the Euro symbol etc (much as I would expect)

Tom Petch

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Tue, 19 Sep 2017 04:09:18 -0700

> This sends the message that BOM pollution is actually OK, even maybe the right thing everybody else should be doing as well, and the standards documents are for preaching on Sundays but to be ignor…
The RFC Format Design Team discussed this at length. I'm copying (with
permission) the reports on the test that had us agreeing to use a BOM in
these files. The end result was that too many apps could not display a
UTF-8 file correctly without a BOM.

-Heather

-------- Forwarded Message --------
Subject: 	[rfc-design] UTF-8 text format and BOM
Date: 	Sun, 3 Nov 2013 18:08:08 +0000
From: 	Dave Thaler <[email-masked]>
To: 	RFC Design Team <[email-masked]>

Reality check...

I just ran a test with two UTF-8 files, one with a BOM and one without.

In case you want to try them yourself, they're at

http://research.microsoft.com/~dthaler/Utf8NoBom.txt
<http://research.microsoft.com/%7Edthaler/Utf8NoBom.txt>

http://research.microsoft.com/~dthaler/Utf8WithBom.txt
<http://research.microsoft.com/%7Edthaler/Utf8WithBom.txt>

I tried opening them with a bunch of utilities, and browsers (opening
local files not using HTTP),

and used browsershots.org to get screenshots of HTTP access across many
browsers and platforms.

Note the HTTP server provides no content encoding headers so it's up to
the app to detect.

I just copied the files to a generic web server, and we may expect
others would do the same

with their own I-Ds and RFC mirrors.

Results:

2) Some apps failed with both files (probably written to be ASCII only).
These include things like

Windiff, stevie (a vi clone), textpad, and the Links browser (on
Ubuntu), and the Konquerer browser (on Ubuntu)

This included:

Windows apps: Wordpad

Windows using local files (no HTTP): IE, Firefox, Chrome

Windows using HTTP: IE, Firefox, Chrome, Navigator

Mac OSX: Safari, Camino

Debian: Opera, Dillo

Ubuntu: Luakit, Iceape

Conclusion: If we want people to use UTF-8 RFCs and I-Ds with existing
tools and browsers today, any UTF-8 text format

needs to include a BOM.

-Dave

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 19 Sep 2017 13:22:07 +0200

On Sep 19, 2017, at 13:09, Heather Flanagan (RFC Series Editor) <[email-masked]> wrote:
> 
> The RFC Format Design Team discussed this at length. I'm copying (with permission) the reports on the test that had us agreeing to use a BOM in these files. The end result was that too many apps could not display a UTF-8 file correctly without a BOM.

Hi Heather,

this is a fine survey, and its result would be imperative if the objective of the RFC series were to maximize the viewing pleasure when using legacy applications.  I can understand that a design team tasked with this will arrive at exactly this conclusion.

However, the point of the RFC series is not the above, but to make the net work better.

BOM pollution does not make the net work better.

Worse, you are sending out the following signal loud and clear on behalf of the IETF:

	It is OK for an application to mistreat UTF-8 input unless it is BOM-polluted.

I’m unable to recreate the group dynamics of the design team that led to this unfortunate outcome, but I’m able to sympathize with the group-think focusing on deployability of this solution rather than evolution of the Internet at large.  All I intended to do here is point out that there are other considerations than legacy support (and that legacy support, while still relevant in some environments, is going away as a concern with the UTF-8 transition completing and also never really has been a concern for our target audience, which on legacy systems already has to fight with the non-DOS line ends and will mostly use the unaffected Web browsers anyway).

My disappointment is in that these considerations apparently were overruled by trying to please everyone.
This is not representative for the IETF that I want to work in.

Grüße, Carsten

---

From: Ted Lemon <[email-masked]>
Date: Tue, 19 Sep 2017 08:39:49 -0400

Carsten, actually I think the main deciding factor was that text file presentations of RFCs are notionally legacy, not current.   No sensible person reads an ASCII RFC, because the typography is terrible.   So we don't have to care about this: in the long run, the ASCII RFC is dead anyway.

The reason I jumped on your bandwagon is that I actually agree with you that the IETF is sending a message by doing this, and I agree that it's the wrong message.   It would be nice if we could craft a solution that works for naive users of ASCII RFCs, and is comprehensible to non-naive users like Adam.

But if we don't get sufficient consensus to talk Heather out of this policy, the earth will continue to rotate around the sun.   :)

---

From: "Paul Hoffman" <[email-masked]>
Date: Tue, 19 Sep 2017 07:29:26 -0700

> This is not representative for the IETF that I want to work in.

The drafts leading to the document that specified this action, RFC 7994, 
was discussed on an active mailing list with lots of members. The IESG 
asked, on this list, for people to review the set of documents that 
included this one and asked people to send input to the IAB before the 
drafts were turned into RFCs.

Maybe the IETF that you want to work in doesn't let the IAB publish its 
own documents, even after soliciting review on this list, but that is 
the IETF you have been working in for as long as we have known each 
other.

--Paul Hoffman

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 19 Sep 2017 17:28:12 +0200

On Sep 19, 2017, at 16:29, Paul Hoffman <[email-masked]> wrote:
> The drafts leading to the document that specified this action, RFC 7994, was discussed on an active mailing list with lots of members. The IESG asked, on this list, for people to review the set of documents that included this one and asked people to send input to the IAB before the drafts were turned into RFCs.

Yes, and it is of course my fault for not paying attention.  (I’m not making a process appeal here or anything.  I’m just trying to make more people aware that this is not a good way forward.)

But then, “be liberal in abusing UTF-8” wasn’t one of the stated objectives of the work, so maybe I can be forgiven for paying close attention to the details only now, and for being disappointed in what I found.

> Maybe the IETF that you want to work in doesn't let the IAB publish its own documents, even after soliciting review on this list, but that is the IETF you have been working in for as long as we have known each other.

We publish many documents and we make many mistakes while doing so.  That is a result of running an organization with humans.
And of course the IAB can publish its own documents, and is as entitled to make its own mistakes as anyone else here.

The IETF I have been working for (and the IAB I have been working with) generally has been way above the average of other SDOs in deflecting the reflex of justifying past decisions and instead fixing what needs to be fixed.  I wonder why justifying the past decision is taking so much more room here than thinking about the issue and the effects of this decision.

The IETF also has been reasonably effective in thinking about the signals its decisions send.  This hasn’t worked so well here.  (Maybe its a matter of thinly spread experience — maybe too few people here work with implementers that will soak up this signal giddily as their future excuse for not doing the work.)

Anyway, there is very little that can be done about this right now, except maybe for noting the issue.  RFC 8187 is published in its present form and cannot be changed.  RFC 7994 will probably be considered the specification in force until it is superseded (thankfully, we have a moratorium for beyond-ASCII RFCs for other reasons).  But maybe it has become clear that there is at least a vocal minority here that thinks using UTF-8 right not just in our protocols but even in our documents is way more important than minimizing the theoretical amount of mojibake for the largest number of people that enjoy viewing plain text RFCs while employing legacy software for that.

Grüße, Carsten

---

From: John C Klensin <[email-masked]>
Date: Tue, 19 Sep 2017 08:49:54 -0700

Ted, Carsten, and others,

I'm on travel and using a borrowed machine and have not been
able to follow this rather long thread carefully.  Apologies if
what I'm about to say rehashes issue covered by others, but...

(1) As far as I know, we still believe in running code,
especially widely-deployed running code.  I hate the idea of
BOMs in Utf-8 and have seen the harm, but, if there are widely
deployed and heavily used applications out there that depend on
it, our breaking those applications is just not what we should
be doing.

(2) I note that Dave's tests applied to Microsoft bundled
applications.   If they are the main problem, then Microsoft
should be ashamed of themselves for updating those applications
to handle non-ASCII codes and then violating the clear rules for
UTf-8 (if they allow UTF-8 at all -- if they decided to not do
that and only allow, e.g., UTF-16, that would be a different
matter).  While I hope bug reports have been filed, the IETF (or
RFC Editor) setting out to break those applications is just not
what we do.

(3) As someone who does normally read RFCs in ASCII (and
plain-text, see below), I consider statements such as "No
sensible person reads an ASCII RFC..." to be demeaning,
insulting, and inappropriate in the IETF.   My reasons for doing
that are not important here (if anyone cases, write me offlist)
and I am not advocating them for anyone else, but I believe we
can have useful discussions here without asserting that people
in the community, especially ones who occasionally manage to
make presumably-useful contributions, are not sensible.  I
suppose I should add that I don't use any of the applications
Dave listed to read RFCs or, normally, any other ASCII files.  

(4) At the same time, if the complaint is about terrible
typography, that is a complaint about plain-text files without
any formatting controls and markup, not about ASCII.  If someone
dislikes plain-text files, they should, IMO, be looking for a
way to do something else (e.g., PDF or HTML), not trying to
"fix" plain-text files. 

(5) Failure to carefully read and study the documents that
specify new RFC formats has been a complex matter for me, one
that because complex when the IAB took actions that I
interpreted as not being interested in my expertise or opinions
on the matter.  I believe the IAB is not the IESG and has the
right to independent decisions that are inconsistent with IETF
consensus and, indeed, to publish statements and RFCs that claim
the consensus is ill-informed or just plain wrong.  That
includes feeling a need to defend their right to ignore my
advice and even to make it clear that they aren't interested.
If that result is that participation becomes too burdensome for
me, so be it.  And, if I don't think the IAB's conclusions or
style in that regard are appropriate, my main recourse is to try
to convince the Nomcom that the IAB membership is in need of an
overhaul.

(6) If any of the new norms and tools result in plain-text files
with only ASCII characters in them starting with a BOM because
ASCII is just a subset of UTF-8, I'd consider that seriously
broken, a violation of the ASCII standard, and a few other
things.  I hope tools and test suites would check for that case
and complain if it is encountered. 

And, yeah, I think some (perhaps many) of us are going to need
to have simple BOM adding and removing tools around, just as we
have had tools that convert from LF-only to CRLF formats handy
and get to use them often (I note, e.g., that the online
version, of XML2RFC, used to generate and save a file using
several browsers on Windows produces LF-delimited lines).  

best,
    john

--On Tuesday, September 19, 2017 8:39 AM -0400 Ted Lemon
<[email-masked]> wrote:

> Carsten, actually I think the main deciding factor was that
> text file presentations of RFCs are notionally legacy, not
> current.   No sensible person reads an ASCII RFC, because the
> typography is terrible.   So we don't have to care about this:
> in the long run, the ASCII RFC is dead anyway.
> 
> The reason I jumped on your bandwagon is that I actually agree
> with you that the IETF is sending a message by doing this, and
> I agree that it's the wrong message.   It would be nice if we
> could craft a solution that works for naive users of ASCII
> RFCs, and is comprehensible to non-naive users like Adam.
> 
> But if we don't get sufficient consensus to talk Heather out
> of this policy, the earth will continue to rotate around the
> sun.   :)
>

---

From: Ted Lemon <[email-masked]>
Date: Tue, 19 Sep 2017 13:02:25 -0400

Hey, John.   I'm a little disappointed that you think I think "sensible" is a compliment, and "not sensible" is a pejorative.   That comment was directed at Adam, who is on the IESG, hence "not sensible."   It just means that we are all a bunch of people who are are more interested in exploring and tooling to do things in ways that work for us than in doing what everybody else does.   That's why we are good at developing standards.   But we are not our target market.

That said, I just did the exercise of having the latest version of Windows 10 download a file with "Content-type: text/plain; charset=utf8", and it doesn't put a BOM on it.   Edge displays it correctly; MSIE does not.   If you download it with Edge, WordPad and Notepad both screw it up, although the dev editor gets it right.

This is really terrible, and Microsoft ought to fix it.   There is no excuse for using the iso-8859 encoding when you've been explicitly told the encoding is utf8, and if utf8 is not the default encoding, then when you download a file encoded that way, you have to do what it takes to make sure the file is processed correctly.

Windows is the only operating system I tested with this problem.   MacOS does fine.   Debian does fine.   So I think there is actually a serious problem here in need of correcting, but I don't think it's something we have the power to correct, other than in the sense that I hope someone from Microsoft is reading this and either nodding in agreement because they've been trying to get this fixed and haven't been able, or because they see it as an action item.

Anyway, I think we should stop arguing about this right now, and the people who have the power to fix the one remaining BOM offender should try to do that.   And we should revisit it again in a few years.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Sep 2017 19:05:04 +0200

+1

> RFC Editor) setting out to break those applications is just not

Microsoft's support for non-ASCII characters predates Unicode (AFAIU). 
Notepad has been dealing with non-ASCII characters for ages.

Not *defaulting* to UTF-8 is not a bug. It may not be what our 
preference is nowadays, but that's it.

> way to do something else (e.g., PDF or HTML), not trying to

Nobody is doing that, as far as I can tell. And yes, we'll have official 
HTML variants with better typography. In the meantime, people can look 
at unofficial ones.

> and complain if it is encountered.

I'd consider that a feature, far better than adding it on a case-by-case 
basis. And no, it's not a violation of the ASCII standard, as that 
standard wouldn't apply anymore.

> and get to use them often (I note, e.g., that the online

dos2unix does this very well.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Sep 2017 19:16:53 +0200

> But then, “be liberal in abusing UTF-8” wasn’t one of the stated objectives of the work, so maybe I can be forgiven for paying close attention to the details only now, and for being disappointed in…

I don't see any abuse of UTF-8.

> The IETF I have been working for (and the IAB I have been working with) generally has been way above the average of other SDOs in deflecting the reflex of justifying past decisions and instead fixi…

I can only speak for me: I'm justifying the decision because I believe 
it was the right one, and I happen to disagree that the negative effects 
outweigh the benefits. If you want to convince me that the decision was 
wrong, it would be helpful if you gave concrete examples of breakage - 
not in theory, but in practice, not for *any* content, but for RFCs in 
text/plain format.

> The IETF also has been reasonably effective in thinking about the signals its decisions send.  This hasn’t worked so well here.  (Maybe its a matter of thinly spread experience — maybe too few peop…

With all due respect, I have no idea what you're talking about.

> Anyway, there is very little that can be done about this right now, except maybe for noting the issue.  RFC 8187 is published in its present form and cannot be changed.  RFC 7994 will probably be considered the specification in force until it is superseded (thankfully, we have a moratorium for beyond-ASCII RFCs for other reasons)...

Actually, more are coming 
(<https://www.rfc-editor.org/cluster_info.php?cid=C326>).

> But maybe it has become clear that there is at least a vocal minority here that thinks using UTF-8 right not just in our protocols but even in our documents is way more important than minimizing the theoretical amount of mojibake for the largest number of people that enjoy viewing plain text RFCs while employing legacy software for that.
> ...

Can you please point to *something* that says it's wrong to use the BOM 
in UTF-8 encoded documents of type text/plain?

Best regards, Julian

---

From: John C Klensin <[email-masked]>
Date: Tue, 19 Sep 2017 10:17:41 -0700

--On Tuesday, September 19, 2017 7:05 PM +0200 Julian Reschke
<[email-masked]> wrote:

>> should be ashamed of themselves for updating those
>> applications to handle non-ASCII codes and then violating the
>> clear rules for UTf-8 (if they allow UTF-8 at all -- if they
>> decided to not do that and only allow, e.g., UTF-16, that
>> would be a different matter).  While I hope bug reports have
>> been filed, the IETF (or RFC Editor) setting out to break
>> those applications is just not what we do.
> 
> Microsoft's support for non-ASCII characters predates Unicode
> (AFAIU). Notepad has been dealing with non-ASCII characters
> for ages.

Understood, my opinions about how well that worked, especially
for non-Latin scripts, notwithstanding.  But, again, using BOM
as a substitute for charset=UTF8, is, at least IMO, not the
brightest of ideas even though I'm also aware that we had a
difficult transition when the web went from a default of Latin-1
to Unicode.
 
> preference is nowadays, but that's it.

See about.  Slightly different discussion.   But I note that it
isn't hard to distinguish between Latin-1 and UTF-8 without
relying on BOM -- the hard problem there involves distinguishing
between the various species of 8859 and assorted code pages.

>> any formatting controls and markup, not about ASCII.  If
>> someone dislikes plain-text files, they should, IMO, be
>> looking for a way to do something else (e.g., PDF or HTML),
>> not trying to "fix" plain-text files.
>> ...
> 
> Nobody is doing that, as far as I can tell. And yes, we'll
> have official HTML variants with better typography. In the
> meantime, people can look at unofficial ones.

Exactly.

>> ...
>> (6) If any of the new norms and tools result in plain-text
>> files with only ASCII characters in them starting with a BOM
>> because ASCII is just a subset of UTF-8, I'd consider that
>> seriously broken, a violation of the ASCII standard, and a
>> few other things.  I hope tools and test suites would check
>> for that case and complain if it is encountered.
> 
> I'd consider that a feature, far better than adding it on a
> case-by-case basis. And no, it's not a violation of the ASCII
> standard, as that standard wouldn't apply anymore.

But there I believe there has been strong community consensus
for preserving the ASCII format and coding for plain-text files
(or at least one type of plain-text files that do not contain
(substantive) non-ASCII characters.  Won't bother me much (my
plain-text tools work well with and without BOM), but I'd expect
that, if the IETF made a decision to dump ASCII entirely, some
people would look for other places to get work done.  I don't
believe that would be in IETF's, or the Internet's best
interests.  YMMD.

>> to have simple BOM adding and removing tools around, just as
>> we have had tools that convert from LF-only to CRLF formats
>> handy and get to use them often (I note, e.g., that the online
>  > ...
> 

Indeed it does.   And that is more or less what I was trying to
say.  I keep dos2unix around, if I needed it, I'd keep some sort
of BOM-remover around.  And I wouldn't expect to invest a lot of
energy into having to use the latter, just as I don't invest a
lot of energy complaining about having to use dos2unix.

best,
    john

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Sep 2017 19:28:06 +0200

> difficult transition when the web went from a default of Latin-1

Where is "charset=..." supposed to come from when you open a file resource?

> between the various species of 8859 and assorted code pages.

I agree that Notepad *could* be (heuristically) sniffing for UTF-8, and 
it would be interesting to hear why Microsoft doesn't do that.

Best regards, Julian

---

From: Lyndon Nerenberg <[email-masked]>
Date: Tue, 19 Sep 2017 11:22:41 -0700 (PDT)

> (In case that was too flippant, my point is that RFC 4217 is really what 
> we now call opportunistic security, and any kind of actual 
> authentication of the endpoint is addressed in the document with a 
> handwave.)

But that basically declares everything that uses a STARTTLS-like machanism 
'insecure'.  Does that mean they're flagging IMAP, POP, and 
SMTP/Submission as insecure, too?

If DNSSEC says the A record for foo.example.net is valid, and the FTP TLS 
negotiation gives me a cert with a CN=foo.example.net, I'd say that's more 
than hand waving.

--lyndon

---

From: Ted Lemon <[email-masked]>
Date: Tue, 19 Sep 2017 14:34:54 -0400

On Sep 19, 2017, at 2:22 PM, Lyndon Nerenberg <[email-masked]> wrote:
> But that basically declares everything that uses a STARTTLS-like machanism 'insecure'.  Does that mean they're flagging IMAP, POP, and SMTP/Submission as insecure, too?

That's correct.   Anything that uses STARTTLS and doesn't fail when STARTTLS doesn't get a TLS connection with a cert that validates is not secure.

> If DNSSEC says the A record for foo.example.net is valid, and the FTP TLS negotiation gives me a cert with a CN=foo.example.net, I'd say that's more than hand waving.

Yes, but the RFC doesn't say to do that.   It says you *could* do that, but doesn't even say so in sufficient detail to implement.   If it's not a requirement that causes interop to fail when it's not followed, it's not secure enough to trust.

This is a bit nitpicky—the operational environment in which the RFC was approved was quite a bit different than the present.   The point is simply that if we want to ensure that people are downloading untainted versions of RFCs, we can't use FTP.   I believe that at least the Chrome dev team is officially moving in this direction.  Dunno if Chrome implements TLS for FTP, though.   If it did, then it could do as you say and trust the destination based on a PKI match; if this were true of all browsers, then I would say that we should set up our FTP server to do that.   :)

---

From: Ted Lemon <[email-masked]>
Date: Tue, 19 Sep 2017 14:35:38 -0400

On Sep 19, 2017, at 1:16 PM, Julian Reschke <[email-masked]> wrote:
> Can you please point to *something* that says it's wrong to use the BOM in UTF-8 encoded documents of type text/plain?

It's pretty clearly wrong to download a document that's labeled "text/plain;charset=utf8" and then store it in a way that will result in it being treated as having a different encoding, or to display it directly using a different encoding, as Explorer does.   Since the BOM is not required or even encouraged by the Unicode Consortium, failing to get this right is clearly a bug.

---

From: Julian Reschke <[email-masked]>
Date: Tue, 19 Sep 2017 21:19:16 +0200

On 2017-09-19 20:35, Ted Lemon wrote:
> On Sep 19, 2017, at 1:16 PM, Julian Reschke <[email-masked] 
>> Can you please point to *something* that says it's wrong to use the 
>> BOM in UTF-8 encoded documents of type text/plain?
> 
> It's pretty clearly wrong to download a document that's labeled 
> "text/plain;charset=utf8" and then store it in a way that will result in 
> it being treated as having a different encoding, or to display it 
> directly using a different encoding, as Explorer does.   Since the BOM 
> is not required or even encouraged by the Unicode Consortium, failing to 
> get this right is clearly a bug.

I'm pretty sure that if the browser modified the downloaded file, 
somebody else would claim that *that* would be a big. For instance, it 
would affect signatures.

Best regards, Julian

---

From: Doug Royer <[email-masked]>
Date: Tue, 19 Sep 2017 13:44:43 -0600

On UNIX/Linux, if I set my LANG environmental variable to 'en_US.UTF-8', 
tools work. 'more' starts the files off with a weird character. I can 
not find another tool that fails in any way.

If I set LANG to 'C' (the default), many tools display strange 
characters with they encounter non-ASCII.

I had set my LANG to UTF-8 years ago when playing with calsch files, and 
would recommend that everyone change to UTF-8 on UNIX/linux.

I do not even own a Microsoft box, if I did, I would try cygwin, or 
mingw on that OS.

-- 

Doug Royer - (http://DougRoyer.US  http://goo.gl/yrxJTu )
[email-masked]
714-989-6135

---

From: Ted Lemon <[email-masked]>
Date: Tue, 19 Sep 2017 16:15:44 -0400

On Sep 19, 2017, at 3:19 PM, Julian Reschke <[email-masked]> wrote:
> I'm pretty sure that if the browser modified the downloaded file, somebody else would claim that *that* would be a big. For instance, it would affect signatures.

Yet another reason why depending on BOMs is a bug.   Text files are hard.   If you want to sign a text file, you have to know the encoding, and you have to ignore the BOM if it's present, or else the signature doesn't work for equivalent versions of the file, only for the exact binary form of the file that was signed.   This is precisely the point Carsten's been making: BOMs are harmful.   There is no way to win with them.   E.g., if a file is UTF-8, what is the signature?   Is it on the actual bits in the file?   The unicode?    Is it okay if when you do EOL conversion, the signature breaks, or does the signature have to ignore EOL characters?   What about whitespace?

We have solved this problems numerous times, and AFAIK we haven't solved it by signing the binary, with the possible exception of subresource integrity.  This works because the data is never not under control of the thing that's responsible for validating it.   OpenPGP requires canonicalization, e.g. see section 5.2.4 of RFC 4880.

Here we are talking about data blobs that are downloaded by one thing and used by another.   They can be canonicalized in a variety of ways.   If we were to specify signatures for RFC text files, we would have to have a specification that says how the signature is computed. An interesting task, perhaps, but kind of orthogonal to this discussion.   Of course we could just say "sign the representation format," but in doing so we would be totally punting on interop.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 19 Sep 2017 16:17:29 -0400

This appears to be the default on Ubuntu—I didn't have to do anything to get cat to work in the terminal, for example.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 20 Sep 2017 08:26:59 +1200

> somebody else would claim that *that* would be a big. For instance, it 

Yes. If you consider that the exact content of a file is what you want
when you transfer it from another machine, hidden changes to the content
are plain wrong. Storing metadata with the file is another matter - in
an ideal world you would store "text/plain;charset=utf8" as part of the
metadata. But that isn't the world we live in; all we store for sure
is the filename, which may include the string ".txt", which may mean
the same as "text/plain" but certainly doesn't imply "charset=utf8".

In any case, regardless of how we believe UTF-8 *strings* should be
embedded in protocols, the decision to prepend the equivalent of
"charset=utf8" to a file containing a UTF-8 string is not a protocol
issue.

Modest suggestion: store BOM-free versions named rfc8187.utx etc.
Start a trend that tool implementers can follow.

   Brian

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 20 Sep 2017 07:06:42 +1000

On 20 Sep. 2017 03:29, "Julian Reschke" <[email-masked]> wrote:

> between the various species of 8859 and assorted code pages.

I agree that Notepad *could* be (heuristically) sniffing for UTF-8, and it
would be interesting to hear why Microsoft doesn't do that.

Historically, because Windows uses/d UTF-16. See this decade old blog post,
and particularly note the `dir > results.txt` snippet [1]

By the way, when it comes to Notepad's heuristics, create a text file that
says "Bill fed the goats" (without the quotes), then save and open it.
Unless IsTextUnicode has been updated recently, this should break the
sniffer.

[1] https://blogs.msdn.microsoft.com/oldnewthing/20070417-00/?p=27223

Cheers
--
Matthew Kerwin

---

From: George Michaelson <[email-masked]>
Date: Wed, 20 Sep 2017 08:32:55 +1000

I have a meta-question. If we wanted to treat MS as a first class
engaged entity and lodge a report, what is the channel we can use,
which doesn't either get ignored, or because it comes in too "low"
gets missed, or worse, repudiated on the basis 'it must be wrong
because we are right'

I have tried this in the past in other contexts, such as X.509. The
response I got was "it is very hard to find a channel into microsoft
with sufficient seniority to effect change, who is willing to listen
to you"

Basically, anyone from microsoft who comes to the IETF does so as a
subject matter expert and has no influence outside their specific
domain. Inside their domain, their influence is strictly limited by
product process. So, on that basis, if we identify problems with
microsoft product and BOM, We cannot assume any named entity with
@microsoft.com on their affiliation email can help us.

The question stands: what channel as an SDO, does the IETF have, to
report to MS corporate they have a problem in UTF text handling?

This is only partly a rhetorical question. My suspicion is, there is
nobody, and we cannot effect change as an SDO, or even as individuals,
except in as much as somebody senior inside MS choses (and I stress
choses) to take notice.

If thats not the case, I'd love to know.

(I left John K on the cc list because past history of activity in 7
and 8 bit character standardisation and subsequently glyphs and
homophones suggests he might know both the subject matter experts, and
senior people in MS)

-G

---

From: Ted Lemon <[email-masked]>
Date: Tue, 19 Sep 2017 21:35:58 -0400

> except in as much as somebody senior inside MS choses (and I stress

I believe the extent to which we can affect Microsoft is simply that we might have access to influential people in Microsoft.   Might.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 22 Sep 2017 14:51:46 +0200

> RFCs with non-ASCII characters in the final text. In order to keep

Out of curiosity: as the current version of xml2rfc produces the UTF-8 
plain text just fine (*), what additional processing are you referring to?

> until the new format tools are in production.

Understood.

Now what does this mean for drafts being written right now, and which 
are not expected to be ready for publication in the next, let's say, 12 
months? Can we start submitting I-Ds with non-ASCII characters right now 
(without getting stopped by id-nits...)?

Best regards, Julian

(*) with the possible exception of the BOM :-)

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Fri, 22 Sep 2017 09:54:17 -0700

> plain text just fine (*), what additional processing are you referring

There are the manual checks to determine if everything renders exactly
as expected. There are a number of discussions as we learn how to deal
with half- and full-width characters in the same document, and try to
determine the best path forward to make such characters in tables to
line up properly. There are even more discussions as we learn more about
invisible space characters outside of the ASCII realm and try to
understand if those are something we need to write scripts to identify.
xml2rfc is just one component of getting these documents published.

> are not expected to be ready for publication in the next, let's say,
> 12 months? Can we start submitting I-Ds with non-ASCII characters
> right now (without getting stopped by id-nits...)?

That's an excellent question. I think this is something to discuss with
both the IESG and the Tools Team, though I am fairly confident that
"right now" isn't possible. An idnits update is one of the tools
contracted for, and is not yet complete (see the expected timeline here:
https://trac.tools.ietf.org/tools/ietfdb/wiki/FormatToolsPlan). Also,
we've seen that the datatracker isn't rendering non-ASCII characters
correctly in the PDF automatically generated from RFC 8187
(https://trac.tools.ietf.org/tools/ietfdb/ticket/2370). So, there is
still some work to do before this will all work.

-Heather

---

From: Julian Reschke <[email-masked]>
Date: Fri, 22 Sep 2017 19:14:14 +0200

> as expected. There are a number of discussions as we learn how to deal

Ok.

> line up properly. There are even more discussions as we learn more about

Seriously: don't bother.

> understand if those are something we need to write scripts to identify.

If this is understood to be a problem, let's have a script that checks 
for these characters.

> https://trac.tools.ietf.org/tools/ietfdb/wiki/FormatToolsPlan). Also,

Well, idnits s just advisory. All it can block is automatic submission. 
(I wish it did not).

> still some work to do before this will all work.

Looking at the PDF linked from the datatracker 
(<https://www.rfc-editor.org/rfc/pdfrfc/rfc8187.txt.pdf>), I don't see a 
problem.

That said: I don't think that a minor bug in a conversion tool for a 
supplemental output format is sufficient reason to block progress...

Best regards, Julian

---

From: Martin Thomson <[email-masked]>
Date: Mon, 25 Sep 2017 16:22:20 +1000

> A *pollution-intolerant* tool is one that can do UTF-8, but reacts adversely to UTF-8 with BOMs.  Many tools that look for text signatures to do some form of file type detection are foiled by BOMs.…

Since I was invoked, I've fixed it.

BTW, markdown != text, so I think that we're entitled to be intolerant
of crap on input; indeed your own tool was intolerant until I reported
the bug.

Having to fix every single tool in existence is a nuisance.  I want to
be able to read character[0] and get a character.  In this case it was
cut(1) passing the problem down to the next fool (me), but grep with
'^---' would have failed too.

---

From: John C Klensin <[email-masked]>
Date: Tue, 26 Sep 2017 04:03:27 -0400

--On Monday, September 18, 2017 18:16 -0500 Adam Roach
<[email-masked]> wrote:

> Today, as we speak, I have a copy of the RFC repository on my
> hard drive. (To be precise, I have it on most of the hard
> drives of the various machines that I use). For my current
> workflow, I *think* all of them got there via rsync, although
> it's possible that some of them are still using an old
> wget-based setup. It's kind of immaterial how they got there,
> because a careful examination of them would show the same
> result between the two methods (and any others I could think
> of, including FTP mirroring and manually downloading via web
> browsers): it's a sequence of bytes, with a ".txt" file
> extension; identical, regardless of which tool downloaded
> them. There is nothing else about the file to indicate its
> encoding.[1]
> 
> Okay. So, now, I open up the local file browser to that file
> on my hard drive, and double-click on an RFC. An application
> is launched. Let's say that application is Wordpad. How does
> it know which character encoding to use for this file?

It doesn't and the presence of absence of a pair of octets it
might interpret as a BOM just feeds another heuristic.   Keep in
mind that, if the content of that file were in 8859-1, that
could be interpreted as Small Thorn followed by small y with
diaeresis (the characters Unicode codes as U+00FE and U+00FF).
Or course, it if were coded in ISO 8859-5 (or 6, 7, 8, 11,
etc.), there would be different interpretations.  I note that
several versions of Wordpad will get just about equally confused
if your rsync (or whatever) fetch results in an object in your
local, Windows-ish, file system with LF as an EOL rather than
CRLF.  

Conventions about file names suffixed in ".txt" have worked as
well as they have only because it has been possible to assume
that ".txt" implies ASCII.  From the early days of the net, even
that has not been perfect, not just because EOL=LF has existed
since early on (IIR, the first version of ASCII required it),
but because there was that EBCDIC problem.  The only real
solution to these problems is files that carry their own
descriptions (the idea of a two-part, or even three-part, file
where one part is a description predates its adoption by Apple
my many years).  Otherwise, it is all heuristics and the other
strong argument against BOM as a "this is UTF-8" clue is that it
often won't work.

I got over believing that it was reasonable to try to abstract
file description information down to a few characters and then
embed it in the file name in the early 1970s, but obviously lost
that battle long ago.   Perhaps, if we need to indicate what is
UTF-8 and what isn't, we should start suffixing files with
".utf8" or, if people like three character suffixes, ".ut8" or
".uf8", rather than relying on in-file indicators that violate
the relevant standards, don't adequately identify the relevant
CCS, and invite assorted file concatenation problems, etc.

    john

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Sep 2017 11:19:59 +0200

On 2017-09-26 10:03, John C Klensin wrote:
> ... > ".uf8", rather than relying on in-file indicators that violate

Precisely what relevant standard is violated by inserting a BOM?

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Sep 2017 11:39:17 +0200

> On Sep 26, 2017, at 11:19, Julian Reschke <[email-masked]> wrote:
> 

The violation is a bit subtle, maybe.

All relevant standards have made a point about forbidding, or in certain cases, NOT RECOMMENDING but tolerating BOM pollution.

By arguing that RFCs are broken without BOM pollution (exploiting the loopholes created by those certain cases), we are making an end-run around the decisions of the standards.

Why is that relevant?  Because those people writing pollution-expecting applications (i.e., applications that don’t work unless you pollute) now can point to that decision and say “even the IETF is doing it, so we re right in requiring pollution”.  (Which in turn of course will bolster the position of pollution-required etc.)

Standards sometime work through more than explicit language.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Sep 2017 11:40:41 +0200

> All relevant standards have made a point about forbidding, or in certain cases, NOT RECOMMENDING but tolerating BOM pollution.

Once again - which?

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Sep 2017 12:12:42 +0200

> Once again - which?

The most directly relevant standards here are Unicode and STD0063 (RFC3629).
RFC5198 is a useful additional data point.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Sep 2017 12:55:09 +0200

> RFC5198 is a useful additional data point.

Please cite *specifically* what you think is relevant with respect to 
the use of BOMs in plain text files.

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Sep 2017 13:06:51 +0200

On Sep 26, 2017, at 12:55, Julian Reschke <[email-masked]> wrote:
> 
> Please cite *specifically* what you think is relevant with respect to the use of BOMs in plain text files.

That’s all already been said in the thread, but to repeat, with links:

STD0063 section 6:

And RFC 5198, section 2, item 5:
https://tools.ietf.org/html/rfc5198#section-2

Of course, BOM-pollution apologists will find enough rope in these documents to hang themselves.
That is really the problem here: the tendency to weasel around decisions in standards.
(Or to make them in the first place.  UCS-2-BE vs. UCS-2-LE all over again.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Sep 2017 13:26:19 +0200

>       identification of character encoding (such as HTTP).

...which is *exactly* what we're discussing here?

> "Use of a BOM is neither required nor recommended for UTF-8": > http://www.unicode.org/versions/Unicode10.0.0/ch02.pdf

That talks about whether a BOM is or is not useful to distinguish 
between Unicode encoding schemes. But that's not really relevant here, 
unless all plain text files were indeed already in one of the Unicode 
encoding schemes. They are not, and that's the problem.

> https://tools.ietf.org/html/rfc5198#section-2

That has the same problem - it assumes a world that is already fully 
Unicode, in which case it's correct to say that the BOM is not needed.

However, plain text files are something that predates all of this, and 
the tools that the consumers of plain text RFCs use deal with this mixed 
encoding world in several ways.

I agree that if the goal was to promote an all-unicode world, the answer 
would be different. But the goal of the RFC Editor is to deliver 
documents that people will be able to read properly with the tools they 
have. The tests we did showed that adding the BOM is beneficial for this.

(That said: this is a in-between period - once the transition to the 
format is finished, the preferred consumption format will be HTML anyway)

> (Or to make them in the first place.  UCS-2-BE vs. UCS-2-LE all over again.)

My point being: none of the things you apparently refer to applies to 
what we are discussing here.

Best regards, Julian

BTW: if you believe that the text *I* quoted from RFC3629 is bad, you 
might want to submit an erratum and/or start a discussion on updating 
the document.

---

From: Matthew Kerwin <[email-masked]>
Date: Tue, 26 Sep 2017 22:24:25 +1000

And so we circle around again...

On 26 September 2017 at 21:06, Carsten Bormann <[email-masked]> wrote:

> ​

RFC 3629 essentially boils down to: A protocol SHOULD forbid the BOM if
UTF-8 is mandated, or otherwise signalled.  That and all the other guidance
seems to say: guessing is bad, don't guess (even with heuristics) if
there's any other way.  For HTTP there is another way (content-type with
charset).  What's the other way for a .txt file on my FAT32 thumb stick?
The standards don't seem to say much about that, because there was no
solution when they were being written, just as there is none now.  (Note
that RFC 5198 doesn't apply to files on my thumb stick since it's not
internet-enabled, and similarly 3629 talks about "protocols" but I don't
think fopen();read() counts as a protocol.)

You're arguing that the standards say "BOM is bad", while I read them as
saying "guessing is bad."  Building from that position, I see the .txt
extension + BOM magic number combination as a better signal (i.e. less
guessy) than the extension alone that a file is likely a UTF-8 encoded
plaintext file.  I'm well aware that makes me a "BOM-pollution apologist",
but I haven't been offered anything better so far.

Somewhere in here there seems to be an argument for having richer metadata
capabilities in file systems, and/or more intelligent downloading tools
that translate between the web's metadata and the file system's (whether
that's rich metadata, or file name patterns, or magic numbers, or
whatever).  And if so, I agree enthusiastically.  Unfortunately I'm not in
a position to much about that.  The penultimate argument seems to be: serve
the RFCs over HTTP as text/plain;charset=utf-8 **without** the BOM (as per
standards), and make the human deal with it if their downloading/viewing
tools won't do the Right Thing™ (as per tradition in IT).  Given that RFCs
are usually read by technically savvy people, I guess that's live-withable.

I'm not going to go flipping any tables whichever way this discussion ends
up;  I prefer the HTML versions anyway (even if they're non-canonical.)

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Sep 2017 15:09:28 +0200

> ...which is *exactly* what we're discussing here?

This is only the pertaining case if we want that to be so.

The “protocol” could be “RFCs are in UTF-8”.  Done.

> I agree that if the goal was to promote an all-unicode world, the answer would be different. But the goal of the RFC Editor is to deliver documents that people will be able to read properly with the tools they have.

The browsers already work with Unicode, so that goal is already achieved.

But there is a fundamental disconnect here:  I *don’t* agree that the goal of the RFC editor is to maximize the viewing pleasure for RFCs.  I believe we are creating RFCs to make the network work better, and we should focus on that goal when making decisions.  The objective to move on to Unicode (UTF-8) has been in the IETF DNA for two decades now, and it is sad when there are places where we are losing sight of this.

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Tue, 26 Sep 2017 15:23:19 +0200

> The “protocol” could be “RFCs are in UTF-8”.  Done.

If the "protocol" was that, than of course a BOM wouldn't be needed. But 
it is not.

> The browsers already work with Unicode, so that goal is already achieved.

"With the tools they have" as in "they are getting the files (http 
download, rsync, ftp, whatnot) and open them locally with whatever is 
registered for *.txt".

> But there is a fundamental disconnect here:  I *don’t* agree that the goal of the RFC editor is to maximize the viewing pleasure for RFCs.  I believe we are creating RFCs to make the network work b…

It is indeed a fundamental disconnect.

I recommend that you review the discussion that got us where we are, and 
I believe you'd be pleased that we actually got to the point where we 
could convince people to move away from plain ASCII.

Best regards, Julian

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 26 Sep 2017 09:25:39 -0400

​
On Tue, Sep 26, 2017 at 4:03 AM, John C Klensin <[email-masked]> wrote:

I agree with John on this. ,txt is broken and is going to remain so.

To understand how broken it is, my main use for notepad is to generate one
off batch files. Notepad throws the BOM in without asking and command.exe
chokes.

I always go to a HTML version if I can, even if it is only caveman format
because it is more likely to work printed out.

The RFC format does not concern me much because I spend most of my time
reading drafts that look like this:

http://www.prismproof.org/Documents/draft-hallambaker-mesh-architecture.html

​Yes, I know that is not RFC format. It has colour and the table of
contents works and several other things that have been fixed. ​But it is a
document I can send to my engineers and not be ashamed of the format. The
plaintext RFC format will always be unacceptable with or without a BOM.

​Lets just move to the proposed HTML format as fast as possible and then we
can set about fixing that.

​

---

From: "tom p." <[email-masked]>
Date: Tue, 26 Sep 2017 18:10:58 +0100

----- Original Message -----
From: "Matthew Kerwin" <[email-masked]>
To: "Carsten Bormann" <[email-masked]>
Cc: "Julian Reschke" <[email-masked]>; "IETF" <[email-masked]>
Sent: Tuesday, September 26, 2017 1:24 PM

On 26 September 2017 at 21:06, Carsten Bormann <[email-masked]> wrote:

> That is really the problem here: the tendency to weasel around
decisions
> in standards.

UTF-8 is mandated, or otherwise signalled.  That and all the other
guidance

plaintext file.  I'm well aware that makes me a "BOM-pollution
apologist",

Somewhere in here there seems to be an argument for having richer
metadata
capabilities in file systems, and/or more intelligent downloading tools
that translate between the web's metadata and the file system's (whether
that's rich metadata, or file name patterns, or magic numbers, or
whatever).  And if so, I agree enthusiastically.  Unfortunately I'm not
in
a position to much about that.  The penultimate argument seems to be:
serve
the RFCs over HTTP as text/plain;charset=utf-8 **without** the BOM (as
per
standards), and make the human deal with it if their downloading/viewing
tools won't do the Right Thing™ (as per tradition in IT).  Given that
RFCs
are usually read by technically savvy people, I guess that's
live-withable.

I'm not going to go flipping any tables whichever way this discussion
ends
up;  I prefer the HTML versions anyway (even if they're non-canonical.)

<tp>

With syslog, RFC5424, BOM is mandated when the message is encoded in
UTF8, but that is a protocol and not a filestore.

We do have metadata in a filestore.  As has already been pointed out,
the suffix .txt indicates ASCII and not UTF8 so applications that rely
on that for genuine plain text will likely fail.  As Brian pointed out,
sowhat we need now is a new suffix for UTF.

Tom Petch

Cheers

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 Sep 2017 08:38:03 +1300

On 27/09/2017 06:10, tom p. wrote:
...
> sowhat we need now is a new suffix for UTF.

So why don't we, the Internet standards people who believe in rough
consensus and running code, request the RFC Editor (a friend of ours)
to supply two text versions of each RFC, like

https://www.rfc-editor.org/rfc/rfc8187.txt   as today, with BOM if relevant
https://www.rfc-editor.org/rfc/rfc8187.ut8   containing pure UTF-8 with no BOM ever

It might just become an Internet meme.

    Brian

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 27 Sep 2017 06:12:44 +1000

On 27 Sep. 2017 3:14 am, "tom p." <[email-masked]> wrote:

----- Original Message -----
From: "Matthew Kerwin" <[email-masked]>
To: "Carsten Bormann" <[email-masked]>
Cc: "Julian Reschke" <[email-masked]>; "IETF" <[email-masked]>
Sent: Tuesday, September 26, 2017 1:24 PM

decisions

<tp>

Tom Petch

I think that falls in the category of things I'm not in a position to do
much about (making Notepad handle files differently if they have a
different extension).  If it can be made to work, though, then it becomes
one of the things the human has to manually fix (renaming the downloaded
files) until all the tools catch up and do it for them.

In any case, I'm not sure what we (IETF) should be doing differently.

Cheers
--
Matthew Kerwin

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 27 Sep 2017 06:21:51 +1000

On 27 Sep. 2017 5:39 am, "Brian E Carpenter" <[email-masked]>
wrote:

https://www.rfc-editor.org/rfc/rfc8187.txt   as today, with BOM if relevant
https://www.rfc-editor.org/rfc/rfc8187.ut8   containing pure UTF-8 with no
BOM ever

    Brian

It would be prudent to also include a Content-Disposition header with an
appropriate 'filename' parameter, since browsers tend to name all
downloaded text/plain files as .txt irrespective of the URL.

Not sure if this is bike-shedding or rat-holing.

Cheers
--
Matthew Kerwin

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 26 Sep 2017 22:50:33 +0200

> https://www.rfc-editor.org/rfc/rfc8187.ut8   containing pure UTF-8 with no BOM ever

Because that sends an even stronger signal that the IETF has given up on UTF-8 being the standard text encoding.

A better way to do this — just add:

> https://www.rfc-editor.org/rfc/rfc8187.txt?and-get-me-a-bom-for-that-because-my-environment-cant-handle-utf8-without

(Possibly slightly abbreviated, as in ?agmabftbmechuw.)

So everybody else can still easily wget/curl the undamaged .txt file.

Grüße, Carsten

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 27 Sep 2017 07:41:24 +1000

> https://www.rfc-editor.org/rfc/rfc8187.ut8   containing pure UTF-8 with
no BOM ever

Because that sends an even stronger signal that the IETF has given up on
UTF-8 being the standard text encoding.

Stronger than https://tools.ietf.org/html/rfc6657#section-4 ?

A better way to do this — just add:

> https://www.rfc-editor.org/rfc/rfc8187.txt?and-get-me-a-
bom-for-that-because-my-environment-cant-handle-utf8-without

Grüße, Carsten

That's the same way -- two URLs are two URLs, the exact spelling is paint.

In the background, how does the presentation of file names as URLs interact
with the names of files that are rsynced? Is it a 1:1 mapping (are they
served from the same place)?

Cheers
--
Matthew Kerwin

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 27 Sep 2017 10:51:23 +1300

> served from the same place)?

Pretty much. https://www.rfc-editor.org/retrieve/rsync/ explains.

   Brian

---

From: "tom p." <[email-masked]>
Date: Wed, 27 Sep 2017 08:22:55 +0100

----- Original Message -----
From: "Matthew Kerwin" <[email-masked]>
To: "Brian E Carpenter" <[email-masked]>
Cc: <[email-masked]>; "tom p." <[email-masked]>
Sent: Tuesday, September 26, 2017 9:21 PM

> Not sure if this is bike-shedding or rat-holing.

Matthew

Where in the FTP protocol do I see a Content-Disposition header?

I always use FTP to download RFC because it  (or at least my version
thereof) preserves the metadata whereas my HTTP client (Internet
Explorer) always mangles it.

Do both sounds like an excellent idea!

Tom Petch

---

From: Matthew Kerwin <[email-masked]>
Date: Wed, 27 Sep 2017 17:44:46 +1000

On 27 September 2017 at 17:22, tom p. <[email-masked]> wrote:

> > On 27 Sep. 2017 5:39 am, "Brian E Carpenter"
> ​ ​
> <[email-masked]>
> ​ ​
> wrote:
> >
​Probably the same place in it says you can use FTP to download https
URLs.  But the question of ftp/rsync/other more direct (non-http) methods
of accessing the repository was why I was asking earlier about the
behind-the-curtain storage.  If they all serve from the same directory on
the server, then it'd be best to have renamed/duplicate files (and
appropriate httpd configuration for serving them);  if they're in different
places then there may be other/better options.​

> Explorer) always mangles it.
​Oh aye, that's always been part of my argument.  The tools aren't great,
the filesystems aren't great, nothing is great.  If they were, there
wouldn't be a problem.​

> Do both sounds like an excellent idea!
​Indeed, that's why I wrote "also".  The more annoying holes we patch over,
the fewer people are likely to slip through one of them.

Cheers

---

From: Julian Reschke <[email-masked]>
Date: Wed, 27 Sep 2017 10:01:55 +0200

> Explorer) always mangles it.

Exactly what does it mangle?

Best regards, Julian

---

From: "tom p." <[email-masked]>
Date: Thu, 28 Sep 2017 09:57:59 +0100

----- Original Message -----
From: "Julian Reschke" <[email-masked]>
Sent: Wednesday, September 27, 2017 9:01 AM

> Exactly what does it mangle?

Julian

The obvious one, which disrupts my work, is the Date Created.  FTP gives
me date which is, or is close to, the creation date of the RFC by the
RFC Editor.

Internet Explorer makes the Date Created the date on which I perform the
download, which may be years later and so thoroughly misleading (to me).

So standalone FTP every time.

Tom Petch

---

From: John C Klensin <[email-masked]>
Date: Thu, 28 Sep 2017 18:45:08 -0400

--On Thursday, September 28, 2017 09:57 +0100 "tom p."
<[email-masked]> wrote:

> The obvious one, which disrupts my work, is the Date Created.
> FTP gives me date which is, or is close to, the creation date
> of the RFC by the RFC Editor.
> 
> Internet Explorer makes the Date Created the date on which I
> perform the download, which may be years later and so
> thoroughly misleading (to me).
> 

Because FTP, by design, has a command in the protocol to
transmit a data type (primitive version of content type) and a
canonical form for text on the wire, competent implementations
are also capable of delivering files with EOL conventions, and
even character encodings, appropriate to the receiving
environment.  We learned lessons a _very_ long time ago from
EBCDIC and two (or, depending on how you count, at least four)
different encoding forms for ASCII that lead to that feature and
the TYPE command.

Perhaps unfortunately from where we stand today, the community
effectively decided to discard that feature, with a number of
client implementations deciding that binary transfers (in
FTP-speak, TYPE I) were enough that that receiving systems
should just get exact copies of whatever the sending system had
and sort it out themselves, in the process ignoring the FTP
requirement [RFC959, Section 4.1.2, "Representation Type"] that
the default in TYPE is not specified is ASCII non-print.  While
TFTP [RFC1350] has a similar feature ("netascii" mode), AFAIK,
other FTP alternatives for transferring data under different
conditions, including SFTP over SSH and Rsync, simply assume
image copies are fine.

Trying to transfer files containing non-ASCII characters makes
the problem worse because, while the spec isn't explicit about
it, an FTP implementation should presumably fail if TYPE A is
used and the contents of the data file cannot be interpreted as
ASCII.  Attempts to add a "TYPE U" (for "Unicode" or "UTF-8") to
FTP to solve that problem for another canonical text
representation have gotten absolutely no problem, leading me to
presume that the community has completely lost interest in these
issues.  

Someone who believed in the existence of character coding and
transmission deities and divine retribution from them might
conclude that the community deserves this BOM mess, along with
UTF-16 on the wire, as a result of not dealing with the issue
effectively in FTP, TFTP, and a variety of other transfer
measures.  I couldn't possibly comment on that.

    john

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 29 Sep 2017 10:48:44 -0400

How strange, I have always found FTP to be an absolute dog of a protocol
precisely because it mandates idiot defaults and implementations are
required to perform heuristic hacks. I always used to have to transmit
files twice because the first time the transfer would be corrupted by the
default being 'trash my data in case I am using an IBM machine'.

The separation of the command and data is a good one but separating it over
separate TCP/IP streams is an utter disaster for reliability. None of the
FTPS clients I have found for Windows is able to reliably transmit a Web
site update.

There are much better options, rsync and STFP (SSH FTP) do work well but
unfortunately the hosting provider does not support them which is why I am
leaving my current provider when I get round to it.

FTP to HISTORIC. The time has come.