Subject: Format=flowed quoting (was "Re: IETF...the unconference of SDOs")

From: Randall Gellens <[email-masked]>
Date: Mon, 15 Oct 2012 19:46:36 -0700

At 9:12 AM -0400 9/5/12, Michael Richardson {quigon} wrote:

>  Maybe I'm also concerned because many in the former "elite" have 
> moved to Apple Mail, and it seems that it is bug
>  compatible with Outlook in it's assumption that format=flowed is 
> the default, an act which destroys email quoting, and therefore 
> discussion.

I just noticed this assertion, which is quite false.  Format=flowed 
protects and preserved quoting.  It's the only way to avoid ludicrous 
and impossible to read quoting (which happens after quoted passages 
get line-wrapped at odd points).  Also, as far as I know, Exchange 
does not support format=flowed at all.  My understanding is that it 
insists on HTML quoting, which is entirely different.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly selected tag: ---------------
Once a job is fouled up, anything done to improve it only makes
it worse.

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Fri, 19 Oct 2012 11:42:04 +0100

Warning: this message was generated by Apple Mail.  I just can't stop my insatiable appetite for shiny things.  :-( [1]

On 16 Oct 2012, at 03:46, Randall Gellens <[email-masked]> wrote:

>> Maybe I'm also concerned because many in the former "elite" have moved to Apple Mail, and it seems that it is bug
>> compatible with Outlook in it's assumption that format=flowed is the default, an act which destroys email quoting, and therefore discussion.
> 
> I just noticed this assertion, which is quite false.  Format=flowed protects and preserved quoting.  It's the only way to avoid ludicrous and impossible to read quoting (which happens after quoted passages get line-wrapped at odd points).  Also, as far as I know, Exchange does not support format=flowed at all.  My understanding is that it insists on HTML quoting, which is entirely different.

You're right that this is the virtue of F/F, but that's not what Michael said.  Perhaps it was the context of the quoting, but the discussion refers to the fact that both LookOut and Apple Mail have (as you can see, by looking at this message) a nasty habit of *assuming* an F/F semantic by default.  In other words, they generate hugely long long long long long long long long long lines that only another equally broken client can interpret correctly, but dynamically reflowing those lines, by applying consistent quoting indicators to the lines of such reflowed lines, and so forth.  That, I believe, is a legitimate complaint, because it effectively creates an island between those who have support for Format/Flowed and those who do not.  At least Apple has the decency to use quoted-printable encoding to protect the actual transmitted lines, but it means nothing if competent MIME readers reconstitute the corruption at the other end, by reassembling long lines, and then failing to reflow them.  And it's worse yet if a news reader without MIME tries to quote the lines, complete with Q-P line terminators.  Or if a reply comes from somebody whose editor wraps the lines, but using soft line breaks which are never generated externally, resulting in whole paragraphs being quoted using a single quoting indicator.  And so on, and so on, and so on …

I filed bug 7989556 (Mail uses quoted-printable indiscriminately) with Apple on 16 May 2010.  It was marked as duplicate of 7547565 on 25 May 2010.  The system is closed; you can only see your own bug reports.  I therefore have no idea what's going on with that.  Meantime, I apologise for the inconvenience.  No, really. :-(

Cheers,
Sabahattin

[1] This isn't strictly true.  I'm blind, and Apple are the only guys who have stepped up to accessibility in anything approaching a serious manner, by including the screen reader in the OS and ensuring accessibility across their applications.  For better or worse, they're the only choice for a smartphone, and the only choice for a Windows convert (notwithstanding textmode Linux, of course, which I still enjoy in VMs for braille support).  I'm objective, so this could change, but on both Windows and Mac OS the only really accessible choices are the built-in MUAs (I used to use Pegasus Mail under Windows, when I had the screen reader support for it, although I now understand that Thunderbird has taken its place).

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Fri, 19 Oct 2012 19:46:28 +0100

On 19 Oct 2012, at 17:45, Randall Gellens <[email-masked]> wrote:
> At 11:42 AM +0100 10/19/12, Sabahattin Gucukoglu wrote:
>> Warning: this message was generated by Apple Mail.
> 
> But not using Format=Flowed.

Precisely.

>> You're right that this is the virtue of F/F, but that's not what Michael said.  Perhaps it was the context of the quoting, but the discussion refers to the fact that both LookOut and Apple Mail have (as you can see, by looking at this message) a nasty habit of *assuming* an F/F semantic by default.  In other words, they generate hugely long long long long long long long long long lines that only another equally broken client can interpret correctly, but dynamically reflowing those lines, by applying consistent quoting indicators to the lines of such reflowed lines, and so forth.  That, I believe, is a legitimate complaint, because it e
> 
> This reflects a misunderstanding of Format=Flowed.  Properly generated F=F has lines of no more than 78 characters.  One of the primary goals of F=F is good interoperability between clients that support F=F and those that support traditional pure plain text email.  What you're describing is a symptom of HTML quoting (or a surprisingly poor F=F implementation):

Yes, absolutely.  We do not disagree.

Let's clear up the confusion.  I made two mistakes, firstly by calling this "F/F semantics" when what I mean is some sort of long-line-aware reflowing and quoting.  We'll have to find a name for it.  The other mistake was to call plain text plain text of any description, irrespective of the definition of text/plain.

So we are talking about three formats:
* text/plain, 78 characters wide
* format=flowed, text/plain with soft breaks signalled by trailing whitespace, 78 characters
* text/paragraphs (or whatever), a completely different identity that violates the length limits

Apple Mail and Microsoft use this text/paragraphs.  It's not interoperable with text/plain (because it *isn't* text/plain, as you very correctly note) and therefore it gives everyone a really hard time.  Unless you've not dealt with plain text files from Windows or OS X, you may be sure that "Plain text" most certainly doesn't conform to RFC 822, either.

We should now all be on the same page, I think. :-)

> When generating Format=Flowed text, lines SHOULD be 78 characters or
>   shorter, including any trailing white space and also including any
>   space added as part of stuffing (see Section 4.4).  As suggested
>   values, any paragraph longer than 78 characters in total length could
>   be wrapped using lines of 72 or fewer characters.  While the specific
>   line length used is a matter of aesthetics and preference, longer
>   lines are more likely to require rewrapping and to encounter
>   difficulties with older mailers.  (It has been suggested that 66
>   character lines are the most readable.)
> 
>   The restriction to 78 or fewer characters between CRLFs on the wire
>   is to conform to [MSG-FMT].
> 
> 
>> At least Apple has the decency to use quoted-printable encoding to protect the actual transmitted lines,
> 
> Again, this shows a fundamental misunderstanding of F=F.  Properly generated F=F does not use quoted-printable unless otherwise needed:

Yes.  Apple Mail doesn't do this--it uses QP to keep the line lengths valid in transmission only.  Some mailers, or gateways, wouldn't do that, but would instead spew out long lines that compliant SMTP servers would helpfully corrupt, tail-drop, chop into pieces, etc (for lack of robustness or to protect themselves).

>> but it means nothing if competent MIME readers reconstitute the corruption at the other end, by reassembling long lines, and then failing to reflow them.  And it's worse yet if a news reader without MIME tries to quote the lines, complete with Q-P line terminators.  Or if a reply comes from somebody whose editor wraps the lines, but using soft line breaks which are never generated externally, resulting in whole paragraphs being quoted using a single quoting indicator.  And so on, and so on, and so on Š
> 
> All of these are part of the problem set that F=F was created to fix.  Proper F=F plays nice with pure plain text.

My first exposure to large numbers of affected people was on Usenet, when I sent mail from Apple Mail through a gateway.  Apparently, newsreaders fall into the categories of having no MIME at all, or having MIME but not reflowing text.  In any case, they were unhappy. :-(

> I haven't looked into Apple's mail's generation of either F=F or plain text or HTML.  If it generates F=F with long lines and/or indiscriminate q-p, then those would in my view be bugs that should be fixed.  But I don't know if the symptoms described occur with Apple Mail's F=F or some other format.  To me, they sound like lack of F=F.

At one point--in Leopard (10.5), I think--they were using F=F.  It almost looks like the current behaviour is what's left of the F=F implementation, in that it renders and sends emails that can be flowed, though using text/paragraphs rather than F=F.

Cheers,
Sabahattin

---

From: Randall Gellens <[email-masked]>
Date: Fri, 19 Oct 2012 09:45:12 -0700

>>>  Maybe I'm also concerned because many in the 
>>> former "elite" have moved to Apple Mail, and 
>>> it seems that it is bug
>>>  compatible with Outlook in it's assumption 
>>> that format=flowed is the default, an act 
>>> which destroys email quoting, and therefore 
>>> discussion.
>>
>>  I just noticed this assertion, which is quite 
>> false.  Format=flowed protects and preserved 
>> quoting.  It's the only way to avoid ludicrous 
>> and impossible to read quoting (which happens 
>> after quoted passages get line-wrapped at odd 
>> points).  Also, as far as I know, Exchange 
>> does not support format=flowed at all.  My 
>> understanding is that it insists on HTML 
>> quoting, which is entirely different.
>
>  You're right that this is the virtue of F/F, 
> but that's not what Michael said.  Perhaps it 
> was the context of the quoting, but the 
> discussion refers to the fact that both LookOut 
> and Apple Mail have (as you can see, by looking 
> at this message) a nasty habit of *assuming* an 
> F/F semantic by default.  In other words, they 
> generate hugely long long long long long long 
> long long long lines that only another equally 
> broken client can interpret correctly, but 
> dynamically reflowing those lines, by applying 
> consistent quoting indicators to the lines of 
> such reflowed lines, and so forth.  That, I 
> believe, is a legitimate complaint, because it e

This reflects a misunderstanding of 
Format=Flowed.  Properly generated F=F has lines 
of no more than 78 characters.  One of the 
primary goals of F=F is good interoperability 
between clients that support F=F and those that 
support traditional pure plain text email.  What 
you're describing is a symptom of HTML quoting 
(or a surprisingly poor F=F implementation):

>  At least Apple has the decency to use 
> quoted-printable encoding to protect the actual 
> transmitted lines,

Again, this shows a fundamental misunderstanding 
of F=F.  Properly generated F=F does not use 
quoted-printable unless otherwise needed:

    [Quoted-Printable] encoding SHOULD NOT be used with Format=Flowed
    unless absolutely necessary (for example, non-US-ASCII (8-bit)
    characters over a strictly 7-bit transport such as unextended
    [SMTP]).  In particular, a message SHOULD NOT be encoded in Quoted-
    Printable for the sole purpose of protecting the trailing space on
    flowed lines unless the body part is cryptographically signed or
    encrypted (see Section 4.6).

>  but it means nothing if competent MIME readers 
> reconstitute the corruption at the other end, 
> by reassembling long lines, and then failing to 
> reflow them.  And it's worse yet if a news 
> reader without MIME tries to quote the lines, 
> complete with Q-P line terminators.  Or if a 
> reply comes from somebody whose editor wraps 
> the lines, but using soft line breaks which are 
> never generated externally, resulting in whole 
> paragraphs being quoted using a single quoting 
> indicator.  And so on, and so on, and so on Š

All of these are part of the problem set that F=F 
was created to fix.  Proper F=F plays nice with 
pure plain text.

>  I filed bug 7989556 (Mail uses quoted-printable 
> indiscriminately) with Apple on 16 May 2010. 
> It was marked as duplicate of 7547565 on 25 May 
> 2010.  The system is closed; you can only see 
> your own bug reports.  I therefore have no idea 
> what's going on with that.  Meantime, I 
> apologise for the inconvenience.  No, really. 
> :-(

I haven't looked into Apple's mail's generation 
of either F=F or plain text or HTML.  If it 
generates F=F with long lines and/or 
indiscriminate q-p, then those would in my view 
be bugs that should be fixed.  But I don't know 
if the symptoms described occur with Apple Mail's 
F=F or some other format.  To me, they sound like 
lack of F=F.

-------------- Randomly selected tag: ---------------
Perhaps the real reason why we have always been able to champion free
speech in this country is that we know perfectly well that hardly
anybody has got anything to say, and that no one will listen to anyone
that has.       -- Editorial, Daily Mail [British Paper, date unknown]

---

From: Michael Richardson <[email-masked]>
Date: Wed, 24 Oct 2012 20:21:16 -0400

>> Warning: this message was generated by Apple Mail.

    RG> But not using Format=Flowed.

    RG> This reflects a misunderstanding of Format=Flowed.  Properly
    RG> generated F=F has lines of no more than 78 characters.  One of
    RG> the primary goals of F=F is good interoperability between
    RG> clients that support F=F and those that support traditional pure
    RG> plain text email.  What you're describing is a symptom of HTML
    RG> quoting (or a surprisingly poor F=F implementation):

so, really, what you are saying is that my complaint is not about a poor
F=F implementation, but in fact, not even being attempted.  It's just a
really poor text/plain part generated to go along with an (often absent)
text/html part, combined with a forced "upgrade" from text/plain to
text/html on the display side.

That's why they aren't marking, or paying attention to format=, because
they didn't even know about it.

-- 
Michael Richardson
-on the road-

---

From: Michael Richardson <[email-masked]>
Date: Wed, 24 Oct 2012 20:25:54 -0400

Sabahattin Gucukoglu <[email-masked]> wrote:
SG> Let's clear up the confusion.  I made two mistakes, firstly by
SG> calling this "F/F semantics" when what I mean is some sort of
SG> long-line-aware reflowing and quoting.  We'll have to find a name
SG> for it.  The other mistake was to call plain text plain text of any
SG> description, irrespective of the definition of text/plain.  

    SG> So we are talking about three formats:
    SG> * text/plain, 78 characters wide
    SG> * format=flowed, text/plain with soft breaks signalled by trailing whitespace, 78 characters
    SG> * text/paragraphs (or whatever), a completely different identity that violates the length limits

    SG> Apple Mail and Microsoft use this text/paragraphs.  It's not

Do you think it would be worth writing a specification for text/paragraphs?

Heuristically, it's not that hard to identify, and a small patch for
mailman would at least mark email as being in that format, so that at
least, IETF lists could have email that complies to some standard.

(Whether or not we then drop email that doesn't have a text/plain part
is a second conversation)

-- 
Michael Richardson
-on the road-

---

From: Sabahattin Gucukoglu <[email-masked]>
Date: Fri, 26 Oct 2012 00:43:27 +0100

On 25 Oct 2012, at 01:25, Michael Richardson <[email-masked]> wrote:
>    SG> Apple Mail and Microsoft use this text/paragraphs.
> 

Only if we can get a whole bunch of defective code reading and writing it.  There might be a case for that, given the laziness they've demonstrated in simply abusing text/plain, but it's also just possible that they might, just might implement format=flowed instead.  But it's unlikely. :-(

> least, IETF lists could have email that complies to some standard.

It would be easier and simpler and probably more participant-friendly (though not, see below, quite as good for changing the running code :-) ) to perform imperfect conversions from text/paragraphs to text/plain; format=flowed.  All we've got to do is identify any line longer than 80 characters and, assuming that it runs to the end of a paragraph at all times, add a forced soft line break encoded using F=F's stuffing rules.  Any line terminated before it's too long is assumed to be a manually inserted hard line break.

Now that I come to think of it, now is the time to see if the Tcl MIME parser will help me out writing a quick and dirty proxy server for my own machine …

> is a second conversation)

You want to punish Apple and/or Microsoft?  That's how to do it. :-)

Cheers,
Sabahattin