Subject: Re: content-md5
From: "Charles Lindsey" <[email-masked]>
To: [email-masked]
Date: Wed, 26 Nov 2003 17:17:06 GMT
Newsgroups: local.mime
List-Id: <ietf-822.imc.org>

In <[email-masked]> Dave Crocker <[email-masked]> writes:

>Folks,

>What is the current status of content-md5 usage?

>I am wondering whether this sort of signing of the body is reliable and
>cheap, and whether it has become at all popular.

Well dtmail is the only mailer that generates it AFAIK, and it gets it
wrong (or does in Solaris 7 - it may be fixed in later versions). I know
for a fact that Turnpike checks it.

But as a feature, I like it, because it is proof against changes of CTE as
the message propagates. Also, on occasions when headers of a message have
to be signed (e.g. in PGPVerify), it would be far nicer to exclude the
body of the message from the signature and, instead, sign the Content-MD5
header alongside whatever other headers were being signed. That way, if
something fails (e.g. some trailing white lines got added to the body),
the recipient has a much better chance of working out what went wrong, and
at least he knows that the headers were received intact.

So, it ain't broke and doesn't need to be fixed.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2795
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Date: Wed, 26 Nov 2003 17:10:23 GMT
Lines: 18
Sender: [email-masked]

In <[email-masked]> "Adam M. Costello" <[email-masked]> writes:

>But its classification as a signature separator for this purpose doesn't
>mean that "Joe Smith" will be displayed as a sig.  A message body has
>only one sig, and in this case it's "Jane Cooper".

OK, if that is always the effect of the definitions, then I am happy.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 25 Nov 2003 18:55:47 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Charles Lindsey <[email-masked]> wrote:

> C. A message either has a signature line, or it doesn't have a
> signature line.
>
> D. The answer to question C (does the user see a signature line on
> his display) MUST be the same whether viewed by an A-like client or a
> B-like client.

Perhaps the draft is not clear enough about the distinction between
"signature separator for the purpose of rewrapping" versus "signature
separator for the purpose of displaying a sig".  The draft is talking
about the former, and has nothing to say about the latter.  For example:

---begin-example---

>The quick brown fox 
>jumps over the lazy 
>dog.
>-- 
>Joe Smith

How now, brown cow?
-- 
Jane Cooper

---end-example---

The first "signature separator" is classified as a signature separator, rather
than a flowed line, in order to prevent it from being rewrapped like so:

>The quick brown fox jumps
>over the lazy dog.
>-- Joe Smith

But its classification as a signature separator for this purpose doesn't
mean that "Joe Smith" will be displayed as a sig.  A message body has
only one sig, and in this case it's "Jane Cooper".

AMC

Date: Tue, 25 Nov 2003 07:33:11 -0800
From: Dave Crocker <[email-masked]>
Reply-To: Dave Crocker <[email-masked]>
Organization: Brandenburg InternetWorking
To: ietf-822 <[email-masked]>
Subject: content-md5
Sender: [email-masked]

Folks,

What is the current status of content-md5 usage?

I am wondering whether this sort of signing of the body is reliable and
cheap, and whether it has become at all popular.

d/
----- 
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA USA <tel: +1.408.246.8253>; <fax: +1.408.850.1850>

To: [email-masked]
Xref: clerew local.mime:2792
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Date: Tue, 25 Nov 2003 10:25:30 GMT
Lines: 38
Sender: [email-masked]

In <[email-masked]> "Adam M. Costello" <[email-masked]> writes:

> 1. Count & strip quote indicators.
> 2. Unstuff: delete leading space if present.
> 3. Check for signature separator: dash-dash-space.
> 4. Check for trailing space to determine flowed/fixed (unless the
>    line has already been classified as a signature separator).

Hmmm! I have not followed all the discussion above, but there is one
property that should hold regardless:

A. Some clients will be flowed-aware, and will display the message
accordingly.

B. Some clients will not be flowed-aware, and will display the message as
received.

C. A message either has a signature line, or it doesn't have a signature
line.

D. The answer to question C (does the user see a signature line on his
display) MUST be the same whether viewed by an A-like client or a B-like
client.

Does what you propose have that property? The bit I have quoted above
suggests to me that it does not.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Tue, 25 Nov 2003 03:46:08 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Randall Gellens <[email-masked]> wrote:

> The grammar in the -03 and -04 versions is based on the suggested
> replacement you provided, plus fixes...to distinguish between quoted
> and unquoted.  I agree that this latter change does make the grammar
> larger and more complex.

Perhaps there is a middle ground.  Here is an excerpt from the -04
grammar:

    flowed-line      = ( flowed-line-qt / flowed-line-unqt ) flow CRLF
    flowed-line-qt   = quote ( ( stuffing stuffed-flowed ) /
                               unstuffed-flowed )
    flowed-line-unqt = ( stuffing stuffed-flowed ) / unstuffed-flowed
    fixed-line       = fixed-line-qt / fixed-line-unqt
    fixed-line-qt    = quote ( ( stuffing stuffed-fixed ) /
                               unstuffed-fixed ) CRLF
    fixed-line-unqt  = ( stuffed-fixed / unstuffed-fixed ) CRLF
    quote            = 1*">"

Here is what I had proposed:

    fixed-line       = quote ( (stuff stuffed-fixed) /
                               unstuffed-fixed         ) CRLF
    flowed-line      = quote ( (stuff stuffed-flowed) /
                               unstuffed-flowed         ) flow CRLF
    quote            = *">"

Here is a tweak to that proposal:

    fixed-line       = [quote] ( (stuff stuffed-fixed) /
                                 unstuffed-fixed         ) CRLF
    flowed-line      = [quote] ( (stuff stuffed-flowed) /
                                 unstuffed-flowed         ) flow CRLF
    quote            = 1*">"

My original proposed grammar presented the view that the quoting is
always present, but might have a depth of zero.  The tweaked grammar
presents a view more in line with the text of the draft: the quoting can
be present or not present, and if it is present it has a non-zero depth.
Unlike the -04 draft, the tweaked proposal doesn't give explicit names
to the quoted/unquoted versions of the lines, it just shows "[quote]"
in brackets, implying that the line is either quoted or not.  Forgoing
the explicit names shaves four rules off the grammar.  Just something to
consider.

> I believe this has been cleaned up in -04 so that it is now clear from
> all references in the text as well as the grammar that signature lines
> can be quoted or quoted and stuffed but they can't be stuffed without
> being quoted.

Section 5.1 is the first introduction to the syntax, and it says:

    Logically, this test for quoted lines is done before any other tests
    (that is, before checking for space-stuffed and flowed).

    Logically, this leading space is deleted before examining the line
    further (that is, before checking for flowed).

    If the line ends in a space, the line is flowed.  Otherwise it is
    fixed.  The exception to this rule is a signature separator line,
    described in Section 5.3.  Such lines end in a space but are neither
    flowed nor fixed.

It seems clear from reading 5.1 that the test for signature separator
lines happens along with the test for flowed lines, after the quoting
and space-stuffing have been stripped off.  But that inference is
inconsistent with the grammar.  I was hoping that section 5.1 would
give the complete procedure for analyzing the structure of a flowed
body.  Later sections tell what to do with that structure & why & how,
but I was hoping that 5.1 would contain all the same information as
the grammar (in plain English procedural form rather than a formal
declarative form).

Section 5.3 in -04 says:

    A receiving agent needs to test for a signature line both before the
    test for a quoted line (see Section 5.5) and also after logically
    counting and deleting quote marks and stuffing (see Section 5.4)
    from a quoted line.

If that's true, then I'd like to see those two tests in section 5.1
along with the other tests.  But I'm skeptical of testing for a
signature line after deleting the stuffing.  I don't see how that's
useful, because after the stuffing is deleted, there is no memory of it
(unlike the quote indicators, which are remembered in the quote depth).
If the line is dash-dash-space at this point, it might be a signature
line, but it might not be (if it had been stuffed and not quoted).

To decode the line syntax indicated in the -04 grammar, I think the
actual decoding steps are:

 1. Count & strip quote indicators.
 2. Check for signature separator:
    dash-dash-space is a separator (always),
    space-dash-dash-space is a separator if quote depth is nonzero,
 3. Unstuff: delete leading space if present.
 4. Check for trailing space to determine flowed/fixed (unless the
    line has already been classified as a signature separator).

Another way to achieve the same effect is:

 1. Count & strip quote indicators.
 2. Unstuff: delete leading space if present, but remember that the line
    was stuffed.
 3. If the line was quoted or neither-quoted-nor-stuffed, check for
    signature separator: dash-dash-space.
 4. Check for trailing space to determine flowed/fixed (unless the
    line has already been classified as a signature separator).

I still think it's very counterintuitive that I can use space-stuffing
to hide a signature line in unquoted text, but I can't do the same in
quoted text.  I think the grammar and parsing would be simpler and more
intuitive if the sequence were either:

 1. Count & strip quote indicators.
 2. Unstuff: delete leading space if present.
 3. Check for signature separator: dash-dash-space.
 4. Check for trailing space to determine flowed/fixed (unless the
    line has already been classified as a signature separator).

or:

 1. Count & strip quote indicators.
 2. Check for signature separator: dash-dash-space.
 3. Unstuff: delete leading space if present.
 4. Check for trailing space to determine flowed/fixed (unless the
    line has already been classified as a signature separator).

In other words, stuffing either hides signature separators or it
doesn't, regardless of quoting.

In any case, I'd like to see section 5.1 include all the steps for
decoding a line, whatever they are.

> > can you produce a concrete example of a line that ought to be flowed
> > and doesn't match my suggested flowed-line production?
> >
> > flowed-line       = quote (stuff stuffed-flowed / unstuffed-flowed) flow 
> > CRLF
> > stuffed-flowed    = [non-dash *text-char] /
> >                     "-" [non-dash *text-char / "-" 1*text-char]
> >                     ; Is not "--".
> > unstuffed-flowed  = non-sp-quote-dash *text-char /
> >                     "-" [non-dash *text-char / "-" 1*text-char]
> >                     ; Not empty, not "--", does not begin with SP or ">".
> > quote             = *">"
> > stuff             = SP
> > flow              = SP
> 
> I think it's a good idea to use parentheses to explicitly group the 
> ABNF constructs, to avoid confusion.
> 
> So, taking your suggested 'unstuffed-flowed' to be
> 
>     unstuffed-flowed  = ( non-sp-quote-dash *text-char ) /
>                         ( "-" [non-dash *text-char ) /
>                         ( "-" 1*text-char] )

Notice the position of your parentheses relative to my square brackets.
Perhaps I should have put spaces around the brackets:

stuffed-flowed    = [ non-dash *text-char ] /
                    "-" [ (non-dash *text-char) / ("-" 1*text-char) ]
                    ; Is not "--".
unstuffed-flowed  = (non-sp-quote-dash *text-char) /
                    "-" [ (non-dash *text-char) / ("-" 1*text-char) ]
                    ; Not empty, not "--", does not begin with SP or ">".

Maybe it would be clearer in a more verbose form without the brackets
and without any nesting:

stuffed-flowed    = "" / "-" /
                    (non-dash *text-char) /
                    ("-" non-dash *text-char) /
                    ("--" 1*text-char)
                    ; Is not "--".
unstuffed-flowed  = "-" /
                    (non-sp-quote-dash *text-char) /
                    ("-" non-dash *text-char) /
                    ("--" 1*text-char)
                    ; Not empty, not "--", does not begin with SP or ">".

Remember that these rules were written under the assumption that
signature separators can be quoted and/or stuffed in any combination.
These rules would need to be adjusted to reflect draft -04 sig-sep
syntax, or to reflect a syntax that never allows stuffed sig-sep lines.
But I have no doubt that we can write an unambiguous grammar for any of
these syntaxes.

AMC

Date: Wed, 19 Nov 2003 15:07:05 -0800
To: IETF RFC-822 list <[email-masked]>
From: Randall Gellens <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Cc: "Adam M. Costello" <[email-masked]>
Sender: [email-masked]

At 9:43 AM +0000 11/19/03, Adam M. Costello wrote:

>   > Note that the idea was to allow signature separator lines to be
>   > quoted, but not stuffed unless also quoted.
>
>  I failed to notice that subtlety.  Hmmm, are you sure that's what you
>  want?  That would be the only place where quoting and stuffing are not
>  orthogonal.

True, but then signature separator lines are a weird anomaly anyway. 
The RFC 2646 ABNF rules are clear that signature separator lines 
can't be stuffed unless they are also quoted, so I just carried that 
forward in the update.

>  It looks like it might be asking for trouble.  Consider
>  this format=flowed message body:
>
>  Foo
>   --
>  bar.
>
>  The first two lines end with a space, and the third does not.  According
>  to your definition of separator lines, this is a single paragraph,
>  because the middle line is a flowed line, not a separator line.  But
>  what happens if I try to quote that paragraph by inserting a quote
>  indicator before each line:
>
>>Foo
>>  --
>>bar.
>
>  According to your definition of separator lines, this is now an invalid
>  format=flowed message body, because it contains a flowed line followed
>  by a separator line.

True, but it would be rather foolish for a client to generate such a 
thing.  Why would it choose to wrap immediately before and after a 
SPACE DASH DASH SPACE sequence?

>   > The suggested definitions for flowed lines attempt to eliminate the
>>  ambiguity between a flowed line and a signature separator but I don't
>>  think they allow for all cases of flowed lines.
>
>  I wouldn't be surprised if I made a mistake in that tricky part of the
>  grammar, but can you produce a concrete example of a line that ought to
>  be flowed and doesn't match my suggested flowed-line production?  That
>  would be the surest way to expose the problem.
>
>  It's also possible that the grammar is correct, but structured in a
>  confusing way.  If that's the case, I welcome suggestions for less
>  confusing ways of expressing the same syntax.
>
>  For convenience, here's another copy of the relevant parts of the
>  grammar that I suggested:
>
>  flowed-line       = quote (stuff stuffed-flowed / unstuffed-flowed) flow CRLF
>  stuffed-flowed    = [non-dash *text-char] /
>                      "-" [non-dash *text-char / "-" 1*text-char]
>                      ; Is not "--".
>  unstuffed-flowed  = non-sp-quote-dash *text-char /
>                      "-" [non-dash *text-char / "-" 1*text-char]
>                      ; Not empty, not "--", does not begin with SP or ">".
>  quote             = *">"
>  stuff             = SP
>  flow              = SP

I think it's a good idea to use parentheses to explicitly group the 
ABNF constructs, to avoid confusion.  RFC 2234 recommends this as 
well.

So, taking your suggested 'unstuffed-flowed' to be

     unstuffed-flowed  = ( non-sp-quote-dash *text-char ) /
                     ( "-" [non-dash *text-char ) /
                     ( "-" 1*text-char] )

I think DASH DASH would match the third alternative for this 
'unstuffed-flowed', since the first DASH matches the "-" and the 
second DASH matches text-char, and only one text-char is required. 
So a line of DASH DASH SPACE would match 'flowed-line'.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The well-bred contradict other people.  The wise contradict themselves.
                                                         --Oscar Wilde

To: [email-masked]
Xref: clerew local.mime:2789
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Date: Wed, 19 Nov 2003 10:50:16 GMT
Lines: 48
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>"Charles Lindsey" <[email-masked]> writes:

>> But I still think that MUST is too strong. It is going to happen whether
>> you like it or not, so you have to warn of the dangers and give the aware
>> clients a mechanism to get around it.

>What mechanism?  All mechanisms that has been proposed so far are
>flawed in one way or another, as far as I can tell.

The essential mechanism you need is an ability for the user to ask to see
the text as sent (maybe after CTE decoding, but before flowing). If he
can at least see that, then he can arrange to check inline PGP signatures.
Without that, he is stuck.

>  Suggesting one of
>them doesn't do much good.  Although I lean towards MUST NOT, a SHOULD
>NOT would work.

I could live with SHOULD.

>  The main point is that the document don't say
>implementations SHOULD use a known broken mechanism.

Indeed.

>No, Opera's format=flowed handling is broken.  Look at the raw
>messages and you will probably find that each format=flowed
>"paragraph" is just one long (>76 octets) line with a terminating SPC.
>Perhaps it has been fixed in later releases.

The latest Opera versions (anything later than version 7) use their m2
mail client, which is a complete rewrite. It seems to make a reasonable
attempt at format=flowed, but I have not used it sufficiently to have
spotted all the niggles. As I pointed out yesterday, its compose window
seems not to do the right things always, but its display windows seem OK.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Wed, 19 Nov 2003 09:43:15 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Randall Gellens <[email-masked]> wrote:

> Note that the idea was to allow signature separator lines to be
> quoted, but not stuffed unless also quoted.  That way, stuffing can
> be used to guard against a line being confused with a signature
> separator.

I failed to notice that subtlety.  Hmmm, are you sure that's what you
want?  That would be the only place where quoting and stuffing are not
orthogonal.  It looks like it might be asking for trouble.  Consider
this format=flowed message body:

Foo 
 -- 
bar.

The first two lines end with a space, and the third does not.  According
to your definition of separator lines, this is a single paragraph,
because the middle line is a flowed line, not a separator line.  But
what happens if I try to quote that paragraph by inserting a quote
indicator before each line:

>Foo 
> -- 
>bar.

According to your definition of separator lines, this is now an invalid
format=flowed message body, because it contains a flowed line followed
by a separator line.

There would be no such gotcha if quoting and stuffing were kept
independent.  The rule could be that separator lines are still separator
lines after being stuffed, or that separator lines cease to be separator
lines when they are stuffed, as long as it is the same for both quoted
and unquoted lines.

> The suggested definitions for flowed lines attempt to eliminate the
> ambiguity between a flowed line and a signature separator but I don't
> think they allow for all cases of flowed lines.

I wouldn't be surprised if I made a mistake in that tricky part of the
grammar, but can you produce a concrete example of a line that ought to
be flowed and doesn't match my suggested flowed-line production?  That
would be the surest way to expose the problem.

It's also possible that the grammar is correct, but structured in a
confusing way.  If that's the case, I welcome suggestions for less
confusing ways of expressing the same syntax.

For convenience, here's another copy of the relevant parts of the
grammar that I suggested:

flowed-line       = quote (stuff stuffed-flowed / unstuffed-flowed) flow CRLF
stuffed-flowed    = [non-dash *text-char] /
                    "-" [non-dash *text-char / "-" 1*text-char]
                    ; Is not "--".
unstuffed-flowed  = non-sp-quote-dash *text-char /
                    "-" [non-dash *text-char / "-" 1*text-char]
                    ; Not empty, not "--", does not begin with SP or ">".
quote             = *">"
stuff             = SP
flow              = SP

I haven't yet looked at the -04 draft.  I'll hold off making any more
comments until I have.

AMC

To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
From: Simon Josefsson <[email-masked]>
Date: Wed, 19 Nov 2003 04:34:09 +0100
Sender: [email-masked]

"Charles Lindsey" <[email-masked]> writes:

>> All this means is that we have to say format=flowed MUST 
>>NOT be used with inline-signed messages, otherwise there is no guarantee 
>>that an arbitrary recipient will be able to verify it.
>
> But I still think that MUST is too strong. It is going to happen whether
> you like it or not, so you have to warn of the dangers and give the aware
> clients a mechanism to get around it.

What mechanism?  All mechanisms that has been proposed so far are
flawed in one way or another, as far as I can tell.  Suggesting one of
them doesn't do much good.  Although I lean towards MUST NOT, a SHOULD
NOT would work.  The main point is that the document don't say
implementations SHOULD use a known broken mechanism.

> I just tried it using Opera (which is the only client I have which offers
> format-flowed). When doing Copy/Paste from the compose window, it showed up
> as one long line, but after it had been posted and read (it was a local news
> article rather than an email), it behaved like a new format=flowed object.
> I.e. at the point where it wrapped in the viewing window, the pasted
> version showed a line end with a trailing SP (but it was longer than 76
> characters, though). Is that the behaviour we typically expect?

No, Opera's format=flowed handling is broken.  Look at the raw
messages and you will probably find that each format=flowed
"paragraph" is just one long (>76 octets) line with a terminating SPC.
Perhaps it has been fixed in later releases.

Date: Tue, 18 Nov 2003 15:34:07 -0800
To: IETF RFC-822 list <[email-masked]>, "Adam M. Costello" <[email-masked]>, Keith Moore <[email-masked]>, [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Format=Flowed/RFC 2646 Bis (-04)
Cc: [email-masked]
Sender: [email-masked]

I've created an -04 based on the additional comments received in the 
past few days.

Changes from -03 to -04:

* Clarified and made consistent treatment of signature separator lines.
* Fixes to ABNF.
* Reworded section on digital signatures and encryption.
* Noted meaning of "paragraph".

The updated text is available at 
<ftp://ftp.pensive.org/Public/Randy/draft-gellens-format-bis-04.txt>

Changes from -02 to -03:

* Added mention of quoting to Abstract and Introduction.
* Deleted line analysis table.
* Added MUST NOT for OpenPGP and SHOULD for OpenPGP-MIME.
* Replaced ABNF rules to remove ambiguity
* Added note that c-t-e is irrelevant to flowed text processing
* Added text indicating that end of data terminates a paragraph
* Moved sig-sep out of fixed-line ABNF
* Changed some SHOULDs to MUSTs (space-stuffing, quoted paragraphs)
* Added note to ABNF that space and ">" are encoded according to charset
* Mentioned exceptions in section on interpreting
* Moved section on interpreting before section on generating.
* Reworded non-normative "should"s.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Frequently, people can tell you they've read a book and
liked it, but they can't tell you why.  We don't want the
reader to have to do the hard work of figuring that out.
                    --Duncan Smith, creator of NoveList.

Date: Tue, 18 Nov 2003 15:34:22 -0800
To: IETF RFC-822 list <[email-masked]>
From: Randall Gellens <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Cc: "Adam M. Costello" <[email-masked]>
Sender: [email-masked]

At 5:47 AM +0000 11/17/03, Adam M. Costello wrote:

>  Two of the steps I listed need to be swapped.  Step 3 checks for the
>  sig line, and step 4 unstuffs.  I wrote them in that order because
>  section 5.3 says "an (optionally quoted) line consisting of DASH DASH
>  SP is not considered flowed."  But now I notice that the grammar
>  says sig-sep = [quote [stuffing]] "--" SP CRLF.  To be consistent,
>  section 5.3 should say "(optionally quoted, optionally stuffed)", and
>  the interpreting section should not check for a sig line until after
>  unstuffing.
>
>  (Or do you want to resolve the inconsistency the other way, by changing
>  the grammar?)

The -03 version fixed the grammar to treat signature separator lines 
as a third type of line.  I just created an -04 which changes "an 
(optionally quoted) line consisting of DASH DASH SP" to "an 
(optionally quoted or quoted and stuffed) line consisting of DASH 
DASH SP".

Note that the idea was to allow signature separator lines to be 
quoted, but not stuffed unless also quoted.  That way, stuffing can 
be used to guard against a line being confused with a signature 
separator.

>   > Adam also had a number of concerns over ambiguity in the grammar, with
>>  suggestions for improvement.  I generally like the replacement text,
>>  except for the removal of the distinction between quoted and unquoted
>>  lines.  I thought it was helpful to identify a non-quoted line on
>>  its own, and not just as a line with a quote-depth of zero.  This
>>  is based on a perceived need to treat the two somewhat differently,
>>  in particular, quoted lines need extra handling, state, and display
>>  semantics.
>
>  I would think it would be easier for an implementor to write a single
>  handler & data structure for all paragraphs of all quote depths, rather
>  than make quote-depth-zero a special case.

Making the distinction in the grammar still allows a client to 
implement either way.

>   Of course it's possible to
>  expand the grammar to make the distinction, but I think it makes the
>  grammar appear more complex than it really is.

This is a good point.  The grammar in the -03 and -04 versions is 
based on the suggested replacement you provided, plus fixes to treat 
signature lines as a third type, and to distinguish between quoted 
and unquoted.  I agree that this latter change does make the grammar 
larger and more complex.

>   (And I just noticed that
>  while the grammar in the current draft distinguishes between quoted
>  flowed and unquoted flowed, it does not distinguish between quoted fixed
>  and unquoted fixed.)

This was fixed in -03.

>  By the way, my proposed grammar forgot to handle the Usenet sig
>  exception.

I fixed this in -03.

>    Here's a fixed version that also incorporates my three
>  suggestions regarding improperly terminated paragraphs:
>
>  flowed-body       = * ( paragraph / fixed-line / sig-line )
>  paragraph         = 1*flowed-line fixed-line
>                      ; That is the grammar for proper paragraphs, which
>                      ; always end with a fixed line.  Improper paragraphs
>                      ; are instead terminated by a change in quote-depth,
>                      ; end of input, or a sig-line (which is not included
>                      ; in the paragraph).
>  sig-line          = quote [stuff] "--" SP CRLF
>  fixed-line        = quote (stuff stuffed-fixed / unstuffed-fixed) CRLF
>  flowed-line       = quote (stuff stuffed-flowed / unstuffed-flowed) flow CRLF
>  stuffed-fixed     = [*text-char non-sp]
>                      ; Does not end with SP.
>  unstuffed-fixed   = non-sp-quote [*text-char non-sp]
>                      ; Does not begin with SP or ">", does not end with SP.
>  stuffed-flowed    = [non-dash *text-char] /
>                      "-" [non-dash *text-char / "-" 1*text-char]
>                      ; Is not "--".
>  unstuffed-flowed  = non-sp-quote-dash *text-char /
>                      "-" [non-dash *text-char / "-" 1*text-char]
>                      ; Not empty, not "--", does not begin with SP or ">".
>  quote             = *">"
>  stuff             = SP
>  flow              = SP
>  non-sp-quote-dash = <any character except NUL, CR, LF, SP, ">", "-">
>  non-sp-quote      = <any character except NUL, CR, LF, SP, ">">
>  non-sp            = <any character except NUL, CR, LF, SP>
>  text-char         = <any character except NUL, CR, LF>
>  non-dash          = <any character except NUL, CR, LF, "-">

That definition for sig-line allows it to be stuffed but not quoted, 
which we have been prohibiting.

The suggested definitions for flowed lines attempt to eliminate the 
ambiguity between a flowed line and a signature separator but I don't 
think they allow for all cases of flowed lines.  I think we'd need 
something like:
	non-sp-quote-dash *text-char /
	"-" non-dash *text-char /
	"--" non-space *text-char /
	"--" 2*text-char

But this wouldn't work since the definition of flowed-line includes 
the flow (space) at the end.  I don't see any way to resolve the 
ambiguity without really convoluting the ABNF.  I think it may be OK 
to just note the potential ambiguity, especially since a check for 
signature line can be made before checking for a flowed line.  I've 
added a note in the ABNF section of -04 about it.

>  One of my suggested grammar tweaks was that a sig line should not get
>  sucked into a paragraph, even if it is preceeded by a flowed line,
>  because then it could get re-wrapped and no longer appear at the start
>  of a line (and therefore cease to be a sig line).  I notice now that
>  this suggestion amounts to having a third type of line.  A sig line is
>  neither fixed nor flowed, because fixed and flowed lines can be inside
>  paragraphs, while sig lines can never be inside paragraphs.

I agree, and I believe this was fixed in -03.

>  Another comment regarding the grammar:  It is nice for a grammar to give
>  names to the meaningful syntactic constructs.  For example, we'd like
>  a name for the quote-marks (and we have one), we'd like names for the
>  special spaces that act as flags (and we have them), and we'd like a
>  name for the actual content of the line without the quotes and flags,
>  but the grammar in the draft doesn't give us that.  Consider for example
>  stuffed-flowed.  In the draft, this means a line that *was* flowed and
>  *is* stuffed (it includes the stuff space but not the flow space).  In
>  the grammar in the old message above, stuffed-flowed means a line that
>  *was* flowed and *was* stuffed (it includes neither the stuff space nor
>  the flow space, only the actual content).

This is a nice feature, and I believe I've achieved it in -04 (at the 
expense of some extra parentheses).

>  Section 5.1 says:
>
>      If the line ends in a space, the line is flowed.  Otherwise it is
>      fixed.  The exception to this rule is a signature separator line,
>      described in Section 5.3.  Such lines end in a space but are not
>      flowed.
>
>  That leaves the following question unanswered:  Are separator lines
>  fixed, or are they a third type of line?
>
>  According to the grammar in section 7, signature separator lines do not
>  match fixed-line.  That seems to suggest that they are a third type of
>  line, which is the view that seems most intuitive to me.  That could be
>  clarified by changing the last sentence of the quoted paragraph to "Such
>  lines end in a space but are neither flowed nor fixed."

I agree and made this change in -04.

>  Section 5.3 says:
>
>      This is a special case; an (optionally quoted) line consisting of
>      DASH DASH SP is not considered flowed.
>
>  Sections 5.1 (interpreting) and 7 (grammar) both indicate that a
>  signature line can be quoted and/or stuffed.  It is confusing for 5.3
>  to mention "optionally quoted" without also mentioning "optionally
>  stuffed".  Also, if in section 5.1 "not flowed" is changed to "neither
>  flowed nor fixed", the same change ought to be made here.

Thanks; I believe this has been cleaned up in -04 so that it is now 
clear from all references in the text as well as the grammar that 
signature lines can be quoted or quoted and stuffed but they can't be 
stuffed without being quoted.

>  Section 5.3 goes on to say:
>
>      Generating agents MUST NOT end a paragraph with such a signature
>      line, since doing so would indicate that the separator line is part
>      of the paragraph.
>
>  It would not indicate that the separator line is part of the paragraph,
>  it would indicate that the body is malformed (according to the grammar
>  and according to section 5.1); the receiver would not believe that the
>  separator is part of the paragraph (according to 5.1).

That jumped out at me as I was making another change.  I deleted the 
second clause, so now it just says "Generating agents MUST NOT end a 
paragraph with such a signature line".

>  Perhaps the intention is something like this:
>
>      When placing soft line breaks in a paragraph, generating agents MUST
>      NOT place them in a way that causes any line of the paragraph to
>      be a signature separator line, because paragraphs cannot contain
>      signature separator lines (see sections 5.1 and 7).

I'm not sure if that was the original intent or not, but I liked the 
text you suggest and so added it to the section on generating f=f 
(with references to the section on signature lines and on the abnf).

>  Section 5.4 says:
>
>      Space-stuffing adds a single space to the start of any line which
>      needs protection when the message is generated.  On reception, if
>      the first character of a line is a space, it is logically deleted.
>      This occurs after the test for a quoted line, and before the test
>      for a flowed line.
>
>  It's not only after testing for a quoted line, but more importantly
>  after stripping the quoting.

But the test for quoted line is what deletes the quote marks.  I 
changed the text in -04 to say "This occurs
after the test for a quoted line (which logically counts and deletes 
any quote marks)".

>    And it's not only before the test for
>  a flowed line, but also before the test for a separator line.

I think the test for a signature line has to happen both before the 
test for a quoted line and also after deleting quote marks and 
stuffing.

>  Section 5.5 says:
>
>      When generating quoted flowed lines, an agent needs to pay attention
>      to changes in quote depth.  A sequence of quoted lines of the same
>      quote depth immediately followed by lines of a different quote
>      depth MUST be encoded so that lines of the same quote depth are a
>      paragraph, with the last line generated as fixed and prior lines
>      generated as flowed.
>
>  That seems to be a much stronger requirement than you intend.  Within a
>  single quote depth, there might be multiple paragraphs, non-paragraph
>  fixed-lines, and separator lines.  But the sentence quoted above seems
>  to say that because all of this text is a bunch of "lines at the same
>  quote depth", it must be encoded as "a paragraph", with the last line
>  fixed and all other lines flowed.  Perhaps the intention is something
>  like this:
>
>      When generating quoted flowed lines, an agent needs to pay attention
>      to changes in quote depth.  All lines of a paragraph MUST be
>      unquoted, or else they MUST all be quoted and have the same quote
>      depth.  Therefore, whenever there is a change in quote depth, or a
>      change from quoted to unquoted, or change from unquoted to quoted,
>      the line immediately preceeding the change MUST NOT be a flowed
>      line.

Indeed.  Thanks for catching this.

>  Section 5.5 goes on to say:
>
>      If a receiving agent wishes to reformat flowed quoted lines (joining
>      and/or wrapping them) on display or when generating new messages,
>      the lines SHOULD be de-quoted, reformatted, and then re-quoted.  To
>      de-quote, the number of close angle brackets in the quote indicator
>      at the start of each line is counted.  Consecutive lines with the
>      same quote depth are considered one paragraph and are reformatted
>      together.  To re-quote after reformatting, a quote indicator
>      containing the same number of close angle brackets originally
>      present are prefixed to each line.
>
>  I think one sentence there is inaccurate: "Consecutive lines with the
>  same quote depth are considered one paragraph and are reformatted
>  together."  Consecutive lines with the same quote depth could be one
>  paragraph or several paragraphs or non-paragraph fixed lines (in which
>  case no reformatting is requested) or separator lines.  I think that
>  sentence can simply be removed.  Reformatting is covered elsewhere; this
>  section is about quoting.

Another good catch.  The sentence is deleted in -04.

>  The next two paragraphs are inconsistent with section 5.1:
>
>      On reception, if a change in quote depth occurs on a flowed line,
>      this is an improperly formatted message.  The receiver SHOULD handle
>      this error by using the 'quote-depth-wins' rule, which is to ignore
>      the flowed indicator and treat the line as fixed.  That is, the
>      change in quote depth ends the paragraph.
>
>      In other words, whenever two adjacent lines have different quote
>      depths, senders MUST ensure that the earlier line is fixed (does
>      not end in a space), and receivers SHOULD treat the earlier line as
>      fixed regardless of whether it ends with a space.
>
>  According to section 5.1, the paragraph ends with the flowed line; it is
>  possible therefore to have an improperly terminated paragraph consisting
>  of a single flowed line, and such a paragraph would be reformatted.  If
>  the flowed indicator is ignored and the line is treated as fixed, then
>  we have a single fixed line, which is not a paragraph at all and would
>  not be reformatted.  Also, it is possible for the line before the change
>  in quote depth to be a separator line, which is arguably not fixed (see
>  the discussion above).  The inconsistency could be resolved like so:
>
>      ...the 'quote-depth-wins rule', which is to consider the paragraph
>      to end with the flowed line immediately preceeding the change in
>      quote depth.
>
>      In other words, whenever two adjacent lines have different quote
>      depths, senders MUST ensure that the earlier line is not flowed
>      (does not end in a space), and receivers finding a flowed line there
>      SHOULD treat it as the last line of a paragraph.

I agree; thanks again.

>  Here we have more instances of the phrase "change in quote depth".  If
>  we keep the current view that unquoted lines have no quote depth and
>  quoted lines have non-zero quote depth, then we really ought to be
>  saying "change in quote depth, or change from quoted to unquoted, or
>  change from unquoted to quoted".  If we adopt the view that all lines
>  have a quote depth, which can be zero, then the simple phrase "change in
>  quote depth" will mean what we want it to mean.

I think "change in quote depth" can include a changing between quoted 
and unquoted.  Even though I have retained the ABNF distinction 
between quoted and unquoted, I don't think the text has to be too 
rigid about it.

-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Nothing astonishes men so much as common sense and plain dealing.
                                           --Ralph Waldo Emerson.

To: [email-masked]
Xref: clerew local.mime:2784
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Date: Tue, 18 Nov 2003 15:56:05 GMT
Lines: 74
Sender: [email-masked]

In <2147483647.1069075144@[10.0.1.8]> Cyrus Daboo <[email-masked]> writes:

>OK - so that just goes to prove that there is no way to construct an 
>inline-signed format=flowed messages such that it will be verifiable by 
>both format=flowed aware and non-aware clients using the clipboard method 
>unless the aware client copies the non-flowed text to the clipboard - which 
>I think is wrong.

I think we are agreed that inline PGP signing will be understood by the
non-aware clients. What is now clear is that an aware client MUST have
some mechanism to make the un-re-flowed (i.e. on the wire) text available
to the user if he asks to see it. That might be dome by making the
Clipboard take the un-re-flowed version, or it might be done some other
way.

> All this means is that we have to say format=flowed MUST 
>NOT be used with inline-signed messages, otherwise there is no guarantee 
>that an arbitrary recipient will be able to verify it.

But I still think that MUST is too strong. It is going to happen whether
you like it or not, so you have to warn of the dangers and give the aware
clients a mechanism to get around it.

>| Which raises another issue. Mail clients which support format=flowed
>| SHOULD put the original (unflowed) version of the text onto the Clipboard.
>| IOW, you should be able to take the displayed text, put it into the
>| clipboard (or dran'n'drop it), and plonk it into another (format=flowed
>| aware) window, and it should automagically appear reflowed to suit the new
>| window.

>I think this is wrong. First off where are you going to find a 
>format=flowed aware 'window' that is not in an email client? I want to be 
>able to copy the flowed text from a message and paste it into a text editor 
>or some other application and have it appear flowed.

If you plonk it into a window in an email client (say as part of
generating a new outgoing message) then it can be shown reflowed (but when
sent on the wire it will be as its 'true' self). I think if you plonk it
into a window that is not aware (e.g. into some editor) then it will show
up un-re-flowed (which is still a valid representation of the text). So
with my suggestion, ordinary unaware windows need do nothing with the
Clipboard than they did not do before.

But if copying the text of a flowed message to the Clipboard is going to
take it as it had been reflowed in the donor window, then you have to ask
exactly how it is supposed to appear. Is it just a text with lines of a
length as determined by the donor window and no trailing WS, or is it
still in a format=flowed form with trailing WS to show where it might be
reflowed yet again? Or is it just one long line of text for each
paragraph, in which case what about any quote marks in it?

I think these questions need to be addressed, though whether the standard
goes to the length of codifying the behaviours is another matter.

I just tried it using Opera (which is the only client I have which offers
format-flowed). When doing Copy/Paste from the compose window, it showed up
as one long line, but after it had been posted and read (it was a local news
article rather than an email), it behaved like a new format=flowed object.
I.e. at the point where it wrapped in the viewing window, the pasted
version showed a line end with a trailing SP (but it was longer than 76
characters, though). Is that the behaviour we typically expect?

BTW, it didn't do the right thing when pasted into a new compose window :-(.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 17 Nov 2003 13:25:29 -0500
From: Keith Moore <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Sender: [email-masked]

> > I also don't see the problem with calling a group of lines intended
> > to be re-flowed a "paragraph".
> 
> "Paragraph" seems like a nice intuitive term to me.  In TeX, for
> example, a "paragraph" is the one and only construct that flows (has
> line breaks inserted automatically).  Keith hinted that "paragraph"
> too loaded a term; maybe he'd like to explain that in more detail.

The word "paragraph" describes the semantics of the block of text; it
implies, for instance, that the block of text contains one or more 
sentences (some say two or more) on a single topic that are spoken by 
a single speaker.  For instance, it would not be appropriate for a
text-to-whatever translator to map a f=f "paragraph" into a "paragraph"
in some semantic markup language.

But if "paragraph" is the closest term you can find, use it, and just
point out that it's not quite the same as a "real" paragraph.

Date: Mon, 17 Nov 2003 13:19:04 -0500
From: Cyrus Daboo <[email-masked]>
To: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Sender: [email-masked]

Hi Charles,

--On Monday, November 17, 2003 12:24 +0000 Charles Lindsey 
<[email-masked]> wrote:

|> A non-format=flowed aware client, when processing a received
|> format=flowed  message, will first do CTE decoding and then display the
|> message.  Verification of the signature will then be done on the
|> unflowed text (i.e.  with trailing spaces and CRLFs etc).
|
| No, that will not do, because mail clients, generally speaking, do not do
| PGP verification (whether inline or PGP/MIME). What usually happens is
| that the user takes the signed text displayed on his screen, and verifies
| it by a separate process. This usually starts by copying it all to the
| Clipboard, and then passing the Clipboard to some verifier.

OK - so that just goes to prove that there is no way to construct an 
inline-signed format=flowed messages such that it will be verifiable by 
both format=flowed aware and non-aware clients using the clipboard method 
unless the aware client copies the non-flowed text to the clipboard - which 
I think is wrong. All this means is that we have to say format=flowed MUST 
NOT be used with inline-signed messages, otherwise there is no guarantee 
that an arbitrary recipient will be able to verify it.

| Which raises another issue. Mail clients which support format=flowed
| SHOULD put the original (unflowed) version of the text onto the Clipboard.
| IOW, you should be able to take the displayed text, put it into the
| clipboard (or dran'n'drop it), and plonk it into another (format=flowed
| aware) window, and it should automagically appear reflowed to suit the new
| window.

I think this is wrong. First off where are you going to find a 
format=flowed aware 'window' that is not in an email client? I want to be 
able to copy the flowed text from a message and paste it into a text editor 
or some other application and have it appear flowed.

-- 
Cyrus Daboo

To: [email-masked]
Xref: clerew local.mime:2777
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Date: Mon, 17 Nov 2003 12:13:54 GMT
Lines: 79
Sender: [email-masked]

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>I agree with you that it isn't right, but it is what the draft says.
>That's my point.  The text was quoted in my replace, but here is the
>text again:

>5.6.  Digital Signatures and Encryption

>    If a message is digitally signed or encrypted it is important that
>    cryptographic processing use the on-the-wire Format=Flowed format.
>    That is, during generation the message SHOULD be prepared for
>    transmission, including addition of soft line breaks,
>    space-stuffing, and [Quoted-Printable] encoding (to protect soft
>    line breaks) before being digitally signed or encrypted; similarly,
>    on receipt the message SHOULD have the signature verified or be
>    decrypted before [Quoted-Printable] decoding and removal of stuffed
>    spaces, soft line breaks and quote marks, and reflowing.

>The text say that MUAs SHOLD do things differently from what you
>describe.

Yes. It is clear that the text in the present draft is plain wrong; but I
had been reading it as indicating the use of PGP/MIME, and if you add a
few words about PGP/MIME to it, then it makes sense again.

>FWIW, your approach has the security problem that started this thread;
>it makes it possible for someone to in transit add trailing SPC to the
>PGP message, and add a format=flowed tag to the headers, without being
>detected.

I don't think that is all that important. There is not much you can do by
way of malicious alteration to a message if all you can do is to change
the where the line breaks are perceived to be. Yes, it would be better to
ensure all the trailing spaces get included in the signature, and hence
PGP/MIME is strongly to be recommended. But people are undoubtedly going
to try to use inline PGP whatever we say, in which case a warning of its
limitations is all that is needed in the document.

>I haven't seen a way to support both inline PGP and format=flowed
>without creating one problem or another.

Well one possibility is to turn off textmode before signing (but people on
Unix systems will then have to edit in explicit CRs first). The bad news
is that all PGP systems turn off clearsigning when textmode is off.

Here is a signed text with some trailing whitespace in it (and it will be
proof against systems that munge WS too).

But it is not much use to those who are not pgp-aware :-( .

>> 2. Use RFC 3156 (PGP/MIME).

>Right.  This is what I believe should be used instead.

Indeed so.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2778
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Date: Mon, 17 Nov 2003 12:24:15 GMT
Lines: 36
Sender: [email-masked]

In <2147483647.1068836160@[10.0.1.5]> Cyrus Daboo <[email-masked]> writes:

>A non-format=flowed aware client, when processing a received format=flowed 
>message, will first do CTE decoding and then display the message. 
>Verification of the signature will then be done on the unflowed text (i.e. 
>with trailing spaces and CRLFs etc).

No, that will not do, because mail clients, generally speaking, do not do
PGP verification (whether inline or PGP/MIME). What usually happens is
that the user takes the signed text displayed on his screen, and verifies
it by a separate process. This usually starts by copying it all to the
Clipboard, and then passing the Clipboard to some verifier.

Which raises another issue. Mail clients which support format=flowed
SHOULD put the original (unflowed) version of the text onto the Clipboard.
IOW, you should be able to take the displayed text, put it into the
clipboard (or dran'n'drop it), and plonk it into another (format=flowed
aware) window, and it should automagically appear reflowed to suit the new
window.

>Thus a format=flowed aware client that generates a message would first wrap 
>the text doing flow, then do the PGP signature, and then do CTE encoding. 
>That is the only way to work with non-aware clients.

Yes, except that it should still work, modulo malicious changes to the
line endings, even if the CTE stage is omitted.

Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

To: [email-masked]
Xref: clerew local.mime:2779
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Date: Mon, 17 Nov 2003 12:27:04 GMT
Lines: 24
Sender: [email-masked]

In <p0610044dbbdb254d732e@[130.129.134.206]> Randall Gellens <[email-masked]> writes:

>I've created a -03 based on comments received in the past few days.

>Changes from -02:

>* Added MUST NOT for OpenPGP and SHOULD for OpenPGP-MIME.

I think that MUST NOT is too severe. People are going to do it whether you
like it or not. The best you shjould do is to warn of the dangers (which
are not actually all that bad - just that your trailing spaces my get
munged by a malicious man in the middle) and deprecate it in favour of
PGP/MIME.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

Date: Mon, 17 Nov 2003 06:31:31 -0800 (PST)
From: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
To: Randall Gellens <[email-masked]>
Cc: [email-masked], Keith Moore <[email-masked]>, Randall Gellens <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> At 3:28 PM -0800 11/16/03, [email-masked] wrote:

> >  As long as the document makes it clear
> >  we're dealing with the characters that result from the application of the
> >  charset to the sequence of octets we should be good to go.

> Here is the sentence I added in -03 (in the ABNF section):

>      Note that the SP (space) and ">" characters are encoded according
> to the charset parameter.

> Do you think this is sufficient?

Yes I do.

				Ned

Date: Mon, 17 Nov 2003 05:47:27 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Randall Gellens <[email-masked]> wrote:

> I've created a -03 based on comments received in the past few days.

Doh, I just noticed that a message that I sent to the list on Nov-09 had
the wrong From: address and therefore was not permitted to go to the
list (presumably it is still awaiting moderator approval).  I'll include
that message here.  Please remember that when it says "current draft",
it means draft -02.

---begin-old-message---

Date: Mon, 10 Nov 2003 07:46:05 +0000
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Message-ID: <[email-masked]>
References: <[email-masked]>
In-Reply-To: <[email-masked]>

[email-masked] wrote:

> it would have been even better had this activity occurred sometime
> earlier in the past few years, since most of the text in question
> hasn't changed in some time.

Indeed.  Somehow I had never heard of format=flowed until quite
recently.

> I also don't see the problem with calling a group of lines intended to
> be re-flowed a "paragraph".

"Paragraph" seems like a nice intuitive term to me.  In TeX, for
example, a "paragraph" is the one and only construct that flows (has
line breaks inserted automatically).  Keith hinted that "paragraph" too
loaded a term; maybe he'd like to explain that in more detail.

> Adam also suggests mentioning in the section on interpreting f=f the 
> exceptions for Usenet signatures and  changing-quote-depth, which 
> seem like good ideas to me.

Two of the steps I listed need to be swapped.  Step 3 checks for the
sig line, and step 4 unstuffs.  I wrote them in that order because
section 5.3 says "an (optionally quoted) line consisting of DASH DASH
SP is not considered flowed."  But now I notice that the grammar
says sig-sep = [quote [stuffing]] "--" SP CRLF.  To be consistent,
section 5.3 should say "(optionally quoted, optionally stuffed)", and
the interpreting section should not check for a sig line until after
unstuffing.

(Or do you want to resolve the inconsistency the other way, by changing
the grammar?)

> Adam also had a number of concerns over ambiguity in the grammar, with
> suggestions for improvement.  I generally like the replacement text,
> except for the removal of the distinction between quoted and unquoted
> lines.  I thought it was helpful to identify a non-quoted line on
> its own, and not just as a line with a quote-depth of zero.  This
> is based on a perceived need to treat the two somewhat differently,
> in particular, quoted lines need extra handling, state, and display
> semantics.

I would think it would be easier for an implementor to write a single
handler & data structure for all paragraphs of all quote depths, rather
than make quote-depth-zero a special case.  Of course it's possible to
expand the grammar to make the distinction, but I think it makes the
grammar appear more complex than it really is.  (And I just noticed that
while the grammar in the current draft distinguishes between quoted
flowed and unquoted flowed, it does not distinguish between quoted fixed
and unquoted fixed.)

By the way, my proposed grammar forgot to handle the Usenet sig
exception.  Here's a fixed version that also incorporates my three
suggestions regarding improperly terminated paragraphs:

flowed-body       = * ( paragraph / fixed-line / sig-line )
paragraph         = 1*flowed-line fixed-line
                    ; That is the grammar for proper paragraphs, which
                    ; always end with a fixed line.  Improper paragraphs
                    ; are instead terminated by a change in quote-depth,
                    ; end of input, or a sig-line (which is not included
                    ; in the paragraph).
sig-line          = quote [stuff] "--" SP CRLF
fixed-line        = quote (stuff stuffed-fixed / unstuffed-fixed) CRLF
flowed-line       = quote (stuff stuffed-flowed / unstuffed-flowed) flow CRLF
stuffed-fixed     = [*text-char non-sp]
                    ; Does not end with SP.
unstuffed-fixed   = non-sp-quote [*text-char non-sp]
                    ; Does not begin with SP or ">", does not end with SP.
stuffed-flowed    = [non-dash *text-char] /
                    "-" [non-dash *text-char / "-" 1*text-char]
                    ; Is not "--".
unstuffed-flowed  = non-sp-quote-dash *text-char /
                    "-" [non-dash *text-char / "-" 1*text-char]
                    ; Not empty, not "--", does not begin with SP or ">".
quote             = *">"
stuff             = SP
flow              = SP
non-sp-quote-dash = <any character except NUL, CR, LF, SP, ">", "-">
non-sp-quote      = <any character except NUL, CR, LF, SP, ">">
non-sp            = <any character except NUL, CR, LF, SP>
text-char         = <any character except NUL, CR, LF>
non-dash          = <any character except NUL, CR, LF, "-">

One of my suggested grammar tweaks was that a sig line should not get
sucked into a paragraph, even if it is preceeded by a flowed line,
because then it could get re-wrapped and no longer appear at the start
of a line (and therefore cease to be a sig line).  I notice now that
this suggestion amounts to having a third type of line.  A sig line is
neither fixed nor flowed, because fixed and flowed lines can be inside
paragraphs, while sig lines can never be inside paragraphs.

Someone should double-check that grammar, especially the rules for
[un]stuffed-{fixed,flowed}.

AMC

---end-old-message---

Sorry you didn't get a chance to see that before revising the draft.
Henceforth, when I say "current draft", I mean draft -03.

> * Added mention of quoting to Abstract and Introduction.
> * Deleted line analysis table.
> * Added note that c-t-e is irrelevant to flowed text processing
> * Added text indicating that end of data terminates a paragraph
> * Moved sig-sep out of fixed-line ABNF
> * Mentioned exceptions in section on interpreting
> * Moved section on interpreting before section on generating.
> * Reworded non-normative "should"s.

All good.

> * Changed some SHOULDs to MUSTs (space-stuffing, quoted paragraphs)

I haven't given the distinction much thought for this protocol.

> * Added MUST NOT for OpenPGP and SHOULD for OpenPGP-MIME.

I'll defer to Simon and Cyrus, who seem to have that issue covered.

> * Added note to ABNF that space and ">" are encoded according to charset

But what about the decoding side?  I think Ned has the right idea--the
clarification could say that the grammar is in terms of characters,
and therefore an encoder using the grammar to generate sequences
of characters would then need to transform the characters to bytes
according to the charset, and a decoder using the grammar to parse
sequences of characters would first need to transform the bytes to
characters according to the charset.

> * Replaced ABNF rules to remove ambiguity

One ambiguity still remains.  Consider these two lines:

-- 
foo

The first ends with a space, and the second does not.  We would like
this to parse as a sig-sep and a fixed-line, but according to the
current grammar it can also parse as a paragraph, because "-- " matches
the flowed-line production.

It is possible to eliminate that ambiguity; see the grammar in the old
message above.

Another comment regarding the grammar:  It is nice for a grammar to give
names to the meaningful syntactic constructs.  For example, we'd like
a name for the quote-marks (and we have one), we'd like names for the
special spaces that act as flags (and we have them), and we'd like a
name for the actual content of the line without the quotes and flags,
but the grammar in the draft doesn't give us that.  Consider for example
stuffed-flowed.  In the draft, this means a line that *was* flowed and
*is* stuffed (it includes the stuff space but not the flow space).  In
the grammar in the old message above, stuffed-flowed means a line that
*was* flowed and *was* stuffed (it includes neither the stuff space nor
the flow space, only the actual content).

Section 5.1 says:

    If the line ends in a space, the line is flowed.  Otherwise it is
    fixed.  The exception to this rule is a signature separator line,
    described in Section 5.3.  Such lines end in a space but are not
    flowed.

That leaves the following question unanswered:  Are separator lines
fixed, or are they a third type of line?

According to the grammar in section 7, signature separator lines do not
match fixed-line.  That seems to suggest that they are a third type of
line, which is the view that seems most intuitive to me.  That could be
clarified by changing the last sentence of the quoted paragraph to "Such
lines end in a space but are neither flowed nor fixed."

Another way to view the situation is that sig-lines are fixed, and
paragraphs end with non-sig-sep fixed lines.  The grammar would then be:

flowed-body = *( paragraph / fixed-line )
fixed-line  = sig-sep / non-sig-sep-fixed-line
paragraph   = 1*flowed-line non-sep-fixed-line

But that looks unnaturally convoluted to me.  I prefer the existing
grammar:

flowed-body = *( paragraph / fixed-line / sig-sep )
paragraph   = 1*flowed-line fixed-line

Section 5.3 says:

    This is a special case; an (optionally quoted) line consisting of
    DASH DASH SP is not considered flowed.

Sections 5.1 (interpreting) and 7 (grammar) both indicate that a
signature line can be quoted and/or stuffed.  It is confusing for 5.3
to mention "optionally quoted" without also mentioning "optionally
stuffed".  Also, if in section 5.1 "not flowed" is changed to "neither
flowed nor fixed", the same change ought to be made here.

Section 5.3 goes on to say:

    Generating agents MUST NOT end a paragraph with such a signature
    line, since doing so would indicate that the separator line is part
    of the paragraph.

It would not indicate that the separator line is part of the paragraph,
it would indicate that the body is malformed (according to the grammar
and according to section 5.1); the receiver would not believe that the
separator is part of the paragraph (according to 5.1).  Perhaps the
intention is something like this:

    When placing soft line breaks in a paragraph, generating agents MUST
    NOT place them in a way that causes any line of the paragraph to
    be a signature separator line, because paragraphs cannot contain
    signature separator lines (see sections 5.1 and 7).

Section 5.4 says:

    Space-stuffing adds a single space to the start of any line which
    needs protection when the message is generated.  On reception, if
    the first character of a line is a space, it is logically deleted.
    This occurs after the test for a quoted line, and before the test
    for a flowed line.

It's not only after testing for a quoted line, but more importantly
after stripping the quoting.  And it's not only before the test for
a flowed line, but also before the test for a separator line.  Maybe
change the last sentence to:

    This occurs after deleting quote marks, and before testing for
    fixed, flowed, and separator lines.

Section 5.5 says:

    When generating quoted flowed lines, an agent needs to pay attention
    to changes in quote depth.  A sequence of quoted lines of the same
    quote depth immediately followed by lines of a different quote
    depth MUST be encoded so that lines of the same quote depth are a
    paragraph, with the last line generated as fixed and prior lines
    generated as flowed.

That seems to be a much stronger requirement than you intend.  Within a
single quote depth, there might be multiple paragraphs, non-paragraph
fixed-lines, and separator lines.  But the sentence quoted above seems
to say that because all of this text is a bunch of "lines at the same
quote depth", it must be encoded as "a paragraph", with the last line
fixed and all other lines flowed.  Perhaps the intention is something
like this:

    When generating quoted flowed lines, an agent needs to pay attention
    to changes in quote depth.  All lines of a paragraph MUST be
    unquoted, or else they MUST all be quoted and have the same quote
    depth.  Therefore, whenever there is a change in quote depth, or a
    change from quoted to unquoted, or change from unquoted to quoted,
    the line immediately preceeding the change MUST NOT be a flowed
    line.

The wording could be simplified if an unquoted line were simply a line
with a quote depth of zero:

    When generating quoted flowed lines, an agent needs to pay attention
    to changes in quote depth.  All lines of a paragraph MUST have the
    same quote depth.  Therefore, whenever there is a change in quote
    depth, the line immediately preceeding the change MUST NOT be a
    flowed line.

Section 5.5 goes on to say:

    If a receiving agent wishes to reformat flowed quoted lines (joining
    and/or wrapping them) on display or when generating new messages,
    the lines SHOULD be de-quoted, reformatted, and then re-quoted.  To
    de-quote, the number of close angle brackets in the quote indicator
    at the start of each line is counted.  Consecutive lines with the
    same quote depth are considered one paragraph and are reformatted
    together.  To re-quote after reformatting, a quote indicator
    containing the same number of close angle brackets originally
    present are prefixed to each line.

I think one sentence there is inaccurate: "Consecutive lines with the
same quote depth are considered one paragraph and are reformatted
together."  Consecutive lines with the same quote depth could be one
paragraph or several paragraphs or non-paragraph fixed lines (in which
case no reformatting is requested) or separator lines.  I think that
sentence can simply be removed.  Reformatting is covered elsewhere; this
section is about quoting.

The next two paragraphs are inconsistent with section 5.1:

    On reception, if a change in quote depth occurs on a flowed line,
    this is an improperly formatted message.  The receiver SHOULD handle
    this error by using the 'quote-depth-wins' rule, which is to ignore
    the flowed indicator and treat the line as fixed.  That is, the
    change in quote depth ends the paragraph.

    In other words, whenever two adjacent lines have different quote
    depths, senders MUST ensure that the earlier line is fixed (does
    not end in a space), and receivers SHOULD treat the earlier line as
    fixed regardless of whether it ends with a space.

According to section 5.1, the paragraph ends with the flowed line; it is
possible therefore to have an improperly terminated paragraph consisting
of a single flowed line, and such a paragraph would be reformatted.  If
the flowed indicator is ignored and the line is treated as fixed, then
we have a single fixed line, which is not a paragraph at all and would
not be reformatted.  Also, it is possible for the line before the change
in quote depth to be a separator line, which is arguably not fixed (see
the discussion above).  The inconsistency could be resolved like so:

    ...the 'quote-depth-wins rule', which is to consider the paragraph
    to end with the flowed line immediately preceeding the change in
    quote depth.

    In other words, whenever two adjacent lines have different quote
    depths, senders MUST ensure that the earlier line is not flowed
    (does not end in a space), and receivers finding a flowed line there
    SHOULD treat it as the last line of a paragraph.

Here we have more instances of the phrase "change in quote depth".  If
we keep the current view that unquoted lines have no quote depth and
quoted lines have non-zero quote depth, then we really ought to be
saying "change in quote depth, or change from quoted to unquoted, or
change from unquoted to quoted".  If we adopt the view that all lines
have a quote depth, which can be zero, then the simple phrase "change in
quote depth" will mean what we want it to mean.

AMC

To: Cyrus Daboo <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
From: Simon Josefsson <[email-masked]>
Date: Mon, 17 Nov 2003 02:55:37 +0100
Sender: [email-masked]

Cyrus Daboo <[email-masked]> writes:

> Hi Simon,
>
> --On Saturday, November 15, 2003 2:42 +0100 Simon Josefsson
>   <[email-masked]> wrote:
>
> | Receive (MIME-aware, PGP aware): CTE decode PGP armor, verify, CTE
> |   body, display.
>
> Sorry - but this is a non-starter - you are expecting existing
> deployed clients to magically change their behaviour to cope with
> format=flowed inline signed messages.

No, I expect clients to follow specifications, which I think is about
all we can hope for.  If deployed clients follow section 4.6 of RFC
2646 they already treat inline format=flowed differently.  They would
also be incompatible with both our proposals.  I don't believe many
clients follow the recommendation in the RFC though.  When fixing the
RFC text, we might as well evaluate all options available.  If all
deployed clients implemented format=flowed inline PGP in the same way,
and that way worked, then of course we should change to that method.
But my understanding is that this isn't the case.

> The only way to ensure format=flowed inline signed messages work
> with existing clients is to use the procedure I outlined

It would not work with clients that follow RFC 2646.  It would not
work with any client that compute the OpenPGP signature over the QP
encoded ("wire") data.  Are there no such clients?  I believe there
are gateways and some plugins that fall in the latter category.  But
if I'm mistaken, it may be worthwhile to add a description of your
proposal to 2646bis, to document earlier practices.  Of course, the
recommended solution should still be PGP/MIME.

Regards,
Simon

Date: Sun, 16 Nov 2003 17:24:32 -0800
To: [email-masked], Keith Moore <[email-masked]>
From: Randall Gellens <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Cc: Randall Gellens <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

At 3:28 PM -0800 11/16/03, [email-masked] wrote:

>  As long as the document makes it clear
>  we're dealing with the characters that result from the application of the
>  charset to the sequence of octets we should be good to go.

Here is the sentence I added in -03 (in the ABNF section):

     Note that the SP (space) and ">" characters are encoded according 
to the charset parameter.

Do you think this is sufficient?
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
The idea that Bill Gates has appeared like a knight in shining armor
to lead all customers out of a mire of technological chaos neatly
ignores the fact that it was he who, by peddling second-rate technology,
led them into it in the first place.                    --Douglas Adams

Date: Sun, 16 Nov 2003 19:23:41 -0500
From: Cyrus Daboo <[email-masked]>
To: Simon Josefsson <[email-masked]>
cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Sender: [email-masked]

Hi Simon,

--On Saturday, November 15, 2003 2:42 +0100 Simon Josefsson 
<[email-masked]> wrote:

| Receive (MIME-aware, PGP aware): CTE decode PGP armor, verify, CTE
|   body, display.

Sorry - but this is a non-starter - you are expecting existing deployed 
clients to magically change their behaviour to cope with format=flowed 
inline signed messages. The only way to ensure format=flowed inline signed 
messages work with existing clients is to use the procedure I outlined (or 
just not use format=flowed). Yes it is vulnerable to a man-in-the-middle 
attack but that is true for anything that does not also sign the message 
headers. A man-in-the-middle that changes text/plain to text/html will 
result in pretty much the same display 'corruption' without the need to 
even change the body content, so its not a problem specific to 
format=flowed.

-- 
Cyrus Daboo

Date: Sun, 16 Nov 2003 15:28:11 -0800 (PST)
From: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
To: Keith Moore <[email-masked]>
Cc: Randall Gellens <[email-masked]>, [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Sender: [email-masked]

> On Sun, 16 Nov 2003 14:54:54 -0800
> Randall Gellens <[email-masked]> wrote:

> > One thing I'm not totally sure about is the encoding of space and
> > ">".  I added a note to the ABNF section that says that these
> > characters are encoded according to the charset, but as I recall,
> > during the discussions on how f=f can work with non-Western
> > languages/charsets, especially Chinese, Japanese and Korean, it was
> > mentioned that ASCII space is sometimes used in some of these
> > languages.  So perhaps the statement needs to include the possibility
> > that space might be encoded in ASCII as well?

> I guess the question is what to do when there is more than one way to
> represent the space character and/or ">", as can certainly happen
> when using iso-2022 code-switching.

This is one of the reasons why charsets are defined as a mapping from octets to
characters, not the other way around: It lets us talk about the processing of
various characters without having to worry about whether there is one way to
represent them or fifty. If follows that simply scanning the content for 0x20
and 0x3E is unacceptable; the transformation from octets to characters must be
performed first, then the resulting sequence of *characters* can be checked for
space, greater than, etc. 

We got this wrong in text/richtext but we fixed it in text/enriched, although
the language in RFC 1896 isn't as clean as I would like. Let's please not
repeat this whole argument yet again. As long as the document makes it clear
we're dealing with the characters that result from the application of the
charset to the sequence of octets we should be good to go.

				Ned

P.S. I note in passing that while iso-2022 does allow for very general things,
including the ability to have the same character bound to multiple different
octet values at the same time and for common characters like space and greater
than to be bound to less than obvious octet values, in practice real charsets
defined in terms of iso-2022 tend to restrict themselves to a very small subset
of iso-2022's capabilities. The result is that while you cannot assume that,
say, 0x3E is always a greater than, when it appears greater than will be
represented as 0x3E. I haven't found this characteristic to be particularly
helpful when coding support for this stuff, but it sure helps a lot when
inspecting input iso-2022-based text manually.

Date: Sun, 16 Nov 2003 17:53:50 -0500
From: Keith Moore <[email-masked]>
To: Randall Gellens <[email-masked]>
Cc: [email-masked], [email-masked], [email-masked], [email-masked], [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-03)
Sender: [email-masked]

On Sun, 16 Nov 2003 14:54:54 -0800
Randall Gellens <[email-masked]> wrote:

> One thing I'm not totally sure about is the encoding of space and 
> ">".  I added a note to the ABNF section that says that these 
> characters are encoded according to the charset, but as I recall, 
> during the discussions on how f=f can work with non-Western 
> languages/charsets, especially Chinese, Japanese and Korean, it was 
> mentioned that ASCII space is sometimes used in some of these 
> languages.  So perhaps the statement needs to include the possibility 
> that space might be encoded in ASCII as well?

I guess the question is what to do when there is more than one way to
represent the space character and/or ">", as can certainly happen 
when using iso-2022 code-switching.

Date: Sun, 16 Nov 2003 14:54:54 -0800
To: IETF RFC-822 list <[email-masked]>, "Adam M. Costello" <[email-masked]>, Keith Moore <[email-masked]>, [email-masked]
From: Randall Gellens <[email-masked]>
Subject: Format=Flowed/RFC 2646 Bis (-03)
Cc: [email-masked]
Sender: [email-masked]

I've created a -03 based on comments received in the past few days.

Changes from -02:

* Added mention of quoting to Abstract and Introduction.
* Deleted line analysis table.
* Added MUST NOT for OpenPGP and SHOULD for OpenPGP-MIME.
* Replaced ABNF rules to remove ambiguity
* Added note that c-t-e is irrelevant to flowed text processing
* Added text indicating that end of data terminates a paragraph
* Moved sig-sep out of fixed-line ABNF
* Changed some SHOULDs to MUSTs (space-stuffing, quoted paragraphs)
* Added note to ABNF that space and ">" are encoded according to charset
* Mentioned exceptions in section on interpreting
* Moved section on interpreting before section on generating.
* Reworded non-normative "should"s.

One thing I'm not totally sure about is the encoding of space and 
">".  I added a note to the ABNF section that says that these 
characters are encoded according to the charset, but as I recall, 
during the discussions on how f=f can work with non-Western 
languages/charsets, especially Chinese, Japanese and Korean, it was 
mentioned that ASCII space is sometimes used in some of these 
languages.  So perhaps the statement needs to include the possibility 
that space might be encoded in ASCII as well?

The updated text is available at 
<ftp://ftp.pensive.org/Public/Randy/draft-gellens-format-bis-03.txt>
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
You have the right to remain silent.  Anything you say will be
misquoted, then used against you.

To: Cyrus Daboo <[email-masked]>
Cc: Charles Lindsey <[email-masked]>, [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
From: Simon Josefsson <[email-masked]>
Date: Sat, 15 Nov 2003 02:42:20 +0100
Sender: [email-masked]

Cyrus Daboo <[email-masked]> writes:

> Hi Simon,
>
> --On Friday, November 14, 2003 22:46 +0100 Simon Josefsson
>   <[email-masked]> wrote:
>
> | I haven't seen a way to support both inline PGP and format=flowed
> | without creating one problem or another.
>
> That's been my experience and in fact I deliberately turn off
> format=flowed when generating inline PGP signed messages. It remains
> on for PGP/MIME messages (which is the default for new users).

I'm coming to the same conclusion myself, but it would be nice if
people implementing this aren't forced to handle the same problems.
One way to achieve that goal is to include text in the document.

> | I'd love to be proved wrong on this, since currently I'm stuck in my
> | implementation of inline PGP and format=flowed, because of my
> | perceived problems.
> |
> | I thing Gellens is right in that the there are more modes than what
> | you describe.  There are at least 4 incompatible ways to do inline
> | PGP; QP before, QP after sign (but don't touch PGP armor), QP after
> | sign (including PGP armor), don't QP at all.  I have seen all of them
> | in the wild.  Incidentally, my experiences is that the last one works
> | best, although it breaks the most specifications.  The format=flowed
> | draft says only one of the SHOULD be used even though it violate MIME.
>
> I think you need to look at this problem from the recipients end to
> actually decide what it is the sender needs to do. In particular its
> important that the signature verify for both types of recipients:
> those that are format=flowed aware and those that aren't. The later
> type effectively determines the order of processing for flowing, PGP
> signing and CTE:
>
> A non-format=flowed aware client, when processing a received
> format=flowed message, will first do CTE decoding and then display
> the message. Verification of the signature will then be done on the
> unflowed text (i.e. with trailing spaces and CRLFs etc).

Following the current format=flowed specification, the client would
encounter errors during QP decode, because the '=' within the base64
encoded PGP armor is not QP encoded.

> Thus a format=flowed aware client that generates a message would first
> wrap the text doing flow, then do the PGP signature, and then do CTE
> encoding. That is the only way to work with non-aware clients.

It is not the only way that would work, here's another that I believe
is better:   ('PGP aware' means aware of inline PGP, and 'format=flowed
aware' imply 'MIME-aware')

Generate: Do CTE encoding of message, sign, and CTE encode PGP armor.

Receive (non-MIME, non-PGP aware): display.

Receive (MIME-aware, non-PGP aware): CTE decode, display.

Receive (format=flowed aware, non-PGP aware): CTE decode,
  format=flowed process, display.

Receive (non-MIME aware, PGP aware): cannot detect valid PGP message,
  display.

Receive (MIME-aware, PGP aware): CTE decode PGP armor, verify, CTE
  body, display.

Receive (format=flowed aware, PGP aware): CTE decode PGP armor,
  verify, CTE decode body, format=flowed process, display.

Comparing your and my approaches: your is vulnerable to the
man-in-the-middle attack inserting trailing SPC and adding
format=flowed header, thus altering the rendering of the message.  The
PGP signature would not detect the added trailing SPC for you.  Both
approaches break for non-MIME aware, but PGP aware, clients.  Your
approach cannot interoperate with RFC 1991 implementions, but my can
(by QP encoding '^-' in the body).  Neither of our approaches can
handle non-ASCII characters within the PGP armor encoded in a charset
that isn't compatible with the body charset.

Using UTF-8 and 8-bit (no QP) is another option.  The advantage is
that it is possible pipe the incoming RFC 2822 article directly into
the OpenPGP implementation, so it would work for non-MIME aware, but
PGP aware, clients.  It also support non-ASCII in PGP armor.  The
(only?) disadvantage is that trailing whitespace may be lost in
transit, invalidating the signature.  But it still works much better
in practice than any of the QP variants.

Regards,
Simon

Date: Fri, 14 Nov 2003 18:56:00 -0600
From: Cyrus Daboo <[email-masked]>
To: Simon Josefsson <[email-masked]>, Charles Lindsey <[email-masked]>
cc: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Sender: [email-masked]

Hi Simon,

--On Friday, November 14, 2003 22:46 +0100 Simon Josefsson 
<[email-masked]> wrote:

| I haven't seen a way to support both inline PGP and format=flowed
| without creating one problem or another.

That's been my experience and in fact I deliberately turn off format=flowed 
when generating inline PGP signed messages. It remains on for PGP/MIME 
messages (which is the default for new users).

| I'd love to be proved wrong on this, since currently I'm stuck in my
| implementation of inline PGP and format=flowed, because of my
| perceived problems.
|
| I thing Gellens is right in that the there are more modes than what
| you describe.  There are at least 4 incompatible ways to do inline
| PGP; QP before, QP after sign (but don't touch PGP armor), QP after
| sign (including PGP armor), don't QP at all.  I have seen all of them
| in the wild.  Incidentally, my experiences is that the last one works
| best, although it breaks the most specifications.  The format=flowed
| draft says only one of the SHOULD be used even though it violate MIME.

I think you need to look at this problem from the recipients end to 
actually decide what it is the sender needs to do. In particular its 
important that the signature verify for both types of recipients: those 
that are format=flowed aware and those that aren't. The later type 
effectively determines the order of processing for flowing, PGP signing and 
CTE:

A non-format=flowed aware client, when processing a received format=flowed 
message, will first do CTE decoding and then display the message. 
Verification of the signature will then be done on the unflowed text (i.e. 
with trailing spaces and CRLFs etc).

Thus a format=flowed aware client that generates a message would first wrap 
the text doing flow, then do the PGP signature, and then do CTE encoding. 
That is the only way to work with non-aware clients.

That means a format=flowed aware client, when processing a received 
format=flowed message, will first do CTE decoding. At that point it will 
need to check for inline PGP and verify, then do flowing. Alternatively, it 
will do CTE decode, flowing and display to the user. But then if the user 
wants to verify, it will have to go back to the CTE decoded but not flowed 
version.

All-in-all this is a mess. Frankly I think its better to state that 
format=flowed SHOULD NOT be done on inline PGP signatures, instead PGP/MIME 
SHOULD be used instead.

-- 
Cyrus Daboo

To: "Charles Lindsey" <[email-masked]>
Cc: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
From: Simon Josefsson <[email-masked]>
Date: Fri, 14 Nov 2003 22:46:54 +0100
Sender: [email-masked]

"Charles Lindsey" <[email-masked]> writes:

> In <[email-masked]> Simon Josefsson <[email-masked]> writes:
>
>>The problem is that the above procedure is flawed, so it is not always
>>possible to use it in the real world.  There are several problems with
>>that text, I believe the two major issues are:
>
>>1) It leads to invalid MIME messages on the wire.  After following the
>>   above procedure, what is sent is a message marked with CTE qp but
>>   only the contents of the inline PGP message actually follow the QP
>>   rules.  The PGP armor itself do not follow the QP rules.
>
> No, I don't think that is right. There are basically two ways of signing
> PGP messages:
>
> 1. The "usual" way. You construct a text A, pass it through a PGP signing
> engine to get text B (it will do nasty things like changing every initial
> "---" on your lines to "- --"). Text B will contain the usual PGP
> wrappers, your text, and the PGP sig.
>
> You then email Text B, and if you can persuade you mailer to encode it as
> 7bit, then any trailing spaces (which were not included in the PGP hash,
> but are still present in your text) will be protected against munging en
> route. Note that it is your original text that is signed, not the QP
> version.

I agree with you that it isn't right, but it is what the draft says.
That's my point.  The text was quoted in my replace, but here is the
text again:

5.6.  Digital Signatures and Encryption

    If a message is digitally signed or encrypted it is important that
    cryptographic processing use the on-the-wire Format=Flowed format.
    That is, during generation the message SHOULD be prepared for
    transmission, including addition of soft line breaks,
    space-stuffing, and [Quoted-Printable] encoding (to protect soft
    line breaks) before being digitally signed or encrypted; similarly,
    on receipt the message SHOULD have the signature verified or be
    decrypted before [Quoted-Printable] decoding and removal of stuffed
    spaces, soft line breaks and quote marks, and reflowing.

The text say that MUAs SHOLD do things differently from what you
describe.

FWIW, your approach has the security problem that started this thread;
it makes it possible for someone to in transit add trailing SPC to the
PGP message, and add a format=flowed tag to the headers, without being
detected.

I haven't seen a way to support both inline PGP and format=flowed
without creating one problem or another.

Btw, it is not clear if the paragraph apply to PGP/MIME too.  Because
it uses generic terminology, I assume it apply to all cryptographic
formats.  RFC 3156 already discuss these issues, repeating them again
with normative text might just be confusing.  And my claim is that,
for inline PGP, the normative text doesn't work.

I'd love to be proved wrong on this, since currently I'm stuck in my
implementation of inline PGP and format=flowed, because of my
perceived problems.

I thing Gellens is right in that the there are more modes than what
you describe.  There are at least 4 incompatible ways to do inline
PGP; QP before, QP after sign (but don't touch PGP armor), QP after
sign (including PGP armor), don't QP at all.  I have seen all of them
in the wild.  Incidentally, my experiences is that the last one works
best, although it breaks the most specifications.  The format=flowed
draft says only one of the SHOULD be used even though it violate MIME.

> 2. Use RFC 3156 (PGP/MIME).

Right.  This is what I believe should be used instead.

> Just for the hell of it, I shall sign this message both ways (so the outer
> signature will actually sign the inner one).

It wasn't a good example, since you didn't use QP.  For pure 7-bit
messages, there are no ambiguities at all, as far as I know.  Well,
except that you might prefer to use QP to protect leading '-' from
hash escaping ('- -'), otherwise RFC 1991 implementation cannot
understand the message.  This is also not written down anywhere, I
believe.

Regards,
Simon

To: [email-masked]
Xref: clerew local.mime:2767
Newsgroups: local.mime
Path: clerew!chl
From: "Charles Lindsey" <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Date: Fri, 14 Nov 2003 15:50:57 GMT
Lines: 88
Sender: [email-masked]

This is a PGP signed message according to RFC 2015

--88888888

-----BEGIN PGP SIGNED MESSAGE-----

In <[email-masked]> Simon Josefsson <[email-masked]> writes:

>The problem is that the above procedure is flawed, so it is not always
>possible to use it in the real world.  There are several problems with
>that text, I believe the two major issues are:

>1) It leads to invalid MIME messages on the wire.  After following the
>   above procedure, what is sent is a message marked with CTE qp but
>   only the contents of the inline PGP message actually follow the QP
>   rules.  The PGP armor itself do not follow the QP rules.

No, I don't think that is right. There are basically two ways of signing
PGP messages:

1. The "usual" way. You construct a text A, pass it through a PGP signing
engine to get text B (it will do nasty things like changing every initial
"---" on your lines to "- --"). Text B will contain the usual PGP
wrappers, your text, and the PGP sig.

You then email Text B, and if you can persuade you mailer to encode it as
7bit, then any trailing spaces (which were not included in the PGP hash,
but are still present in your text) will be protected against munging en
route. Note that it is your original text that is signed, not the QP
version.

2. Use RFC 3156 (PGP/MIME). This time the overall message is a
multipart/related. The first part contains your text, with CTE QP (thus
there are no trailing spaces remaining).

The PGP signature goes in the second part, which may or may not be encoded
(probably not, as it is all pure ASCII). Note that in this method it is
the QP version of the text that gets signed, which means that if some
well-meaning intermediate site undoes the QP, then the signature is
broken.

Just for the hell of it, I shall sign this message both ways (so the outer
signature will actually sign the inner one). Note that the inner signature
does not cover my sig at the bottom, but you can if you like watch out for
the trailing SP in my sig separator. Note that none of it will be QP
encoded because I cannot persuade this particular system to put it in QP,
so you will have to take that part on trust.

And here is a line commencing with "---", so you can see how PGP munges it
(when you check the signature, that will get undone).

- -------------------

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131 Fax: +44 161 436 6133   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

--88888888
Content-Type: application/pgp-signature

--88888888--

To: Randall Gellens <[email-masked]>, [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
From: Simon Josefsson <[email-masked]>
Date: Fri, 14 Nov 2003 01:44:18 +0100
Sender: [email-masked]

Randall Gellens <[email-masked]> writes:

> At 7:59 PM +0100 11/3/03, Simon Josefsson wrote:
>
>>  Thanks for adding the OpenPGP discussion.  Given the subtleness of the
>>  issue, I believe the document should not only mention it, but also
>>  give normative advice on how the combination of OpenPGP and
>>  format=flowed is to be implemented.  Otherwise implementors will
>>  ignore the problem, as they do today.
>>
>>  When I look at how to properly implement both OpenPGP and
>>  format=flowed, I can't come to any other conclusion than that security
>>  is more important than maintaining soft paragraph breaks.  That means
>>  a client should not flow OpenPGP signed data, when it present the
>>  outcome as something that OpenPGP guarantee is what the sender sent.
>>  If the client would flow a message, someone in transit may modify the
>>  rendering of a message without being detected by OpenPGP.
>>
>>  Repeating the text from RFC 2440, saying that PGP/MIME aka RFC 3156
>>  SHOULD be used in messaging applications, may be sufficient.  Perhaps
>>  promote it to MUST within the scope of flowed messages.
>
> The current text says to use quoted-printable to protect the trailing
> spaces so that the signature is calculated on the on-the-wire format:
>
> 5.6.  Digital Signatures and Encryption
>        If a message is digitally signed or encrypted it is important
>        that
>      cryptographic processing use the on-the-wire Format=Flowed format.
>      That is, during generation the message SHOULD be prepared for
>      transmission, including addition of soft line breaks,
>      space-stuffing, and [Quoted-Printable] encoding (to protect soft
>      line breaks) before being digitally signed or encrypted; similarly,
>      on receipt the message SHOULD have the signature verified or be
>      decrypted before [Quoted-Printable] decoding and removal of stuffed
>      spaces, soft line breaks and quote marks, and reflowing.
>        Note that [OpenPGP] specifies (in section 7.1) that "any
>        trailing
>      whitespace (spaces, and tabs, 0x09) at the end of any line is
>      ignored when the cleartext signature is calculated."
>        Thus it would be possible to add, in transit, a format=flowed
>        header
>      to a regular, format=fixed vanilla PGP (not PGP/MIME) signed message
>      and add arbitrary trailing space characters without this addition
>      being detected.  This would change the rendering of the article by a
>      client which supported format=flowed.
>
> In thinking about this some more, I'm not sure that the extra text on
> OpenPGP is really needed, since if the text above is followed it
> shouldn't be an issue.

The problem is that the above procedure is flawed, so it is not always
possible to use it in the real world.  There are several problems with
that text, I believe the two major issues are:

1) It leads to invalid MIME messages on the wire.  After following the
   above procedure, what is sent is a message marked with CTE qp but
   only the contents of the inline PGP message actually follow the QP
   rules.  The PGP armor itself do not follow the QP rules.  More
   precisely, the base64 '=' character that is always part of the
   final CRC value would be a unescaped '=' character.  So transit or
   recipient systems might reject it due to invalid QP, and IMAP
   servers might refuse to search within the message, or refuse it
   altogether because it isn't valid MIME.

   This problem could be solved by performing QP encoding on the PGP
   armor itself as well, after signing the QP encoded message body.
   The text above does not suggest that (there is even a SHOULD
   arguing otherwise), and existing clients do not appear to perform
   QP on the PGP armor (at least not those I'm familiar with).
   Furthermore, because existing clients do not QP encode to the PGP
   armor, I doubt existing clients would understand they have to first
   QP decode the PGP armor, and then pass the resulting QP decoded PGP
   armor plus QP encoded body to the PGP implementation.  Still,
   adding such a clarification would be a way forward.

2) Non-ASCII text within the armor (e.g., 'Comments:') do exist, some
   translated products even add such strings by default.  If the
   charset used for the armor isn't the same as the message body, it
   is not even possible to QP for the armor.  I don't see a solution
   to this problem.  It has been argued in the past that the PGP armor
   cannot contain non-ASCII, but I believe it remains to be clarified
   since it is frequent in the wild.

Saying that cryptographic processing is something that should be done,
on the generating side, after all 822 and MIME processing, and on the
receiving side, before all 822 and MIME processing, does not work.  It
leads to the above problems.

Before we loose sight of the big picture, I'd like to repeat that none
of these issues apply if PGP/MIME is used, which is what RFC 2440 says
SHOULD be used.  (Well, at least, the problems do not _necessarily_
apply; PGP/MIME can be implemented incorrectly that leads to similar
problems.)  Unless it can be demonstrated that implementing PGP and
format=flowed is possible without breaking the MIME standard, I still
maintain that qualifying the SHOULD to a MUST, within the scope of
format=flowed messages, is better.

Thanks,
Simon

Date: Sun, 9 Nov 2003 11:13:34 -0800 (PST)
From: [email-masked]
To: undisclosed-recipients:;
Sender: [email-masked]

pÚP
Message-Id: <p06100404bbd438dd4fff@[12.162.212.214]>
In-Reply-To: <[email-masked]>
References: <p06100308bbc1f72bb5de@[192.168.1.13]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
 <[email-masked]>
X-Mailer: Eudora for Mac OS X v6.1a
Date: Sun, 9 Nov 2003 11:12:52 -0800
To: IETF RFC-822 list <[email-masked]>
From: Randall Gellens <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Cc: "Adam M. Costello" <[email-masked]>,
 Keith Moore <[email-masked]>, [email-masked]
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii" ; format="flowed"
X-Random-Sig-Tag: 1.0b26

Adam and Keith have obviously spent a lot of time thinking about the 
draft and the underlying issues, and offering specific suggestions 
for improvement.  I am very grateful for this (although it would have 
been even better had this activity occurred sometime earlier in the 
past few years, since most of the text in question hasn't changed in 
some time.)  It's not too late, of course, and it's always preferable 
to improve the clarity of a document prior to publication.

On the questions of the terms "line" and "paragraph", I think it is 
best to keep the concept of "line" as a CRLF-terminated sequence of 
characters (in the canonical encoding).  That way "line" remains a 
familiar concept, and we can talk abut "fixed lines", "flowed lines", 
"quoted lines", and so on.  I also don't see the problem with calling 
a group of lines intended to be re-flowed a "paragraph".  If there is 
a consensus that this term is harmful I'm happy to change it, but 
currently I don't see much benefit in doing so.

Keith raises an excellent point about charsets and encoding of ">" 
and space.  Perhaps a clarifying note in the ABNF section that 
indicates that these characters are encoded according to the charset 
parameter?

Adam suggests deleting 5.7.  I have to admit I'm not enamored of this 
section and wouldn't mind deleting it if there is any consensus that 
this improves the document.

Adam also had a number of concerns over ambiguity in the grammar, 
with suggestions for improvement.  I generally like the replacement 
text, except for the removal of the distinction between quoted and 
unquoted lines.  I thought it was helpful to identify a non-quoted 
line on its own, and not just as a line with a quote-depth of zero. 
This is based on a perceived need to treat the two somewhat 
differently, in particular, quoted lines need extra handling, state, 
and display semantics.

Adam proposes labelling the space before the CRLF, as it does have 
importance and needs to be treated differently depending on the value 
of DelSp.  This seems like a good idea, and either "soft" or "flow" 
sound like reasonable names to me.

Adam also had a number of suggestions for moving text around to 
improve the readability of the document, such as having the section 
on interpreting appear before the section on generating.  I'm kind of 
neutral on this; and would appreciate hearing other opinions as to if 
this would be helpful or not.

Adam also suggests mentioning in the section on interpreting f=f the 
exceptions for Usenet signatures and  changing-quote-depth, which 
seem like good ideas to me.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
People demand freedom of speech to make up for the freedom of thought
which they avoid.                                       --Kierkegaard

Date: Sun, 9 Nov 2003 08:47:56 -0800
To: Simon Josefsson <[email-masked]>, Randall Gellens <[email-masked]>
From: Randall Gellens <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Cc: [email-masked]
Sender: [email-masked]

At 7:59 PM +0100 11/3/03, Simon Josefsson wrote:

>  Thanks for adding the OpenPGP discussion.  Given the subtleness of the
>  issue, I believe the document should not only mention it, but also
>  give normative advice on how the combination of OpenPGP and
>  format=flowed is to be implemented.  Otherwise implementors will
>  ignore the problem, as they do today.
>
>  When I look at how to properly implement both OpenPGP and
>  format=flowed, I can't come to any other conclusion than that security
>  is more important than maintaining soft paragraph breaks.  That means
>  a client should not flow OpenPGP signed data, when it present the
>  outcome as something that OpenPGP guarantee is what the sender sent.
>  If the client would flow a message, someone in transit may modify the
>  rendering of a message without being detected by OpenPGP.
>
>  Repeating the text from RFC 2440, saying that PGP/MIME aka RFC 3156
>  SHOULD be used in messaging applications, may be sufficient.  Perhaps
>  promote it to MUST within the scope of flowed messages.

The current text says to use quoted-printable to protect the trailing 
spaces so that the signature is calculated on the on-the-wire format:

5.6.  Digital Signatures and Encryption
    
     If a message is digitally signed or encrypted it is important that
     cryptographic processing use the on-the-wire Format=Flowed format.
     That is, during generation the message SHOULD be prepared for
     transmission, including addition of soft line breaks,
     space-stuffing, and [Quoted-Printable] encoding (to protect soft
     line breaks) before being digitally signed or encrypted; similarly,
     on receipt the message SHOULD have the signature verified or be
     decrypted before [Quoted-Printable] decoding and removal of stuffed
     spaces, soft line breaks and quote marks, and reflowing.
    
     Note that [OpenPGP] specifies (in section 7.1) that "any trailing
     whitespace (spaces, and tabs, 0x09) at the end of any line is
     ignored when the cleartext signature is calculated."
    
     Thus it would be possible to add, in transit, a format=flowed header
     to a regular, format=fixed vanilla PGP (not PGP/MIME) signed message
     and add arbitrary trailing space characters without this addition
     being detected.  This would change the rendering of the article by a
     client which supported format=flowed.

In thinking about this some more, I'm not sure that the extra text on 
OpenPGP is really needed, since if the text above is followed it 
shouldn't be an issue.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Computers ... are not designed, as we are, for ambiguity.  --Thomas

Date: Sat, 8 Nov 2003 19:07:51 +0000
From: "Adam M. Costello" <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Keith Moore <[email-masked]> wrote:

> I avoided using the term "paragraph" because that has semantics other
> than for presentation.

Okay, but I think we need to keep the term "line" referring to its
intuitive and customary sense, so that we can say things like "each
line may begin with one or more quote characters" and "a space at the
beginning or end of a line (after quoting has been removed) acts as a
flag".

I think it would be confusing, therefore, to use a term like "flowed
line" to refer to something that spans multiple lines.  I think the
current meaning of "flowed line" (a line that ends in a space) is more
intuitive.

If "paragraph" needs to be avoided, perhaps a new term could be
invented, like "snake" or "flow-group".

> I don't like using the term "soft" for the SP CR LF sequence in a
> flowed line because it's too easily confused with Q-P soft line
> breaks, even though they have nothing to do with one another.

I was using "soft" to refer just to the SP, not the CR LF.  I was
calling the single spaces at the beginning and end of the line
"stuffing" and "soft", respectively.  Perhaps better names would be
"stuff" and "flow".

> I'm concerned that people will implement format=flowed differently
> for objects depending on the content-transfer-encoding, when the
> content-transfer-encoding should be irrelevant.

That ought to follow from the MIME architecture (format=flowed is a
parameter of the content-type, therefore the content-transfer-encoding
is indeed irrelevant), but it wouldn't hurt to mention it explicitly.

> It's more important to clean up the text than the grammar.

Try this:  Forget everything you've read in the draft and just read this
excerpt (45 lines):
    
    If the first character of a line is a quote mark (">"), the line is
    considered to be quoted (see section 5.5).  Logically, all quote
    marks are counted and deleted, resulting in a line with a non-zero
    quote depth, and content. (The agent is of course free to display
    the content with quote marks or excerpt bars or anything else.)
    Logically, this test for quoted lines is done before any other tests
    (that is, before checking for space-stuffed and flowed).
    
    If the first character of a line is a space, the line has been
    space-stuffed (see section 5.4).  Logically, this leading space is
    deleted before examining the line further (that is, before checking
    for flowed).

    If the line ends in one or more spaces, the line is flowed.
    Otherwise it is fixed.
    
    If the line is flowed and DelSp is "yes", the trailing space
    immediately prior to the line's CRLF is logically deleted.  If the
    DelSp parameter is "no" (or not specified, or set to an unrecognized
    value), the trailing space is not deleted.

    Any remaining trailing spaces are part of the line's content, but
    the CRLF of a soft line break is not.
    
    A series of one or more flowed lines followed by one fixed line is
    considered a paragraph, and MAY be flowed (wrapped and unwrapped) as
    appropriate on display and in the construction of new messages (see
    section 5.5).
    
    A line consisting of one or more spaces (after deleting a stuffed
    space) is considered a flowed line.
    
    An empty line (just a CRLF) is a fixed line.

    There is a convention in Usenet news of using "-- " as the separator
    line between the body and the signature of a message.  When
    generating a Format=Flowed message containing a Usenet-style
    separator before the signature, the separator line is sent as-is.
    This is a special case; an (optionally quoted) line consisting of
    DASH DASH SP is not considered flowed.

    whenever two adjacent lines have different quote depths, senders
    should ensure that the earlier line is fixed (does not end in
    a space), and receivers should treat the earlier line as fixed
    regardless of whether it ends with a space.

That's just sections 5.2 and 5.3 and one paragraph from 5.5.  I find
that if I focus only on this 45-line excerpt and ignore the rest, I
completely understand format=flowed and delsp=yes.  Do you agree (except
for the content-transfer-encoding concern)?

Perhaps it might help to move this material earlier.  Imagine swapping
sections 5.1 (Generating format=flowed) and 5.2 (Interpreting
format=flowed).  I find it much easier to understand a format first from
a decoder's point of view ("this is what can appear, and this is what
it means").  Then, after I understand the format, I can more easily
understand rules/recommendations aimed at encoders.

I think it might also help to include all the essentials of the decoding
algorithm in the interpreting section.  Right now, two details are
omitted and sprung on the reader in later sections: the Usenet sig
exception and the changing-quote-depth exception.  The interpreting
section could include them concisely and provide forward references,
just like it already does for quoting and space-stuffing.

It's easier to follow an explanation if you have a preview of where
it's heading.  Consider an interpreting section that began with such a
preview and then gave all the steps:

    An interpreter of format=flowed text processes the text line by
    line.  ("Line" refers to lines of the text/plain data, because
    format=flowed is a parameter of text/plain.  "Line" does not refer
    to lines of any quoted-printable, base64, or other encoding of the
    text/plain data.)  Each line can have characters removed from the
    beginning and/or end, and each line is tagged with a quote-depth (a
    non-negative integer) and a flow-type ("fixed" or "flowed").  The
    tags are used to group lines into paragraphs/snakes/flow-groups that
    can be re-wrapped for display or construction of other messages.

    For each line (in order), the following steps are applied (in
    order):

    1. All quote marks (">") are removed from the beginning of the line
    and counted; the count becomes the quote-depth of the line. [other
    remarks] [forward reference to quoting section]

    2. If the line is not the first line, and if its quote-depth differs
    from the quote-depth of the previous line, then the previous line
    is expected to have a flow-type of "fixed".  In properly generated
    text, that will be true; if the previous line's flow-type is
    "flowed" then the text was generated improperly.  In that case,
    reset the flow-type of the previous line to "fixed", and re-do
    step 7 for that line. [forward reference to quoting section]

        [[ Suggestion:  Perhaps, instead of overriding the
        flow-type, the line should be left as flowed, but the
        paragraph/snake/flow-group is nevertheless terminated, resulting
        in an improper paragraph/snake/flow-group that does not end
        with a fixed line.  This would yield different treatment for a
        single flowed line followed by a change in quote-depth.  The
        existing rules change it to a single fixed line, which would not
        be re-wrapped.  But clearly it was intended to be re-wrappable.
        The suggested new rule would allow it to be re-wrapped.  The
        existing draft contains the sentence "the change in quote depth
        ends the paragraph", which is inconsistent with the rest of the
        existing changing-quote-depth rule, because when a single flowed
        line is changed to fixed, there is no paragraph.  But that
        sentence would be consistent with the suggested new rule. ]]

    3. If the line is "-- " (dash dash space) then set the flow-type
    to "fixed" and go on to the next line (do not proceed to step 4).
    [forward reference to Usenet sig section]

        [[ Suggestion:  Maybe a sig line should also terminate a
        paragraph/snake/flow-group, same as a change in quote-depth.
        Otherwise a sig line could get re-wrapped so that it no longer
        appears on a line by itself. ]]

    4. If the line begins with a space, the space is removed. [other
    remarks] [forward reference to space-stuffing section]

    5. If the line ends with a space then set the flow-type to "flowed",
    otherwise set it to "fixed".

    6. If the line is flowed (that is, it's flow-type is "flowed") and
    delsp is "yes" then remove the space at the end of the line.

    7. If the line is flowed then it is part of a growing
    paragraph/snake/flow-group.  If the line is fixed and is preceeded
    by a flowed line, then the fixed line is the last line of the
    paragraph/snake/flow-group, which MAY be wrapped and unwrapped as
    appropriate for display and construction of new messages.  If the
    line is fixed and is not preceeded by a flowed line, then it is not
    part of a paragraph/snake/flow-group. [forward reference] [Unicode
    reference].

        [[ Suggestion:  The current draft says nothing about what
        a decoder should do if the very last line is flowed.  I
        suggest that end-of-input be yet another thing (along with
        quote-depth changes and sig lines) that can improperly terminate
        a paragraph/snake/flow-group. ]]

    Note that multiple consecutive spaces have no special significance.
    Only the single spaces at the beginning and end of a line have a
    special meaning; any others are simply part of the line's content.
    Steps 4 and 6 delete at most one space each.

Would an early section along those lines be at all helpful?

By the way, I wonder what purpose section 5.7 serves.  It looks like
just a very verbose way of saying "A line is quoted iff it begins with a
quote indicator.  A line is flowed iff it ends with a space."  I don't
see the point of the first statement, because every quote depth needs
to be displayed differently--I see nothing special about the boundary
between zero and non-zero.  The second statement is imprecise because it
neglects the Usenet sig exception.  The author might want to consider
simply removing section 5.7.

AMC

Date: Thu, 06 Nov 2003 07:28:17 -0800 (PST)
From: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
To: Keith Moore <[email-masked]>
Cc: Randall Gellens <[email-masked]>, [email-masked], [email-masked]
Sender: [email-masked]

> I have to say that I find both RFC 2646 and this draft fairly opaque and
> somewhat ambiguous.

> In particular, what is a "line"?  is it:

> - zero or more characters from the canonical form of a body part,
>   beginning either at the start of the body part
>   or immediately following a CRLF, and ending with a CRLF?

> - zero or more characters from the encoded form of a body part,
>   beginning either at the start of the body part
>   or immediately following a CRLF, and ending with a CRLF
>   whether or not it is preceded by a SP?

> - zero or more characters from the canonical form of a body part,
>   beginning either at the start of the body part
>   or immediately following a CRLF, and ending with a CRLF
>   that isn't preceded by a SP?

I'll leave this for Randy to answer.

> In a charset that isn't compatible with ASCII, are the characters
> ">", SP, CR, LF treated specially using the values of those characters
> from that charset, or are the octet values 0x3E, 0x20, 0x0D, 0x0A,
> treated specially?

If a charset isn't compatible with ASCII insofar as CR and LF are concerned it
cannot be used with the text top-level type. (RFC 2046 section 4.1.1.) So this
is a vacuous concern.

As for ">" and SP, the usual way this is handled is for the specification
to work with those characters regardless of how they are represented.
(You won't find many charsets that muck with space, but there are some
that are modal and depending on mode may use ">" for some other purpose.)

I supposed you could define a charset without a ">" character, or even
one without space, but as a practical concern this seems a bit farfetched.

> does the answer depend on the format in which
> the message is stored?  (e.g. if the message is stored in a file
> on a system whose native charset is ASCII compatible, line endings
> in the storage format might still be a combination of CF and/or LF,
> but they will have no significance for the canonical form of the
> text at all, since that will be UTF-16, EBCDIC, whatever.)

This was decided eons ago. We define things in terms of the canonical form of
the data, and in the canonical form of MIME text line breaks are CRLF and CRLF
is always represented the same way. If a different representation is used on
some system (EBCDIC, counted records, whatever) it is for that system ot figure
out how to adapt the specification accordingly. Trying to account for all the
vagarities of local storage is a sure path to madness.

> Here's a stab at defining this more succinctly and precisely.
> (or perhaps, it's an indication of how much I misunderstood the
> draft...)

> If the format= parameter is set to "fixed" or the parameter is unspecified,
> text/plain is to be interpreted per RFC 2046.

> If the format= parameter is set to "flowed", text/plain is to be interpreted
> per RFC 2046, with the following exceptions:

> 1. The sequence SP CR LF from the canonical form of the body part is to
> be treated as follows:

> a. if the delsp= parameter is set to "yes", the sequence SP CR LF is to
> be ignored when displaying, printing, or otherwise presenting the body part.

> b. if the delsp= parameter is set to "no", or the delsp= parameter is
> unspecified, the sequence SP CR LF is to be treated as SP when displaying,
> printing, or otherwise presenting the body part.

> c. regardless of the value of the delsp= parameter, if the format=
> parameter has a value of "flowed" the sequence SP CR LF is not treated
> as a "line break".  (this changes the rule in section 4.1.1 of RFC 2046
> which states that CR and LF are forbidden outside of line breaks)

> 2. The sequence CR LF from the canonical form of the body part, when
> immediately preceded by SP, is interpreted as a line break.

> 3. A "line" consists of zero or more characters which start immediately at
> the beginning of the canonical form of the body part, or immediately following
> a line break.

> 4. "Lines" in body parts for which format=flowed MAY be "wrapped" as necessary
> to fit the width of the display or output medium, by ceasing the output of
> characters along one horizontal row of the output device or medium, and
> continuing the output of subsequent characters along the next horizontal row
> of the output device or medium.  Such wrapping SHOULD, when possible, be done
> when a character sequence that is to be interpreted as SP is detected (either
> a SP character, or if delsp=no or is unspecified, the sequence SP CR LF)

> 5. One or more ">" characters at the start of a line are taken as an indicator
> that the text on that line are a quotation.  The greater number of ">"
> characters, the greater the "depth" of the quotation.

> 6. User agents MAY display or present quotations using leading ">" characters
> or in any other manner which is suitable for the output device or medium.  If
> ">" characters are used to indicate quotations for display or presentation,
> the number of ">" characters displayed SHOULD equal the number of ">"
> characters at the beginning of the line in the canonical form, if the display
> reasonably permits this.  If some other means is used to indicate quotations
> in the display or output medium, different levels of quotations SHOULD be
> displayed or presented differently, so they can be distinguished by the
> recipient.

> 7. Since the ">" notation applies to the entire "line" (as defined in #3 above),
> when a quotation line is "wrapped", the entire line SHOULD be presented as if
> it were a single quotation (and all at the same level of depth), even if the
> line is "wrapped" for display or presentation purposes.

> 8. The vertical spacing between output display rows SHOULD be the same between
> rows of characters within a "wrapped" line as between separate lines.

> 9. In all of the rules in this section, the characters CR LF SP and ">"
> have code values as defined by the charset parameter, even if those values
> do not correspond to those in ASCII.

I guess I have no objection to this reformulation, but OTOH it really isn't
necessary to restate that we're dealing with the canonical form of the data.

				Ned

Date: Thu, 6 Nov 2003 08:03:38 -0500
From: Keith Moore <[email-masked]>
To: IETF RFC-822 list <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Sender: [email-masked]

> Your description misses one subtlety: while one or more flowed lines and
> a following fixed line constitute a unit that invites re-wrapping, a
> fixed line not preceeded by a flowed line does not invite re-wrapping.

I would rather treat a "flowed line" as a single entity than as 
multiple entities.  But you're right in that a "fixed line" and "flowed line"
should be presented differently, to the extent possible.  

I also left out space stuffing.

> Also, I think "paragraph" is a more intuitive term for the concept
> you're calling "line". 

I avoided using the term "paragraph" because that has semantics other
than for presentation.

Personally, I don't think the grammar helps much.  Most readers will 
pay less attention to the grammar than the text.  It's more important
to clean up the text than the grammar.  But if the text were cleaned
up, cleaning up the grammar would also be useful.

e.g.

text-body = *( fixed-line / flowed-line ) 
             ( fixed-line / flowed-line / fragment )

fixed-line = [quote] *text-char CRLF

flowed-line = [stuffing] [quote] 1*( *text-char SP CRLF ) 
              *text-char text-char-no-sp CRLF

fragment = [quote] *text-char

...

I don't like using the term "soft" for the SP CR LF sequence in a flowed
line because it's too easily confused with Q-P soft line breaks, 
even though they have nothing to do with one another.

Also, I'm concerned that people will implement format=flowed differently
for objects depending on the content-transfer-encoding, when the 
content-transfer-encoding should be irrelevant.

> There is an additional rule that is impossible to express in the
> grammar: a flowed line must have the same quote depth as the next line.
> A flowed line that breaks this rule (has a quote depth different from
> the next line) is to be intepreted as if it were a fixed line.

ah yes, another thing that I missed.

Keith

Date: Thu, 6 Nov 2003 08:07:25 +0000
From: "Adam M. Costello" <[email-masked]>
To: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Reply-To: IETF RFC-822 list <[email-masked]>
Sender: [email-masked]

Keith Moore <[email-masked]> wrote:

> if the format= parameter has a value of "flowed" the sequence SP CR LF
> is not treated as a "line break".
>
> 3. A "line" consists of zero or more characters which start
> immediately at the beginning of the canonical form of the body part,
> or immediately following a line break.
>
> 4. "Lines" in body parts for which format=flowed MAY be "wrapped" as
> necessary to fit the width of the display or output medium,

Your description misses one subtlety: while one or more flowed lines and
a following fixed line constitute a unit that invites re-wrapping, a
fixed line not preceeded by a flowed line does not invite re-wrapping.
Also, I think "paragraph" is a more intuitive term for the concept
you're calling "line".  It's useful to let "line" continue to mean what
it always has meant: a sequence of characters terminated by CR LF.

You raise a very good point about the charset.  The grammar in section 7
allows only ASCII characters, even though the delsp parameter is being
introduced for the express purpose of better supporting non-ASCII
scripts.  I think that can easily be fixed by redoing the non-sp
production:

non-sp = <any character except NUL, CR, LF, SP>

There are some other issues with the grammar.  First, I don't see the
reason for distinguishing between unquoted lines and quoted lines.
Every line has a quote depth, which might happen to be zero, but lines
with zero quote depth are not treated specially.  The grammar currently
says:

flowed-line   = flow-qt / flow-unqt
flow-qt       = quote [stuffing] *text-char 1*SP CRLF
flow-unqt     = [stuffing] *text-char 1*SP CRLF
quote         = 1*">"

Why not simply:

flowed-line   = quote [stuffing] *text-char 1*SP CRLF
quote         = *">"

In any case, the grammar for flowed-line is ambiguous: the spaces before
the last space could match either 1*SP or *text-char.  It's only the
last space that's special; the others are arbitrary text that ought to
match *text-char.  So the production should be:

flowed-line   = quote [stuffing] *text-char SP CRLF

(Similarly, section 5.2 makes the rule sound more complex than it really
is when it says "If the line ends in one or more spaces, the line is
flowed."  The rule is really just "If the line ends in a space, the line
is flowed.")

Even after that last adjustment, the production is still ambiguous.  A
space at the beginning of a line could match [stuffing] or *text-char.
A line that does not begining with the space could still match stuffing,
because stuffing is defined as [SP].  The decoder needs to be able to
distinguish between stuffed lines and unstuffed lines, because it's
supposed to display them differently.  Also, an initial greater-than
sign could match quote or *text-char.  The decoder needs to determine
the quote depth unambiguously.  This will do the trick:

flowed-line   = quote (stuffing stuffed / unstuffed) SP CRLF
stuffed       = *text-char
unstuffed     = non-sp-quote *text-char
quote         = *">"
stuffing      = SP
non-sp-quote  = <any character except NUL, CR, LF, SP, ">">

I think that is unambiguous.  The productions for fixed-line have
similar issues.

The grammar gives a name to the space at the beginning of a line because
it is in some sense not part of the regular text.  Now that the delsp
parameter is introduced, there should be a name for the space at the end
of a line, because it's not part of the regular text when delsp=yes.
And there should be a name for the regular text itself.

Here's a stab at an unambiguous grammar with a name for every noteworthy
syntactic unit:

flowed-body      = * ( paragraph / fixed-line )
paragraph        = 1*flowed-line fixed-line
flowed-line      = quote (stuffing stuffed-flowed / unstuffed-flowed) soft CRLF
fixed-line       = quote (stuffing stuffed-fixed / unstuffed-fixed) CRLF
stuffed-flowed   = *text-char
unstuffed-flowed = non-sp-quote *text-char
stuffed-fixed    = [*text-char non-sp]
unstuffed-fixed  = non-sp-quote [*text-char non-sp]
quote            = *">"
stuffing         = SP
soft             = SP
non-sp-quote     = <any character except NUL, CR, LF, SP, ">">
non-sp           = non-sp-quote / ">"
text-char        = non-sp / SP

There is an additional rule that is impossible to express in the
grammar: a flowed line must have the same quote depth as the next line.
A flowed line that breaks this rule (has a quote depth different from
the next line) is to be intepreted as if it were a fixed line.

AMC

Date: Wed, 5 Nov 2003 23:54:28 -0500
From: Keith Moore <[email-masked]>
To: Randall Gellens <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Sender: [email-masked]

ack!  found a few typos in my attempt at rewording :

> c. regardless of the value of the delsp= parameter, if the format=
> parameter has a value of "flowed" the sequence SP CR LF is not treated 
> as a "line break".  (this changes the rule in section 4.1.1 of RFC 2046
> which states that CR and LF are forbidden outside of line breaks)
> 
> 2. The sequence CR LF from the canonical form of the body part, when 
> NOT immediately preceded by SP, is interpreted as a line break. 
  ^^^  add this here

(I hate it when I do that...)

> 5. One or more ">" characters at the start of a line are taken as an indicator
> that the text on that line are a quotation. 
                             ^^ should be "is"

> 7. Since the ">" notation applies to the entire "line" (as defined in #3 above),
> when a quotation line is "wrapped", the entire line SHOULD be presented as if 
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  delete this part (it's redundant)
> it were a single quotation (and all at the same level of depth), even if the
> line is "wrapped" for display or presentation purposes.

Keith

Date: Wed, 5 Nov 2003 22:49:12 -0500
From: Keith Moore <[email-masked]>
To: Randall Gellens <[email-masked]>
Cc: [email-masked], [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
Sender: [email-masked]

I have to say that I find both RFC 2646 and this draft fairly opaque and
somewhat ambiguous.

In particular, what is a "line"?  is it:

- zero or more characters from the canonical form of a body part,
  beginning either at the start of the body part
  or immediately following a CRLF, and ending with a CRLF?

- zero or more characters from the encoded form of a body part,
  beginning either at the start of the body part
  or immediately following a CRLF, and ending with a CRLF
  whether or not it is preceded by a SP?

- zero or more characters from the canonical form of a body part,
  beginning either at the start of the body part
  or immediately following a CRLF, and ending with a CRLF
  that isn't preceded by a SP?

In a charset that isn't compatible with ASCII, are the characters
">", SP, CR, LF treated specially using the values of those characters
from that charset, or are the octet values 0x3E, 0x20, 0x0D, 0x0A, 
treated specially?  does the answer depend on the format in which
the message is stored?  (e.g. if the message is stored in a file
on a system whose native charset is ASCII compatible, line endings
in the storage format might still be a combination of CF and/or LF,
but they will have no significance for the canonical form of the 
text at all, since that will be UTF-16, EBCDIC, whatever.)

Here's a stab at defining this more succinctly and precisely.
(or perhaps, it's an indication of how much I misunderstood the
draft...)

If the format= parameter is set to "fixed" or the parameter is unspecified,
text/plain is to be interpreted per RFC 2046.

If the format= parameter is set to "flowed", text/plain is to be interpreted
per RFC 2046, with the following exceptions:

1. The sequence SP CR LF from the canonical form of the body part is to 
be treated as follows:

a. if the delsp= parameter is set to "yes", the sequence SP CR LF is to
be ignored when displaying, printing, or otherwise presenting the body part.

b. if the delsp= parameter is set to "no", or the delsp= parameter is 
unspecified, the sequence SP CR LF is to be treated as SP when displaying,
printing, or otherwise presenting the body part.

c. regardless of the value of the delsp= parameter, if the format=
parameter has a value of "flowed" the sequence SP CR LF is not treated 
as a "line break".  (this changes the rule in section 4.1.1 of RFC 2046
which states that CR and LF are forbidden outside of line breaks)

2. The sequence CR LF from the canonical form of the body part, when 
immediately preceded by SP, is interpreted as a line break. 

3. A "line" consists of zero or more characters which start immediately at
the beginning of the canonical form of the body part, or immediately following
a line break.

4. "Lines" in body parts for which format=flowed MAY be "wrapped" as necessary
to fit the width of the display or output medium, by ceasing the output of
characters along one horizontal row of the output device or medium, and
continuing the output of subsequent characters along the next horizontal row
of the output device or medium.  Such wrapping SHOULD, when possible, be done
when a character sequence that is to be interpreted as SP is detected (either
a SP character, or if delsp=no or is unspecified, the sequence SP CR LF)

5. One or more ">" characters at the start of a line are taken as an indicator
that the text on that line are a quotation.  The greater number of ">"
characters, the greater the "depth" of the quotation.  

6. User agents MAY display or present quotations using leading ">" characters
or in any other manner which is suitable for the output device or medium.  If
">" characters are used to indicate quotations for display or presentation,
the number of ">" characters displayed SHOULD equal the number of ">"
characters at the beginning of the line in the canonical form, if the display
reasonably permits this.  If some other means is used to indicate quotations
in the display or output medium, different levels of quotations SHOULD be
displayed or presented differently, so they can be distinguished by the
recipient.

7. Since the ">" notation applies to the entire "line" (as defined in #3 above),
when a quotation line is "wrapped", the entire line SHOULD be presented as if 
it were a single quotation (and all at the same level of depth), even if the
line is "wrapped" for display or presentation purposes.

8. The vertical spacing between output display rows SHOULD be the same between
rows of characters within a "wrapped" line as between separate lines.

9. In all of the rules in this section, the characters CR LF SP and ">"
have code values as defined by the charset parameter, even if those values
do not correspond to those in ASCII.

To: Randall Gellens <[email-masked]>
Cc: [email-masked]
Subject: Re: Format=Flowed/RFC 2646 Bis (-02)
From: Simon Josefsson <[email-masked]>
Date: Mon, 03 Nov 2003 19:59:20 +0100
Sender: [email-masked]

Randall Gellens <[email-masked]> writes:

> An updated draft which is intended to replace RFC 2646 has been sent
> in; because of the crush of last-minute submissions, there may be a
> delay before the announcement appears.  During this time it is
> available at
> <ftp://ftp.pensive.org/Public/Randy/draft-gellens-format-bis-02.txt>.
>
> This version reflects comments received during IETF Last Call.
>
> The changes from the -01 version are a discussion of OpenPGP's
> stripping of trailing whitespace before calculating the signature,
> mention of Unicode Annex 14, and some text clean-ups/clarifications .

Thanks for adding the OpenPGP discussion.  Given the subtleness of the
issue, I believe the document should not only mention it, but also
give normative advice on how the combination of OpenPGP and
format=flowed is to be implemented.  Otherwise implementors will
ignore the problem, as they do today.

When I look at how to properly implement both OpenPGP and
format=flowed, I can't come to any other conclusion than that security
is more important than maintaining soft paragraph breaks.  That means
a client should not flow OpenPGP signed data, when it present the
outcome as something that OpenPGP guarantee is what the sender sent.
If the client would flow a message, someone in transit may modify the
rendering of a message without being detected by OpenPGP.

Repeating the text from RFC 2440, saying that PGP/MIME aka RFC 3156
SHOULD be used in messaging applications, may be sufficient.  Perhaps
promote it to MUST within the scope of flowed messages.

Thanks,
Simon