Subject: [Ietf-calsify] Issue 1: new proposed text

From: lear at cisco.com (Eliot Lear)
Date: Tue Sep 19 02:22:39 2006

<Chair Hat Off>

Having tried to understand the value of any change to the text, I have
the following observations:

    * The concern is only for those who are attempting to read the
      calendar information in a text based mailer, and impacts
      approximately one glyph.  This is a very limited constituency and
      of very limited impact.
    * The text ALREADY says that folding occur between any two
      characters.  That means you should continue to emit legal UTF-8 if
      you are emitting UTF-8 at all.
    * However, my own (admittedly braindamaged) code implements folding
      at a very low level, as folding is one of the last operations to
      take place.
    * I suspect I'm not the only one who has written such brain damaged
      code.
    * After *unfolding* any potential damage should be undone.
    * The amount of code required to fold or unfold text is pretty darn
      minimal.
    * Much of it is already written.

Hence, relaxing of folding is perhaps not advisable.

I  have two recommendations for textual changes to RFC 2445:

   1. Delete the second sentence in the 2nd paragraph after the example
      in 4.1 as redundant (it says the exact same thing in the first
      paragraph);
   2. Add a sentence as follows: "It is possible for very simple
      implementations to have improperly folded lines in the middle of
      UTF-8 characters.  In such cases, implementations SHOULD unfold
      lines in such a way to properly restore the character.

Eliot

---

From: lists at block-online.eu (Oliver Block)
Date: Wed Sep 20 08:23:12 2006

>       calendar information in a text based mailer,

Actually it is for everybody who wants to produce well-formed UTF-8.

>       lines in such a way to properly restore the character.

Would you want to add this to the old text or to one of the new proposals?

Best Regards,

Oliver

-- 
Leben ist mehr als ...
http://www.nak-nrw.de/p_6_4.html

---

From: lear at cisco.com (Eliot Lear)
Date: Wed Sep 20 08:30:22 2006

Again, chair hat off.

> Actually it is for everybody who wants to produce well-formed UTF-8.

But nobody will care unless they're looking at the raw message.  Yes,
it's malformed, but then it's not really meant to be represented.  What
really should be happening is that implementations should use MIME
multipart/alternative and not put a text/calendar message in front of a
user.

> Would you want to add this to the old text or to one of the new proposals?
Old text.

Ok, WG chair hat on:

Don't forget that if you have an alternative please present it.  If I
missed it I apologize in advance.

Eliot

---

From: reinhold at kainhofer.com (Reinhold Kainhofer)
Date: Wed Sep 20 08:48:57 2006

Am Mittwoch, 20. September 2006 17:30 schrieb Eliot Lear:
> > ?
> >
> >> ? ? * The concern is only for those who are attempting to read the
> >> ? ? ? calendar information in a text based mailer,
> >> ? ?
> >
> > ?
> But nobody will care unless they're looking at the raw message. ?Yes,
> it's malformed, but then it's not really meant to be represented. ?What

Actually, who knows what mail processors (e.g. recoding done by several mail 
servers!) will do if they encounter broken UTF-8 bytes, because line folding 
happened in between the bytes of a multi-byte character? If they try to "fix" 
the UTF-8, they actually mess up the calendar attachment.

Cheers,
Reinhold

- -- 
- ------------------------------------------------------------------
Reinhold Kainhofer, Vienna University of Technology, Austria
email: [email-masked], http://reinhold.kainhofer.com/
 * Financial and Actuarial Mathematics, TU Wien, http://www.fam.tuwien.ac.at/
 * K Desktop Environment, http://www.kde.org, KOrganizer maintainer
 * Chorvereinigung "Jung-Wien", http://www.jung-wien.at/

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Wed Sep 20 08:49:39 2006

> multipart/alternative and not put a text/calendar message in front of a

I support Eliot's position.

My position can best be summarized as being "do it right or don't do it at 
all."

That is, we should choose between either "only fold at a point that 
represents a complete glyph" or "it doesn't matter where folding takes 
place, since the text will always be unfolded prior to processing."

It is pointless to talk about "well-formed UTF-8" and say that is "good 
enough."  Yes, you can break a UTF-8 string such that the two substrings 
are both independently interpretable by a UTF-8 decoder.  The problem is 
that if the break is before a combining character codepoint, the string 
after the break is NOT a valid UTF-8 string since it begins with a 
combining character.  Put another way, it's valid UTF-8 encoding of an 
invalid string.

I believe that a half-solution is the worst possible choice, since it 
creates the illusion of viability for a primitive tool that uses the raw 
message without doing proper folding.

If we punt, and say that folding can occur anywhere, then primitive tools 
are broken; they will attempt to deal with dangling UTF-8 strings which 
end in the middle of a codepoint sequence, and with strings that start in 
the middle of a codepoint sequence.

If we go all the way, and say that folding can only occur at a glyph 
point, then primitive tools will be fine.  That is a lot of work to 
undertake just to make it easier for some lazy programmer to foist a 
primitive tool as being acceptable.

In my opinion, we should choose between supporting primitive tools, or 
declaring primitive tools to be totally broken.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: cyrus at daboo.name (Cyrus Daboo)
Date: Wed Sep 20 08:50:01 2006

Hi Eliot,

> multipart/alternative and not put a text/calendar message in front of a

The issue is that it is not out of the realm of possibility that a spam 
filter or some other 'integrity' checker in email will reject an iMIP 
message because it contains invalid utf-8 (utf-8 multi-octet sequence 
broken across two lines). Another example might be reading the utf-8 byte 
stream into a unicode string object BEFORE doing the iCal processing on the 
unicode string data - that will likely fail if there is invalid utf-8. 
That's been my primary concern rather than whether it will be viewable in a 
text editor.

-- 
Cyrus Daboo

---

From: lear at cisco.com (Eliot Lear)
Date: Wed Sep 20 08:58:08 2006

Hi Cyrus,
> The issue is that it is not out of the realm of possibility that a
> spam filter or some other 'integrity' checker in email will reject an
> iMIP message because it contains invalid utf-8 (utf-8 multi-octet
> sequence broken across two lines). Another example might be reading
> the utf-8 byte stream into a unicode string object BEFORE doing the
> iCal processing on the unicode string data - that will likely fail if
> there is invalid utf-8. That's been my primary concern rather than
> whether it will be viewable in a text editor.

Sorry- to be clear RFC 2445 already requires that you not split lines in
the middle of characters.  This implies valid UTF-8 when used, IMHO.

Eliot

---

From: McQuilWP at pobox.com (Bill McQuillan)
Date: Wed Sep 20 09:14:24 2006

> That's been my primary concern rather than whether it will be viewable in a

A further issue is that the definition of "folding" in RFC 2822 explicitly
states (sect.2.2.3) that the unfolded text includes the SP following the
CRLF. Thus, a UTF-8 sequence of any sort would be interrupted by a SP
character unless an idiosyncratic unfolding operation is employed.

-- 
Bill McQuillan <[email-masked]>

---

From: McQuilWP at pobox.com (Bill McQuillan)
Date: Wed Sep 20 09:57:26 2006

> character unless an idiosyncratic unfolding operation is employed.

Sorry, I'm afraid I had a temporary brain glitch. The folding technique is NOT modeled on RFC2822 headers and the issue of the SP does not exist.

Never mind!

---

From: TimHare at comcast.net (Tim Hare)
Date: Wed Sep 20 20:24:24 2006

I don't claim to be a UTF-anything expert, but the current text does require
white space (SP or TAB) after the CRLF for folding.  However - what do we
break if we eliminate the folding altogether as in my proposed text below?

Old Text:
----------------------------------------------------------------------------
4.1 Content Lines

   The iCalendar object is organized into individual lines of text,
   called content lines. Content lines are delimited by a line break,
   which is a CRLF sequence (US-ASCII decimal 13, followed by US-ASCII
   decimal 10).

   Lines of text SHOULD NOT be longer than 75 octets, excluding the line
   break. Long content lines SHOULD be split into a multiple line
   representations using a line "folding" technique. That is, a long
   line can be split between any two characters by inserting a CRLF
   immediately followed by a single linear white space character (i.e.,
   SPACE, US-ASCII decimal 32 or HTAB, US-ASCII decimal 9). Any sequence
   of CRLF followed immediately by a single linear white space character
   is ignored (i.e., removed) when processing the content type.

   For example the line:

     DESCRIPTION:This is a long description that exists on a long line.

   Can be represented as:

     DESCRIPTION:This is a lo
      ng description
       that exists on a long line.

   The process of moving from this folded multiple line representation
   to its single line representation is called "unfolding". Unfolding is
   accomplished by removing the CRLF character and the linear white
   space character that immediately follows.

   When parsing a content line, folded lines MUST first be unfolded
   according to the unfolding procedure described above. When generating
   a content line, lines longer than 75 octets SHOULD be folded
   according to the folding procedure described above.
  The content information associated with an iCalendar object is
   formatted using a syntax similar to that defined by [RFC 2425]. That
   is, the content information consists of CRLF-separated content lines.

   The following notation defines the lines of content in an iCalendar
   object:

     contentline        = name *(";" param ) ":" value CRLF
        ; This ABNF is just a general definition for an initial parsing
        ; of the content line into its property name, parameter list,
        ; and value string

     ; When parsing a content line, folded lines MUST first
        ; be unfolded according to the unfolding procedure
        ; described above. When generating a content line, lines
        ; longer than 75 octets SHOULD be folded according to
        ; the folding procedure described above.
----------------------------------------------------------------------------
---------------------------------

Proposed new text (feel free to make modifications as necessary for Unicode
purposes)
----------------------------------------------------------------------------
---------------------------------
4.1 Content Lines

   The iCalendar object is organized into individual lines of text,
   called content lines. Content lines are delimited by a line break,
   which is a CRLF sequence (US-ASCII decimal 13, followed by US-ASCII
   decimal 10).

   Lines of text may be of arbitrary length. It is up to the CUA
implementation
   to present the text to the user in a manner consistent with the current
user
   interface in use, which may include wrapping the text within margins or
other
   forms of presentation changes. The originator of calendar object text
should
   not expect its display to be limited to any arbitrary length.

   Older implementations of [RFC 2445] may be encountered which have
performed a 
   "folding" acton on longer lines of text by inserting a CRLF sequence
followed by 
   a single linear white space character (SPACE, US-ASCII decimal 32; or
HTAB 
   US-ASCII decimal 09) at logical folding points. This is usually done in
e-mail
   implementations to improve compatibility with some mailing programs.
Implementations
   MAY choose to "unfold" these lines by replacing the CRLF-whitespace
sequence with 
   a single space.

   For example the line:

     DESCRIPTION:This is a lo
      ng description
       that exists on a long line.

   May be presented to the interface as is, or may be unfolded to read:

     DESCRIPTION: This is a long descripton that exists on a long line.

  The content information associated with an iCalendar object is
   formatted using a syntax similar to that defined by [RFC 2425]. That
   is, the content information consists of CRLF-separated content lines.

   The following notation defines the lines of content in an iCalendar
   object:

     contentline        = name *(";" param ) ":" value CRLF
        ; This ABNF is just a general definition for an initial parsing
        ; of the content line into its property name, parameter list,
        ; and value string

     ; When parsing a content line, folded lines MAY first
        ; be unfolded according to the unfolding procedure
        ; described above. 

----------------------------------------------------------------------------
---------------------------------

Since TEXT property values MUST represent intentional line breaks via "\n"
or "\N" (4.3.11) anyone desiring to
Format the content of a long line may include those formatting sequences.
The most likely use case for folding is the
DESCRIPTION: property,  "\n" is completely appropriate to use there and is
already seen in implementations. I also
believe that most current implementations "wrap" text within some UI limit
(margins, window size, etcetera) so the
human factor of folding is already handled in the UI anyway.

Tim Hare
Interested Bystander, Non-Inc.

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Bill McQuillan
Sent: Wednesday, September 20, 2006 12:57 PM
To: Calsify List
Subject: Re: [Ietf-calsify] Issue 1: new proposed text

On Wed, 2006-09-20, I wrote:

> A further issue is that the definition of "folding" in RFC 2822 
> explicitly states (sect.2.2.3) that the unfolded text includes the SP 
> following the CRLF. Thus, a UTF-8 sequence of any sort would be 
> interrupted by a SP character unless an idiosyncratic unfolding operation
is employed.

Sorry, I'm afraid I had a temporary brain glitch. The folding technique is
NOT modeled on RFC2822 headers and the issue of the SP does not exist.

Never mind!

_______________________________________________
Ietf-calsify mailing list
[email-masked]
http://lists.osafoundation.org/mailman/listinfo/ietf-calsify

---

From: bernard.desruisseaux at oracle.com (Bernard Desruisseaux)
Date: Thu Sep 21 11:21:43 2006

Eliot Lear wrote:
 > [...]

+1

>       lines in such a way to properly restore the character.

I would make that a "Note", I would replace "UTF-8 characters" by
"character" and I would lowercase the SHOULD.  That is:

 >    Note: It is possible for very simple implementations to have
 >    improperly folded lines in the middle of a character. In such
 >    cases, implementations should unfold lines in such a way to
 >    properly restore the character.

Thanks,
Bernard

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Thu Sep 21 12:09:35 2006

+1

> >    properly restore the character.

I would also delete "In such cases". A receiving client has no other choice.

---

From: lear at cisco.com (Eliot Lear)
Date: Thu Sep 21 12:42:13 2006

Are there any objections to the below?  Oliver?

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Thu Sep 21 14:23:59 2006

On Thu, 21 Sep 2006, Alexey Melnikov wrote:
>> "character" and I would lowercase the SHOULD.

Suggest "a UTF-8 multi-octet sequence" to make it perfectly clear what we 
are talking about.  Other charsets have multi-octet sequences too, but 
we've deprecated the use of these.

Suggest using "need to" instead of "should", to indicate that this is 
a corrolary technical requirement as opposed to something specifically 
required by the protocol.

>> That is:

Another good point.

In conclusion, suggest:

     Note: It is possible for very simple implementations to generate
     improperly folded lines in the middle of a UTF-8 multi-octet
     sequence. For this reason, implementations need to unfold lines
     in such a way to properly restore the unbroken sequence.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: alexey.melnikov at isode.com (Alexey Melnikov)
Date: Fri Sep 22 02:58:44 2006

Mark Crispin wrote:

> Suggest "a UTF-8 multi-octet sequence" to make it perfectly clear what 
> we are talking about.  Other charsets have multi-octet sequences too, 
> but we've deprecated the use of these.
>

I like that, but "unbroken" reads strangely hear (the sender might have 
broken it).
Maybe "restore the original sequence"?

---

From: lists at block-online.eu (Oliver Block)
Date: Fri Sep 22 04:23:38 2006

> Are there any objections to the below?  Oliver?

None, that I'd like to express in this stage of decision making.

Best Regards,

Oliver

---

From: lear at cisco.com (Eliot Lear)
Date: Fri Sep 22 04:45:35 2006

Alexey Melnikov wrote:
>> Suggest "a UTF-8 multi-octet sequence" to make it perfectly clear
>> what we are talking about.  Other charsets have multi-octet sequences
>> too, but we've deprecated the use of these.
>>
>> a corrolary technical requirement as opposed to something
>> specifically required by the protocol.
>>
>>>> That is:
> I like that, but "unbroken" reads strangely hear (the sender might
> have broken it).
That sounds like consensus wording to me if ever I heard some.  Can we
move along?

Eliot

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Fri Sep 22 09:24:59 2006

> That sounds like consensus wording to me if ever I heard some.  Can we

Sold!

-- Mark --

http://panda.com/mrc