Subject: [Issue 1] Re: [Ietf-calsify] draft-ietf-calsify-rfc2445bis-01.txt / UTF-8

From: bernard.desruisseaux at oracle.com (Bernard Desruisseaux)
Date: Fri Sep 1 08:41:36 2006

Folks, I would like to close this issue with a new proposal.

I propose to add the following statement in section 4.1 Content Lines:

  Folded lines MUST be well-formed according to the character set
  specified for the iCalendar object. For instance, a UTF-8 multi-octet
  sequence MUST not be split across lines when UTF-8 is used as
  the character set. This may require folding a line at less than
  the recommended 75 octet limit.

Old text:

 > Lines of text SHOULD NOT be longer than 75 octets, excluding the line
 > break. Long content lines SHOULD be split into a multiple line
 > representations using a line "folding" technique. That is, a long
 > line can be split between any two characters by inserting a CRLF
 > immediately followed by a single linear white space character (i.e.,
 > SPACE, US-ASCII decimal 32 or HTAB, US-ASCII decimal 9). Any sequence
 > of CRLF followed immediately by a single linear white space character
 > is ignored (i.e., removed) when processing the content type.

New text:

Lines of text SHOULD NOT be longer than 75 octets, excluding the line
break. Long content lines SHOULD be split into a multiple line
representations using a line "folding" technique. That is, a long
line can be split between any two characters by inserting a CRLF
immediately followed by a single linear white space character (i.e.,
SPACE, US-ASCII decimal 32 or HTAB, US-ASCII decimal 9). Folded lines
MUST be well-formed according to the character set  specified for
the iCalendar object. For instance, a UTF-8 multi-octet sequence
MUST not be split across lines when UTF-8 is used as the character set.
This may require folding a line at less than the recommended 75 octet
limit. Any sequence of CRLF followed immediately by a single linear
white space character is ignored (i.e., removed) when processing the
content type.

Cheers,
Bernard

Cyrus Daboo wrote:
> Hi Bernard,
> 
> --On August 23, 2006 11:06:13 AM -0400 Bernard Desruisseaux 
> <[email-masked]> wrote:
> 
>> The question is "Do we actually care?".
>>
>> I would say that as long as the iCalendar stream is a valid
>> UTF-8 document we are ok.
> 
> Agreed - my main concern is ensuring that the stream conforms to UTF-8 
> abnf - i.e. the characters are all valid. I don't really care whether 
> the 'raw' unfolded text displays well or not in a text editor. After 
> all, even in the absence of combining characters, the folding can occur 
> in the middle of a word and the result may be nonsensical or even worse 
> appear to be two valid words giving a different meaning to what was meant.
>

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Fri Sep 1 09:32:37 2006

> the recommended 75 octet limit.

"UTF-8 multi-octet sequence" is neither a technically accurate nor 
unambiguous term.  It can describe:

1) An entire UTF-8 string, comprised of multiple codepoints (Unicode 
characters) and multiple glyphs (printed characters).

2) A sequence of octets, interpreted as UTF-8, that comprise a single 
glyph, but may comprise multiple codepoints.

3) A sequence of octets, interpreted as UTF-8, that comprise a single 
codepoint.

Interpretation (1) is absurd in the context, but it is not precluded by 
"UTF-8 multi-octet sequence".  No matter how absurd a misinterpretation 
may be, someone someday will use it unless it is precluded.

The difference between interpretation (2) and (3) depends upon which 
criteria is used for "well-formed".  Splitting between a character and 
combining characters is well-formed UTF-8 encoding, but is NOT well-formed 
Unicode.

Many people fail to understand that UTF-8 is NOT a character set; it is an 
encoding.  The character set is Unicode.

Therefore, a better wording is:
----------------------------------------------------------------------
Folded lines MUST be well-formed according to the character set specified 
for the iCalendar object.  Folding MUST NOT occur between octets that 
define a single codepoint in a character set that uses multiple octets to 
define a codepoint (e.g., non-ASCII codepoints in UTF-8, East Asian 
character sets).  [Furthermore, folding MUST NOT occur prior to a
codepoints that defines a combining character.]  This may require
folding a line at less than the recommended 75 octet limit.
----------------------------------------------------------------------

The text in [] is used for interpretation (2) but not for interpretation 
(3).

If this working group chooses to use interpretation (3), I question the 
need for this rule at all.  Under interpretation (3), folded text CAN NOT 
interpreted correctly without first applying unfolding.  The two folded 
strings would be well formed UTF-8 encoding, but it is NOT well formed 
Unicode.

On the other hand, there is a possibility that it may not be possible to 
fit a well-formed Unicode string, encoded in UTF-8, in 75 octets.  This 
would happen if a single glyph consumes more than 75 octets when encoded 
in UTF-8.  As far as I recall, the record is a glyph that consumes 18 
codepoints (all in the BMP, so a maximum of 3 octets/codepoint), and that 
glyph has a precomposed codepoint in the BMP.  So this currently is not a 
problem.

If the sense of the group is that it wants to use interpretion (3) to 
avoid this (as-yet theoretical problem), and that it is "good enough" to 
require just well-formed UTF-8 encoding without requiring well-formed 
Unicode, then I request that there be some text to state this rationale.

For example:
----------------------------------------------------------------------
Note: folding is permitted between a character and a combining character. 
The result of such folding in UTF-8 is well-formed UTF-8; however it is 
not well-formed Unicode.  The string prior to the folding will end with a 
printed glyph that is not correct (it misses some combining character) and 
the string subsequent to the folding will start with a combining character 
which is meaningless in Unicode.

Therefore, these strings MUST BE unfolded before displaying them to the 
user.  These foldings are for transport in the iCalendar object, and to 
keep all strings in the iCalendar object as well-formed transport 
encodings (e.g., well-formed UTF-8).  There is NO guarantee that a folded 
string is well-formed in the character set; and thus these foldings MUST 
NOT be used as display foldings. 
----------------------------------------------------------------------

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: lists at block-online.eu (Oliver Block)
Date: Sat Sep 2 03:30:01 2006

> Therefore, a better wording is:

This is quite elaborated. But it should IMHO talk of "charset" (RFC2978, Sect. 
1.3) not "character set". Unfortunately just replacing "characters set" with 
"charset" doesn't work as it is used in two different contexts.

Best Regards,

Oliver

-- 
Leben ist mehr als schneller - weiter - h?her
http://www.nak-nrw.de/p_6_4.html

---

From: sroberts at uniserve.com (Sam Roberts)
Date: Mon Sep 11 09:39:35 2006

> > Therefore, a better wording is:

Is the "[Furthermore,...combining character.]" text supposed to be part
of the the wording?

It appears that there are combining characters defined in many blocks of
unicode characters.

  http://www.unicode.org/faq/char_combmark.html

How do I determine that a code point defines a combining character? Is
there a list of such code-points?

Cheers,
Sam

---

From: mrc at CAC.Washington.EDU (Mark Crispin)
Date: Mon Sep 11 09:46:44 2006

> Is the "[Furthermore,...combining character.]" text supposed to be part

This is a matter under discussion.  The text is in a [] pair since its 
inclusion depends upon the decision of the WG.

> there a list of such code-points?

You don't determine combining character by codepoint value.  You determine 
it by the codepoint's Canonical Combining Class value, as defined in the 
UnicodeData.txt file.  Characters with a Canonical Combining Class value 
of 0 are not combining characters.

-- Mark --

http://panda.com/mrc

---

From: sroberts at uniserve.com (Sam Roberts)
Date: Mon Sep 11 12:20:13 2006

> inclusion depends upon the decision of the WG.

Ok.

> of 0 are not combining characters.

Thanks.

An application would probably not include the 1038607 byte
UnicodeData.txt file [1], it could just have knowledge of the current
(418, by my count) combining characters.

Would that application then become non-conformant with the MUST NOT
under discussion when a new combining character is added to Unicode and
the application isn't updated? Or would we pin iCalendar to a particular
Unicode version? (or whatever they call the currently defined codepoints
and combining classes, if they don't call it a "unicode version")

I think its worth clarifying why folding in the middle of multi-byte
sequences for either a "character" or a "grapheme" may not be
desireable. As I understand it, one reason that has been mentioned is if
the data is folded it can no longer be called valid "utf-8" (for
example). This doesn't matter to an iCalendar application which will
unfold before processing, but would matter to, for example, a mail
application which didn't know how to process the calendar, and was
trying to display it to the user as plain text.

I think the bigger issue is that in order to unfold, you need to find
the line endings. The line endings themselves are encoded as a
"character", possibly multi-byte (in UTF-16, for example). So, a certain
amount of well-formedness is necessary in order to unfold a calendar.
If a long line in UCS-16 was broken in the middle of a UTF-16 codepoint,
and a LINE-FEED, SPACE was added in UTF-16, it would probably be
difficult or impossible to parse.

The necessity of being well-formed enough to unfold the data doesn't
seem to mean that we need to forbid folding between combining
characters... or does it? Would a sequence of

  CAPITAL-A, LINE-FEED, SPACE, COMBINING TILDE, ...

make the SPACE appear to be a "space with a tilde over it" (!) with
some programming languages, thus making it not possible to find the
LINE-FEED, SPACE sequence which needs to be removed during unfolding?

In the languages I work in, finding "codepoints" is no trouble, and
wrapping between codepoints that are part of combining sequences is no
trouble, and the wrapped calendar might have a few "odd" looking bits
when sequences like my example above are viewed in a unicode supporting
terminal (see attached), but thats not a problem for unfolding.

Are there iCalendar implementations out there (real or suspected) that
will fail to interoperate when lines are wrapped between combining
codepoints?

If the goal here is to tighten up the spec to improve interop, and its
not an interop issue, I'd say document that:

- line wrapping SHOULD (MUST?) occur before XX bytes,
- wrapping MUST NOT break a codepoint (if the character set being used
  represents a code-point as more than one byte), and
- that wrapping MAY occur between combining codepoints, and
  move on.

Also, unless I misunderstand, pathological unicode (or maybe not
pathological, just a complex musical score, they use combining
characters a lot) could contain a grapheme consisting of an abitrarily
long sequence of codepoints, so a MUST NOT break between combining
characters rule might be mutually exclusive with a rule that lines MUST
NOT be over a particular byte-length.

Cheers,
Sam

[1]   http://www.unicode.org/Public/UNIDATA/UnicodeData.txt

---

From: sroberts at uniserve.com (Sam Roberts)
Date: Mon Sep 11 13:04:42 2006

>   CAPITAL-A, LINE-FEED, SPACE, COMBINING TILDE, ...

Oops, attached utf-8, but don't know if attachments make it through to
the list.

% xxd __.out
0000000: 410a 20cc 802e 2e2e 0a                   A. ......

xxd -r will get it back.

-------------- next part --------------
A
 ?...

---

From: lists at block-online.eu (Oliver Block)
Date: Mon Sep 11 17:16:40 2006

> inclusion depends upon the decision of the WG.

I would forget that. You can convert from charsets (e.g. ISO-8859) to UTF-8 
with acceptable effort. But adding the part in [] would multiply the effort. 

iCalendar requires valid UTF-8 and with the two texts that were proposed, it  
makes that clear enough. 

I would even prefer Bernards Version with 'charset' instead of 'character set' 
because it leaves some terms out, which make it better readable in my view. 
Marks Version is ahead of the times:) so it should be kept for the next 
iCal-WG. Or in other words: If you want to introduce new terms to this RFC, 
you should do with more than five sentences.

That's what I think.

Best Regards,

Oliver

-- 
Leben ist mehr ...

---

From: MRC at CAC.Washington.EDU (Mark Crispin)
Date: Mon Sep 11 17:19:08 2006

> with acceptable effort. But adding the part in [] would multiply the effort.

Huh?  Any conversion from ISO 8859 to UTF-8 would not have any combining 
characters, since all ISO 8859 characters have precomposed equivalents in 
UTF-8.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.