Subject: RFC: UTF-8 iCalendar bug solution

From: Mark Swanson <[email-masked]>
Date: Sun, 23 Feb 2003 20:18:26 -0500

Hello,

I don't claim to know didly about character sets, but that hasn't stopped
me before so...

I would like to state some problems with rfc2445 wrt character sets
and propose a solution.

The problems:

Problem 1: The current iCalendar specification states that UTF-8 is the 
default character set, yet does not allow all UTF-8 characters.

Problem 2: Since UTF-8 characters are often 1, 2, and 3 bytes we need to
    make alterations to the RFC. Put another way, if the RFC is written
    to support a default
    character set that can have 3 byte characters then defining things
    like NON-US-ASCII to be "%x80-F8" is broken and needs fixing.

#1 in more detail:

(Quote from the spec as a reminder that UTF-8 is the default charset)
<quote>
4.1.4 Character Set
There is not a property parameter to declare the character set used
in a property value. The default character set for an iCalendar
object is UTF-8 as defined in [RFC 2279].
</quote>

Note that the definition of NON-US-ASCII is %x80-F8. This excludes the
following characters:

371   249   F9     ù     LATIN SMALL LETTER U WITH GRAVE
372   250   FA     ú     LATIN SMALL LETTER U WITH ACUTE
373   251   FB     û     LATIN SMALL LETTER U WITH CIRCUMFLEX
374   252   FC     ü     LATIN SMALL LETTER U WITH DIAERESIS
375   253   FD     ý     LATIN SMALL LETTER Y WITH ACUTE
376   254   FE     þ     LATIN SMALL LETTER THORN
377   255   FF     ÿ     LATIN SMALL LETTER Y WITH DIAERESIS

F.E. this (VEVENT holiday) summary is not allowed in iCalendar (%xFA):
SUMMARY:Proclamação da República

A quick fix might be to change NON-US-ASCII to "%x80-FF" but since this only
deals with a single character and does not handle every 2+ byte UTF-8
character this isn't good enough. A unified solution is presented at
the bottom.

#2 in more detail

Note the following definition (and comment):

<quote>
TSAFE-CHAR = %x20-21 / %x23-2B / %x2D-39 / %x3C-5B
%x5D-7E / NON-US-ASCII
; Any character except CTLs not needed by the current
; character set, DQUOTE, ";", ":", "\", ","

Note: Certain other character sets may require modification of the
above definitions, but this is beyond the scope of this document.
</quote>

This is just another example of a range of characters that ignores the range 
of the UTF-8 character set.

Notice the "Note:". Also note that the problem and solution presented here do 
not fall into the "other character sets" category (they fall into the default 
"supported character sets" category).

The solution to problems 1 and 2:

Change all 8-bit byte values to 16-bit byte values like this:

NON-US-ASCII = \u0080 - \uffff

If an RFC is going to state that UTF-8 is supported (which they must IMHO)
then at the very least it MUST define all characters and character ranges
in a minimum of 16-bit values.

It should be noted that several commercial packages already seem to be 
redefining character ranges to 16-bit values and are creating public 
iCalendars that are actually using 2 and 3 byte UTF-8 codes for summary text. 

IMHO this is the right thing to do. Various commercial packages have tweaked 
the spec for other reasons and I agree for tweaking it for UTF-8. I would 
like to propose the RFC be modified accordingly.

Comments please.

-- 
Schedule your world with ScheduleWorld.com
http://www.ScheduleWorld.com/
Java Web Start:
http://www.ScheduleWorld.com/sw/ScheduleWorld.jnlp

---

From: Doug Royer <[email-masked]>
Date: Mon, 24 Feb 2003 09:48:29 -0700

In all uses of the 'NON-US-ASCII' ABNF definition in 2445 it allows for
zero or MORE octets. It looks to me as if it does allow for 1 or more
octet (multibyte) characters.

Assuming you are correct and it should be %x80-FF, that looks to me
as if it would solve the problem by changing that alone. Have you verified
that all of the valid UTF-8 octet sequences that can include 0xF9-FF are
printable (not control) characters? If not we would have to also exclude
those characters sequences.

Changing from UTF-8 to UTF-16 (which I think is your proposal) would
break existing iMIP implementations.

If you are talking about the standard multibyte to wide charset
translation functions you are talking about changing the charset
definition from ASCII valid to ASCII not valid. Although numerically
'A' == 0x41 == 0x0041, in the UTF-8 charset the sequence  0x0041 != 'A'.
When using standard string functions existing implementations would break.
If an existing implementation internally already used standard multibyte to
wide character conversion functions, then when it converted to wide
characters (not knowing it already was as you propose) and it got a a sequence
of 0x0041, it would stop the conversion of UTF-8 at the 0x00 octet
and never see the 'A' character. And for implementations that use
multibyte (non-wide char) aware string functions, they would expect
UTF-8 and again would stop at the 0x00 octet never seeing the 'A'.

Mark Swanson wrote:

> ...

All instances of the ABNF that use NON-US-ASCII ABNF definition
use *<Name> which allows for ZERO or more octets.

> NON-US-ASCII = \u0080 - \uffff

Which is UTF-16 - correct?

-- 

  Doug Royer                     |   http://INET-Consulting.com
  -------------------------------|-----------------------------
  [email-masked]                 | Office: (208)612-INET
  http://Royer.com/People/Doug   |    Fax: (866)594-8574
                                 |   Cell: (208)520-4044

                 We Do Standards - You Need Standards

---

From: Mark Swanson <[email-masked]>
Date: Mon, 24 Feb 2003 12:40:15 -0500

> octet (multibyte) characters.

I've searched all occurences of octet in 2445 and haven't found any references 
to multibyte characters. Since NON-US-ASCII is defined as %x80-xF8 how is it 
possible to even implicitly understand this means multibyte - especially when 
I see some utf-8 chinese characters I'm parsing have 0x05 as one of their 
bytes?

> those characters sequences.

There are no control characters in that range.
UTF-8 is a 1:1 mapping of latin1 wrt 0x80-0xFF. You can verify this here:
http://www.unicode.org/Public/MAPPINGS/

However, it appears that simply allowing 0x80-0xFF will not do because while 
UTF-8 is an 8-bit character set for all of ASCII and latin1, as soon as you 
use Chinese it switches to 2 or more bytes which may contain characters below 
0x80. 

I realize that statement may set off warning bells. But I can safely say that 
it all works in theory and practice as at least 2 commercial iCalendar 
implementations are doing it this way and it works perfectly. The 
implementation details are roughly:

1. Assume the iCalendar text is in UTF-8. (1:1 mapping to ASCII and latin1)
2. Only pass UTF-8 strings to your parser.
3. Define NON-US-ASCII  as \u0080 - \uffff

I made these changes to the ScheduleWorld parser and it parses and displays 
any language defined in Unicode - nothing else in 2445 had to change.

> break existing iMIP implementations.

No, I'm not changing from UTF-8 to UTF-16. It may have appeared that way 
because of the syntax:
NON-US-ASCII = \u0080 - \uffff

However, it is not what you think. I didn't mention it explicitly but I was 
refering to 2 separate things:

1. UTF-8 (we all know what that is)
2. the actual Unicode character ranges that UTF-8 maps to. F.E. it might take 
3 bytes for UTF-8 to define unicode character \u5859 (I think this is one of 
the actual Chinese characters I was testing)

> characters (not knowing it already was as you propose) and it got a a
> sequence of 0x0041, it would stop the conversion of UTF-8 at the 0x00 octet

Please understand that I am proposing to stick with UTF-8 where 0x41 == 0x41, 
completely 1:1 compatible with ASCII _and_ ISO-8859-1 (latin1). 

No string comparrisons will stop working with ASCII and latin1. String 
comparrisons will _start_ to work with characters above \u00ff.

The key seems to be that UTF-8 can represent any character from 0x0000 - 
0xffff and 2445 MUST take this into consideration.

I have a feeling that I may not be clear. Though it all works for me I am not 
a Unicode expert (yet! :-) and I may be missing something too.

> > A quick fix might be to change NON-US-ASCII to "%x80-FF" but since this
> > only deals with a single character and does not handle every 2+ byte
> > UTF-8 character this isn't good enough. A unified solution is presented
> > at the bottom.
>

---

From: Lawrence Greenfield <[email-masked]>
Date: Mon, 24 Feb 2003 12:58:41 -0500

From: Mark Swanson <[email-masked]>
   Date: Mon, 24 Feb 2003 12:40:15 -0500
[...]
   I've searched all occurences of octet in 2445 and haven't found any
   references to multibyte characters. Since NON-US-ASCII is defined
   as %x80-xF8 how is it possible to even implicitly understand this
   means multibyte - especially when I see some utf-8 chinese
   characters I'm parsing have 0x05 as one of their bytes?

It is not possible for 0x05 to appear in UTF-8 text representing (part
of) a Chinese character.

0x05 always represents the UCS-4 character \u0005.

Please consult RFC 2279.

   > Assuming you are correct and it should be %x80-FF, that looks to
   > me as if it would solve the problem by changing that alone. Have
   > you verified that all of the valid UTF-8 octet sequences that can
   > include 0xF9-FF are printable (not control) characters? If not we
   > would have to also exclude those characters sequences.

   There are no control characters in that range.  UTF-8 is a 1:1
   mapping of latin1 wrt 0x80-0xFF. You can verify this here:
   http://www.unicode.org/Public/MAPPINGS/

This is incorrect.

The first 255 characters of UCS-4 (Unicode) map 1:1 to ISO-8859-1.

UTF-8 is an encoding of UCS-4 in which the ASCII subset of UCS-4 is
mapped to a single byte sequence.

   However, it appears that simply allowing 0x80-0xFF will not do
   because while UTF-8 is an 8-bit character set for all of ASCII and
   latin1, as soon as you use Chinese it switches to 2 or more bytes
   which may contain characters below 0x80.

This is incorrect.

   1. Assume the iCalendar text is in UTF-8. (1:1 mapping to ASCII and
   latin1) 
   2. Only pass UTF-8 strings to your parser.
   3. Define NON-US-ASCII as \u0080 - \uffff

UTF-8 consists of variable length characters. You seem to be talking
about UTF-32 or some multi-byte fixed length character encoding.

Please consult RFC 2279.

[...]
   Please understand that I am proposing to stick with UTF-8 where
   0x41 == 0x41, completely 1:1 compatible with ASCII _and_ ISO-8859-1
   (latin1).

This is not UTF-8.

   No string comparrisons will stop working with ASCII and
   latin1. String comparrisons will _start_ to work with characters
   above \u00ff.

   The key seems to be that UTF-8 can represent any character from
   0x0000 - 0xffff and 2445 MUST take this into consideration.

This is correct; a UTF-8 octet sequence can represent any UCS-4
character.

To me it is easiest to talk about iCalendar objects as sequences of
UCS-4 characters, which must be encoded in UTF-8 octets, with the
shortest legal sequence for any UCS-4 character. Trying to cram the
UTF-8 rules into the iCalendar ABNF is just silly, but most other IETF
protocols do it, so iCalendar might as well.

Larry

---

From: Doug Royer <[email-masked]>
Date: Mon, 24 Feb 2003 11:01:53 -0700

> I see some utf-8 chinese characters I'm parsing have 0x05 as one of their 

It does not use the word multibyte - however it allows for zero
or more sequences of characters.

The only 4 uses of 'NON-US-ASCII' in 2445 are:

	'SAFE-CHAR', 'QSAFE-CHAR', 'TSAFE-CHAR', and 'VALUE-CHAR'

All of them define they they allow exactly 1 octet.
And each of those when are used with '*' (zero or more):

	paramtext          = *SAFE-CHAR

	quoted-string      = DQUOTE *QSAFE-CHAR DQUOTE

	value              = *VALUE-CHAR

	text               = *(TSAFE-CHAR / ":" / DQUOTE / ESCAPED-CHAR)

So in all cases it allows for zero or more octets which
would cover multibyte.

And yes the 2445 ABNF is busted in that it assumes that all octets
fall in that range (0x...)!

> There are no control characters in that range.

Great!

> use Chinese it switches to 2 or more bytes which may contain characters below 

I see your point.

Would you like to propose the correct ABNF to make UTF-8 work in 2445? :-)
I have been reading other WG lists where they are running into the
same or similar UTF-8 issues with respect to the ABNF. I think the real
problem is that the authors did not really understand charsets (Sorry Frank
and all) and just did their best.

I agree that the charset part of the 2445 ABNF is busted. I think we rely
on the ABNF too much and should just default to text that says:

	Its UTF-8 - except you can not use control characters XXX,...
         and that YYY needs to be quoted...

--

---

From: Mark Swanson <[email-masked]>
Date: Mon, 24 Feb 2003 13:48:56 -0500

> 0x05 always represents the UCS-4 character \u0005.

OK - the \u0005 was my mistake. 

> mapped to a single byte sequence.

Got it.

>    which may contain characters below 0x80.

That's interesting. If true I need to start verifying the output of my third 
party parser...

> about UTF-32 or some multi-byte fixed length character encoding.

The above works.
I'm talking about the Unicode character. It would not make sense to deal 
directly with UTF-8 octets.

>    0x41 == 0x41, completely 1:1 compatible with ASCII _and_ ISO-8859-1

Got it.

> protocols do it, so iCalendar might as well.

I understand and agree with your first sentence.

I have no idea what you are refering to in your second sentence :-)

To try and summarize:

1. We need to ensure 0xf9-0xff does not contain any non-printable characters 
with the UTF-8 encoding.

2. UTF-8 capable parsers should parse after decoding UTF-8 characters into 
UCS-4.

3. When dealing with UCS-4 characters we should use:
NON-US-ASCII = \u0080 - \uffff

Correct?

---

From: Doug Royer <[email-masked]>
Date: Mon, 24 Feb 2003 12:03:18 -0700

> I have no idea what you are refering to in your second sentence :-)

I think he is referring to the fact that many RFCs try to define
the ABNF for UTF-8 and fail - like 2445 fails.

> 2. UTF-8 capable parsers should parse after decoding UTF-8 characters into 

You can if you want to, however my implementation runs just fine in the UTF-8
charset. It would be up to implementations.

> NON-US-ASCII = \u0080 - \uffff

I think we need to not use 'UCS-4' in 2445 corrections and just
state that it is UTF-8, don't use control characters and
list the ones that force the data to be quoted.

--

---

From: Mark Swanson <[email-masked]>
Date: Mon, 24 Feb 2003 14:08:45 -0500

On February 24, 2003 01:01 pm, Doug Royer wrote:
> > references to multibyte characters. Since NON-US-ASCII is defined as
> > %x80-xF8 how is it possible to even implicitly understand this means
> > multibyte - especially when I see some utf-8 chinese characters I'm
> > parsing have 0x05 as one of their bytes?
>

Ah, I see. Thank you.

> >>as if it would solve the problem by changing that alone. Have you
> >> verified that all of the valid UTF-8 octet sequences that can include
> >> 0xF9-FF are printable (not control) characters? If not we would have to
> >> also exclude those characters sequences.
> >
> > However, it appears that simply allowing 0x80-0xFF will not do because
> > while UTF-8 is an 8-bit character set for all of ASCII and latin1, as
> > soon as you use Chinese it switches to 2 or more bytes which may contain
> > characters below 0x80.
>
> I see your point.
>

Lawrence has schooled me on a few points I made. Apparently UTF-8 is only a 
mapping to ASCII and not latin1 and UTF-8 characters do not use values below 
0x80. I have verified this with rfc 2279.

Based on your above analysis that states all instances of multibyte characters 
are provided for two questions come to mind:

1. Is it clear enough that multibyte octets are supported in 2445? Perhaps I 
should have caught that, but perhaps we could add something. The answer to #2 
would give us all we need to suggest something.

2. I still haven't found what UTF-8 encodings of 0xf9 - 0xff decode to in 
UCS-4. It would be great if they were not control characters.

Cheers.

---

From: "Chris Olds" <[email-masked]>
Date: Mon, 24 Feb 2003 11:13:34 -0800

>> parsing have 0x05 as one of their bytes?

Any utf-8 sequence that includes a byte 0x05 must be encoding a ^E (ENQ)
character.  In UTF-8, 7-bit ASCII is mapped 1:1, and nothing else uses bytes
with the high bit clear.

>> There are no control characters in that range.

Unless you're encoding a character at a code point >= 0x200000, there are no
characters that use values in that range, period.

>> which may contain characters below 0x80.

This is wrong.  UTF-8 is 8-bits for 7-bit ASCII only.  Values in the upper
half of Latin-1 are two bytes (as are all code points < 0x800); if the
character is in an int named 'ch', the first byte will be (0xC0 | ch>>6), and
the second byte will be (0x80 | (ch & 0x3F)).  All non-ascii characters in
UTF-8 are encoded by a lead byte with two or more high bits set, followed by a
0 bit, followed by as many high bits of the code point (absctract character)
as will fit. This lead byte is followed by one or more bytes containing 0x80
and the next six bits of the code point; the number of follow bytes is the
same as the number of leading 1 bits in the lead byte.  Chinese characters
will take 3 or 4 octets to encode, all of which will have at least the high
bit set.

> same or similar UTF-8 issues with respect to the ABNF. I think the 
> real problem is that the authors did not really understand charsets (Sorry 
> Frank and all) and just did their best.
> 
> I agree that the charset part of the 2445 ABNF is busted. I think we 
> rely on the ABNF too much and should just default to text that says:
> 

From a conformance point of view, the only things I see lacking are that it
should be explicit what one should do with invalid UTF-8 sequences and
properly encoded C1 control characters (0x80-0x9F in Latin1); the ABNF allows
all of the currently defined unicode characters, so it is currently permissive
enough.  

For an overview of UTF-8 and some of the issues involved, see [1] (I have
nothing to do with that, but Google liked it, and I agree).

     /cco

[1] http://czyborra.com/utf/

--
GPG Key Fingerprint: B375 A4E7 752B DB8C 4359  852E C3CF BF64 379A E9B2
Debian Project (http://www.debian.org)

---

From: "Chris Olds" <[email-masked]>
Date: Mon, 24 Feb 2003 11:34:14 -0800

On Mon, 24 Feb 2003 13:48:56 -0500, Mark Swanson wrote
> 
> To try and summarize:
> 
> 1. We need to ensure 0xf9-0xff does not contain any non-printable 
> characters with the UTF-8 encoding.

Correct UTF-8 encoding of Unicode 3.1 will never use these octet values.

> 2. UTF-8 capable parsers should parse after decoding UTF-8 
> characters into UCS-4.

That seems more complicated than needed.  Since all of the characters that
matter to the ABNF productions are ASCII, all I think a parser MUST do is
return sequences containing octets matching NON-US-ASCII as-is; unless the
rest of the system is UCS-4 aware, decoding into UCS-4 is a bug.  Being 8-bit
clean should be enough for a lot of systems.  There's no reason to form UCS-4
characters just to parse.

> NON-US-ASCII = \u0080 - \uffff

NON-US-ASCII should be left as-is, possibly with a note saying that a parser
(or CS, or CUA, whatever) MAY or SHOULD validate that NON-US-ASCII sequences
are valid UTF-8 sequences.  In any case, UCS-4 defines code point blocks using
values through 0x1FFFF.

    /cco

---

From: Mark Swanson <[email-masked]>
Date: Mon, 24 Feb 2003 14:45:32 -0500

> characters with the UTF-8 encoding.

I wrote some code to display the above characters in latin-1 and utf-8.
The results are:

utf-8:����
0:�
1:�
2:�
3:�
latin1:ùúûüýþÿ
0:ù
1:ú
2:û
3:ü
4:ý
5:þ
6:ÿ

You may not see the UTF-8 characters
The latin-1 characters are correct as verified by my iso_8859_1 man page:

As for the UTF-8 characters, I see 4 symbols on my Linux terminal but I 
suppose that's no way to be 100% sure. Also, if I scroll the terminal these 
specific UTF-8 characters can change slightly. OK OK, I'm using a 2-year old 
Linux konsole I should probably upgrade to KDE3.1...

If anyone knows the real way to check what these characters are mapped to 
please emal.

BTW, if you want to see a snapshot of me editing a Chinese 2445 document in a 
_vi_ Linux text konsole click below...

http://www.scheduleworld.com/china1.png

Notice some of the strange rendering errors I get there too? Likely because of 
2-year old software...too busy to upgrade...

---

From: Mark Swanson <[email-masked]>
Date: Mon, 24 Feb 2003 15:00:33 -0500

> properly encoded C1 control characters (0x80-0x9F in Latin1); the ABNF
> allows all of the currently defined unicode characters, so it is currently
> permissive enough.

http://czyborra.com/utf/

Thanks for the link. In it I see 0xf9-0xff are legal UTF-8 values.

So this is the single modification to 2445 that I think should be made:

NON-US-ASCII = %x80-xF8

changed to

NON-US-ASCII = %x80-xff

Thoughts?

---

From: Mark Swanson <[email-masked]>
Date: Mon, 24 Feb 2003 15:05:34 -0500

> Correct UTF-8 encoding of Unicode 3.1 will never use these octet values.

Hmm. The link you sent me showed legal values for 0xf9-0xff in a table about 
85% of the way through.

	latin1	utf-1	utf-8	utf-7,5	utf-7	JAVA	HTML
	ù        ù      Ã¹      £ù      +APk-   \u00f9  &#249;
        ú        ú      Ãº      £ú      +APo-   \u00fa  &#250;
        û        û      Ã»      £û      +APs-   \u00fb  &#251;
        ü        ü      Ã¼      £ü      +APw-   \u00fc  &#252;
        ý        ý      Ã½      £ý      +AP0-   \u00fd  &#253;
        þ        þ      Ã¾      £þ      +AP4-   \u00fe  &#254;
        ÿ        ÿ      Ã¿      £ÿ      +AP8-   \u00ff  &#255;

---

From: "Chris Olds" <[email-masked]>
Date: Mon, 24 Feb 2003 12:37:22 -0800

On Mon, 24 Feb 2003 15:05:34 -0500, Mark Swanson wrote
> Hmm. The link you sent me showed legal values for 0xf9-0xff in a 
> table about 85% of the way through.
> 

The UCS values 0xF9-0xFF can be encoded in UTF-8, but the octet values
0xF9-0xFF cannot appear in a valid UTF-8 sequence representing Unicode 3.1 (or
prior) code points.  The example you quote correctly shows that the UCS values
\u00F9-\u00FF are encoded in UTF-8 by Ã (an octet with value 0xC3) followed by
another octet in the range 0xB9-0xBF.

It's not the case that the characters ùúûüýþÿ cannot be used in the messages
defined by rfc2445 (et al), but it is the case that the octet values 0xF9-0xDD
cannot appear in those messages.  This is the difference between UCS-4 (a
character set) and UTF-8 (a transformation format).

    /cco

---

From: Mark Swanson <[email-masked]>
Date: Mon, 24 Feb 2003 16:11:22 -0500

Thanks Chris (and to everyone else for responding).

I haven't found documentation that 0xf9-0xff is not allowed, but I have found 
in the link Chris provided and in RFC 2779:

- -  The octet values FE and FF never appear.

Is there a specific reason the other 5 bytes do not appear?

I see the start octet that states the following bytes are to encode:
0400 0000-7FFF FFFF (UCS-4) 
starts with UTF-8 (1111110x) (0xFC).

If we do not allow 0xFC we will not allow a very large mapping of UCS-4 
characters from 0400 0000-7FFF FFFF.

- --

---

From: "Chris Olds" <[email-masked]>
Date: Mon, 24 Feb 2003 13:47:12 -0800

On Mon, 24 Feb 2003 16:11:22 -0500, Mark Swanson wrote
> Thanks Chris (and to everyone else for responding).

Glad I can help (I'm new here, but I learned a lot about UCS/UTF working on
XML WGs).

> I haven't found documentation that 0xf9-0xff is not allowed, but I 
> have found in the link Chris provided and in RFC 2779:
> 
> If we do not allow 0xFC we will not allow a very large mapping of 
> UCS-4 characters from 0400 0000-7FFF FFFF.

This is true, but not (currently) that big a deal.  Most importantly, Unicode
3.1 (the current version) doesn't define any characters from 0x20000 up (the
jargon is that only the BMP (Basic Multilingual Plane) and the 1st Astral
Plane (0x10000-0x1FFFF) are defined).  For these planes, a lead byte of
11110xxx (0xF0-0xF7) is (more than) enough (in fact, it will encode 21 bits of
data, covering the BMP and the first 31 astral planes).  Since a lead byte of
0xF8 (111110 00) allows 24 bits worth of characters to be encoded, I cannot
imagine a need for greater range anytime soon.

    /cco

---

From: Mark Swanson <[email-masked]>
Date: Mon, 24 Feb 2003 17:14:31 -0500

On February 24, 2003 04:47 pm, Chris Olds wrote:
> This is true, but not (currently) that big a deal.  Most importantly,
> Unicode 3.1 (the current version) doesn't define any characters from
> 0x20000 up (the jargon is that only the BMP (Basic Multilingual Plane) and
> the 1st Astral Plane (0x10000-0x1FFFF) are defined).  For these planes, a
> lead byte of 11110xxx (0xF0-0xF7) is (more than) enough (in fact, it will
> encode 21 bits of data, covering the BMP and the first 31 astral planes). 
> Since a lead byte of 0xF8 (111110 00) allows 24 bits worth of characters to
> be encoded, I cannot imagine a need for greater range anytime soon.

It seems we have a minor issue and I'd like to know how to proceed.
We could change the spec and forget about this forever, or leave it as is and 
forget about it without harm for some period of time. What is the process for 
voting on something like this?

How would we take an issue (even a minor one like this) and its solution and 
see it through to making it part of the RFC?

Thanks.

---

From: Doug Royer <[email-masked]>
Date: Mon, 24 Feb 2003 16:07:02 -0700

> see it through to making it part of the RFC?

I have heard that there are errata pages some place one the IETF web
site. I just did a quick check and could not find them.

Short of that Pat or Bob (our chairs) could post them on the calsch.org
web site :-)

--

---

From: Andrea Campi <[email-masked]>
Date: Tue, 25 Feb 2003 11:25:47 +0100

> voting on something like this?

Picking one semi-random message to reply to:

Even if it's a minor issue, and no character should use that
range, I'd say we should fix RFC2445 to correctly include 0x80-0xff.
That would be more future-proof: with the current text, a conformant
implementation is free to reject valid, albeit very rare, UTF-8
codes. With the proposed change, an implementation will be free to
(mostly) not even care about what's in a value.

Bye,
	Andrea

-- 
Andrea Campi                              mailto:[email-masked]
I.NET S.p.A. - BT Ignite                  http://www.inet.it
Technical Dept. - R&D			  phone: +39 02 32863 ext 1
v. Darwin, 85 - I-20019			  fax: +39 02 32863 ext 7705
Settimo Milanese (MI), Italy

---

From: Mark Swanson <[email-masked]>
Date: Tue, 25 Feb 2003 09:39:49 -0500

> (mostly) not even care about what's in a value.

I agree, 0x80-0xff should be used even though 0xfe and 0xff are not valid 
UTF-8. 

In the perfect world, there would be no competing and conflicting character 
encodings. However, latin1 is quite dominant and I'm seeing latin1 encoded 
characters in iCalendars published on many websites (including Mozilla's and 
Apple's). This is illegal, but if some of the most popular iCalendar clients 
allow people to use latin1 (which is the most popular character set encoding) 
then we are going to see a lot of it in the real world. Note that libical (no 
offense Andrea!) allows latin1 characters and is used by a great many 
iCalendar projects.

Here's something that may make this easier to swallow: the printable latin1 
encodings from 0xA0 - 0xFF would likely never be found side by side.
F.E. ææ is an unlikely combination. So is µµ. etc... If you look at the 
iso_8859_1 man page (or any source of the latin1 character set) you'll see 
this.

This means a UTF-8 decoder will know that the non-ascii character in 
Independência is not UTF-8 and can fall back to latin1. 
(The decoder knows because any UTF-8 character above 0x80 MUST be followed by 
another character above 0x80)
If the character is not encoded in latin1 then you are hosed unless you are 
told what character encoding to fall back to.

Java already works this way. It's how I currently handle the current mix of 
latin1 and UTF-8 iCalendars in the real world. The small number of characters 
we are blocking (0xf9-0xff) actually are being used (German holidays come to 
mind).

So that's my case for supporting 0x80-0xFF.

I'm lucky enough to be working in the Java language that takes care of this 
for me.

I'm really curious to know what options are available for C/C++ and projects 
like libical. What UTF-8 decoding options are available and are they good 
enough to fallback to another character encoding (like latin1) when illegal 
UTF-8 is found?

Cheers.

---

From: John Stracke <[email-masked]>
Date: Tue, 25 Feb 2003 10:03:52 -0500

Chris Olds wrote:

>There's no reason to form UCS-4
However, it would be reasonable for the spec to take that approach, even 
if the implementations find a more efficient way.  If we consider ABNF 
to be a statement about characters, rather than bytes, then it is 
independent of the character set in use (provided all specified 
characters are available--don't go waving EBCDIC in my face ;-).  So, 
conceptually, the "%x80-F8" refers to the Unicode characters \u0080-\u00f8.

Under this interpretation, NON-US-ASCII should be redefined to be 
\u00000080-\uffffffff ("\u" now being taken as specifying 4 bytes; I 
know it usually doesn't).  Better would be if we had a complement 
operator in ABNF, but it doesn't look like we do.  If we did, we could 
write something like "NON-US-ASCII = ! US-ASCII" and "US-ASCII = %x00-7F".

-- 
/==============================================================\
|John Stracke      [email-masked]                       |
|Principal Engineer|http://www.centive.com                     |
|Centive           |My opinions are my own.                    |
|==============================================================|
|The sooner you fall behind, the more time you'll have to catch|
|up!                                                           |
\==============================================================/

---

From: Doug Royer <[email-masked]>
Date: Tue, 25 Feb 2003 09:09:24 -0700

> enough to fallback to another character encoding (like latin1) when illegal 

As all iCalendar objects are MIME objects the charset is in the MIME
header. So there is no need to fall back to anything. For CAP the
charset/lang to use is determined by the commands, so nothing to
fall back to.

If you are using the X charset and get a character not in the X charset,
someone sent you trash.

--

---

From: Mark Swanson <[email-masked]>
Date: Tue, 25 Feb 2003 11:19:54 -0500

> independent of the character set in use (provided all specified

I really like the wording of that statement.

> write something like "NON-US-ASCII = ! US-ASCII" and "US-ASCII = %x00-7F".

I like this interpretation because it clearly states how 2445 works with 
unicode.
Doug mentioned previously that the cases were covered for multiple octets 
(except for the minor 0xf9-0xff issue) but I think using sentences like:

"The ABNF is a statement about characters rather than bytes" and
"NON-US-ASCII = \u00000080-\uffffffff"

make it perfectly clear.

---

From: John Stracke <[email-masked]>
Date: Tue, 25 Feb 2003 11:54:48 -0500

Mark Swanson wrote:

>but I think using sentences like:
>
Mind you, this isn't the standard interpretation of ABNF; the ABNF RFC 
takes the other approach.  From RFC-2234, sectino 2.3:

>   Rules resolve into a string of terminal values, sometimes called
>   characters.  In ABNF a character is merely a non-negative integer.
>   In certain contexts a specific mapping (encoding) of values into a
>   character set (such as ASCII) will be specified.
>
But it would be reasonable for a clarified RFC-2445 to specify that the 
non-negative integers in question are Unicode.

-- 
/=================================================================\
|John Stracke      [email-masked]                          |
|Principal Engineer|http://www.centive.com                        |
|Centive           |My opinions are my own.                       |
|=================================================================|
|Two words that inspire the same feeling of dread are synominous. |
\=================================================================/

---

From: Mark Swanson <[email-masked]>
Date: Tue, 25 Feb 2003 11:57:05 -0500

> If you are using the X charset and get a character not in the X charset,

I think the big problem though is that many real-world applications (not SW) 
are allowing a quickly growing userbase to create latin1 iCalendar objects.

Perhaps if this trend continues for any length of time there will be too much 
momentum and supporting latin1 fallback will be unavoidable?

- --

---

From: Doug Royer <[email-masked]>
Date: Tue, 25 Feb 2003 11:56:25 -0700

Mark Swanson wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 

No. Instead you have some choices:

		Set the charset to iso-8859-1 (latin1) in the MIME
		header and send iCalendar objects using latin1 and
		not UTF-8.

		Take the UTF-8 object and charset convert it to
		iso-8859-1. (or the the other direction)

At no point do you have to guess at the charset. If they send
you latin1 and tag it as UTF-8, send them a bug. If they send
you UTF-8 and tag it as latin1, send them a bug.

The iCalendar default of UTF-8 is NOT mandatory for all objects.

The only things that are mandatory about UTF-8 is that the spec says
that all implementations MUST support UTF-8 and that you correctly
tag the MIME header with the correct charset. The reason that UTF-8
was chosen as a must implement is so that there is common framework
to interchange data. If you know that you are talking same vendor
to same vendor, use any default charset you want. But do not expect
everyone to have to guess at the charset you use.

If you want to run your software in 8859-1 that's fine. But for
interoperability you should send and expect UTF-8 MIME objects.

CAP also allows for the charset and lang to be set at runtime. But
it also mandates that all implementations be able to use UTF-8
and for the same reason. Interoperability requires that
everyone must support at least one common charset.

--

---

From: Mark Swanson <[email-masked]>
Date: Tue, 25 Feb 2003 22:59:09 -0500

Thanks for the response Doug. Comments inline...

> you UTF-8 and tag it as latin1, send them a bug.

I agree wrt the MIME transport.

But what about the HTTP transport? I see popular iCalendar clients (with more 
in the pipeline) using a web server to serve/transport iCalendar data. The 
web server will not (and can not reliably) transmit the character set 
encoding. These web servers are currently serving up a combination of 
iCalendar documents encoded in both latin1 and UTF-8. Since the character 
encoding can not be accurately set in the HTTP response you will have to 
guess at the encoding when you come across illegal UTF-8.

What I'm really hoping to accomplish here is to:

1.  bring to light that certain vendors are allowing their growing customer 
base to create mixed UTF-8 and latin1 iCalendar data. 
2. make a suggestion so we can all interoperate with the mixed character 
encodings until all vendors fix their software.

It may be a good business model for Sun to force compliance, but I obviously 
have no choice but to handle all of the subtle breakage of the spec.

> interoperability you should send and expect UTF-8 MIME objects.

I'm sure you mean "If anyone wants to run their software in 8859-1", but I 
just to be clear that SW will not create 8859-1 encoded characters.

---

From: Doug Royer <[email-masked]>
Date: Tue, 25 Feb 2003 21:27:35 -0700

Mark Swanson wrote:
> But what about the HTTP transport?

**ALL** iCalendar objects are MIME objects - they are NOT
iCalendar objects if they do not contain the MIME headers.
If you read the specs you will find that it states that the
mime headers are omitted from the examples to save space and
are are mandatory.

 >  I see popular iCalendar clients (with more
> encoding.

Absolutely not true - HTTP transports MIME data and it DOES include the MIME
headers including charset.

> These web servers are currently serving up a combination of 

Look one layer lower in the wire protocol (snoop the line) and
you will see that they do in fact send MIME header information.

> base to create mixed UTF-8 and latin1 iCalendar data.

Valid - as long as you include the mandatory MIME header.

> encodings until all vendors fix their software.

We did solve that problem :-)
That is why we mandate that iCalendar objects are MIME objects.

> have no choice but to handle all of the subtle breakage of the spec.

That part is true, however I do not know of any major vendor
that is not transporting them as MIME objects. I have not looked
at the apple implementation in detail yet. But if they are not
using MIME headers - then they will have to fix that to be
able to interoperate.

> just to be clear that SW will not create 8859-1 encoded characters.

It can.

--

---

From: Mark Swanson <[email-masked]>
Date: Wed, 26 Feb 2003 09:04:31 -0500

> mime headers are omitted from the examples to save space and

Fascinating. I have never seen a single iCalendar object on the web that 
contained MIME headers.

> > in the pipeline) using a web server to serve/transport iCalendar data.
> > The web server will not (and can not reliably) transmit the character set
> > encoding.
>
> Absolutely not true - HTTP transports MIME data and it DOES include the
> MIME headers including charset.

It absolutely is true. You misunderstood. I said "not reliably", I did not say 
they were not included. It can not reliably set the MIME headers because if 
the web server is serving two files: file1.ics and file2.ics it has no way of 
knowing file1.ics is UTF-8 and file2.ics is 8859-1.

> you will see that they do in fact send MIME header information.

I never said it wasn't there. Again, I said "accurately".

> > 1.  bring to light that certain vendors are allowing their growing
> > customer base to create mixed UTF-8 and latin1 iCalendar data.
>

I'm dumbfounded. :-)

I'm going to think about this some more before I respond.

> > It may be a good business model for Sun to force compliance, but I
> > obviously have no choice but to handle all of the subtle breakage of the
> > spec.
>

Perhaps now that I've mentioned the problem better wrt using a web server you 
would agree that fixing this would be difficult at best. (file1.ics, 
file2.ics in different encodings).

> > I'm sure you mean "If anyone wants to run their software in 8859-1", but
> > I just to be clear that SW will not create 8859-1 encoded characters.
>
> It can.

LOL. Hey, I wrote SW. If you have found a way to do this then it is a bug - 
and please email me how to reproduce it so I can fix it. Do realize that for 
a javax.swing.JTextField to create non UTF-8 Strings should not be possible 
without extending it...

---

From: John Stracke <[email-masked]>
Date: Wed, 26 Feb 2003 09:13:28 -0500

Doug Royer wrote:

>> in the pipeline) using a web server to serve/transport iCalendar 
>> data. The web server will not (and can not reliably) transmit the 
>> character set encoding.
>
>

The problem is that people expect to be able to take a text/calendar and 
save it to disk; when that happens, the MIME headers are lost, and they 
can't be reconstructed.  There's no way to look at the text/calendar and 
know what character set it's in; if there were, we wouldn't need the 
character set in the MIME header.

This is really a general MIME problem: MIME provides for transmitting 
metadata about objects, but that metadata tends to get lost once it 
leaves the MIME transport.  The only solution is to define 
self-describing formats that don't need the MIME headers.

I hate to say it, but this is something that an XML encoding would give 
us for free.

>> 2. make a suggestion so we can all interoperate with the mixed 
>> character encodings until all vendors fix their software.
>

Mandating it doesn't make it happen, unfortunately.

The only mandate that would work would be to declare that UTF-8 isn't 
the default; it's the only option.  People would gripe about that (for 
Europe, UTF-8 costs more than Latin-1; for Asia, it costs more than 
UTF-16), but it'd be an easy requirement to understand and implement.

-- 
/==========================================================\
|John Stracke      [email-masked]                   |
|Principal Engineer|http://www.centive.com                 |
|Centive           |My opinions are my own.                |
|==========================================================|
|A man's concepts should exceed his vocabulary, or what's a|
|metaphor?                                                 |
\==========================================================/

---

From: John Stracke <[email-masked]>
Date: Wed, 26 Feb 2003 09:40:16 -0500

>a javax.swing.JTextField to create non UTF-8 Strings should not be possible 
Java is not the universe.

Or do you mean "SW" to stand for something other than "software"?

---

From: Mark Swanson <[email-masked]>
Date: Wed, 26 Feb 2003 10:28:00 -0500

On February 26, 2003 09:40 am, John Stracke wrote:
> Mark Swanson wrote:
> >LOL. Hey, I wrote SW. If you have found a way to do this then it is a bug
> > - and please email me how to reproduce it so I can fix it. Do realize
> > that for a javax.swing.JTextField to create non UTF-8 Strings should not
> > be possible without extending it...
>

? LOL.

> Or do you mean "SW" to stand for something other than "software"?

LOL. I refer to ScheduleWorld as SW. I appologize for the confusion.

---

From: John Stracke <[email-masked]>
Date: Wed, 26 Feb 2003 10:58:58 -0500

Mark Swanson wrote:

>I refer to ScheduleWorld as SW.
>
There we are: "software" was already in context in your message, so 
that's what Doug and I dereferenced "SW" to.

-- 
/=============================================================\
|John Stracke      [email-masked]                      |
|Principal Engineer|http://www.centive.com                    |
|Centive           |My opinions are my own.                   |
|=============================================================|
|Rope is rope, and string is string, and never the twine shall|
|meet.                                                        |
\=============================================================/

---

From: Doug Royer <[email-masked]>
Date: Wed, 26 Feb 2003 09:26:18 -0700

> I hate to say it, but this is something that an XML encoding would give 

No and for the same reason. If the XML file is in some uncommon
charset then you can not read it to find the encoding value.

	<xml version="1.0" encoding="one-you-do-not-expect">

> Mandating it doesn't make it happen, unfortunately.

It does as far as the standard and interoperablility is concerned
and that is what the WG is about.

If a vendor wishes to save the data in the SHIFT_JISX0213 charset
they can. However sending that raw file to most implementations
will result in breakage.

> UTF-16), but it'd be an easy requirement to understand and implement.

And they did :-)
That is why the MIME header is not optional.

--

---

From: Doug Royer <[email-masked]>
Date: Wed, 26 Feb 2003 09:19:07 -0700

> knowing file1.ics is UTF-8 and file2.ics is 8859-1.

The ics files should have the MIME headers in them - else they
are not iCalednar objects. If you or anyones elses application
saves iCalendar objects without the MIME headers then they
are not portable.

I sent a test arround a year or so ago with over 200 charset
conversion of the same iCalendar object. Each with a correct
MIME header. There is no known way to figure out the charset
of those 200 data files without the MIME headers.

You can save them without the MIME headers. But that is
not the interoperable standard way to do that. Which is
what this WG is about.

> a javax.swing.JTextField to create non UTF-8 Strings should not be possible 

Viewing data and processing do not have to be in UTF-8. Even java
can read UTF-8 and convert it to its native UTF-16.

Java and any other language that I have seen on Unix and Windows
can create 8859-1 encoded characters, which was the
statement I was responting to.

--

---

From: [email-masked]
Date: Wed, 26 Feb 2003 11:36:59 -0500

The MIME header is the IMIP not the ICAL.  If you save the mime stream it 
is not a .ics file;  It is an .eml file.
_____________________
Note: new email address

[email-masked]

Doug Royer <[email-masked]> 
Sent by: [email-masked]
02/26/2003 11:26 AM
Please respond to
"[email-masked]" <[email-masked]>

To
"[email-masked]" <[email-masked]>
cc

Subject
Re: RFC: UTF-8 iCalendar bug solution

And they did :-)

--

---

From: John Stracke <[email-masked]>
Date: Wed, 26 Feb 2003 12:03:21 -0500

Doug Royer wrote:

>> I hate to say it, but this is something that an XML encoding would 
>> give us for free.
>
>

It works if the character set is an ASCII superset, though, as most are. 
 Plus, that declaration is defined to be at the very start of the 
document; if one finds that it's not readable, one can give up right away.

iCalendar doesn't even attempt to be self-describing this way.

Furthermore, RFC-2445 asserts that iCalendar should be useful outside of 
MIME environments.  From the abstract:

   This MIME media type provides a standard content type for capturing
   calendar event, to-do and journal entry information. It also can be
   used to convey free/busy time information. The content type is
   suitable as a MIME message entity that can be transferred over MIME
   based email systems, using HTTP or some other Internet transport. In

   addition, the content type is useful as an object for interactions
   between desktop applications using the operating system clipboard,
   drag/drop or file systems capabilities.

I am not aware of any filesystem or clipboard that attaches MIME headers 
to files.

> and that is what the WG is about.

The WG is about developing a standard that can feasibly be made to 
interoperate.  Pretending that the entire world is MIME-capable doesn't 
do that.

-- 
/============================================\
|John Stracke      [email-masked]     |
|Principal Engineer|http://www.centive.com   |
|Centive           |My opinions are my own.  |
|============================================|
|Never do card tricks for your poker buddies.|
\============================================/

---

From: John Stracke <[email-masked]>
Date: Wed, 26 Feb 2003 12:12:18 -0500

Doug Royer wrote:

> Mark Swanson wrote:
>
>> It absolutely is true. You misunderstood. I said "not reliably", I 
>> did not say they were not included. It can not reliably set the MIME 
>> headers because if the web server is serving two files: file1.ics and 
>> file2.ics it has no way of knowing file1.ics is UTF-8 and file2.ics 
>> is 8859-1.
>
> are not iCalednar objects.

Nonsense.  The RFC does not say this at all.  From 2445, 3.10:

   The file extension of "ics" is to be used to designate a file
   containing (an arbitrary set of) calendaring and scheduling
   information consistent with this MIME content type.

It says *nothing* about including MIME headers.

> not the interoperable standard way to do that.

I have never, ever, seen any MIME implementation (MUA or HTTP client) 
that includes the MIME headers when it saves a MIME body-part.

One of the goals specified in the calsch charter is "A standard content 
type for capturing calendar event and to-do information. The content 
type should be suitable as a MIME message entity that can be transferred 
over MIME based email systems or HTTP World Wide Web."  Implicit in that 
requirement is that the format needs to be transferred from MUA to CUA 
without loss of information; given that MUAs do not preserve MIME 
headers when they save body-parts to disk, it follows that it is 
impossible for this format to rely on the MIME headers being present.

---

From: Dan Winship <[email-masked]>
Date: 26 Feb 2003 12:22:38 -0500

> are not iCalendar objects.

That can't be right. If the MIME headers were part of the
"text/icalendar" data type, then when you sent an iMIP message, you'd
need to include two copies of the headers; one that was part of the
email message, and one that was part of the iCalendar object.

>From the abstract of RFC 2445:

   This memo is formatted as a registration for a MIME media type per
   [RFC 2048]. However, the format in this memo is equally applicable
   for use outside of a MIME message content type.

-- Dan

---

From: Mark Swanson <[email-masked]>
Date: Wed, 26 Feb 2003 12:27:33 -0500

> not the interoperable standard way to do that. Which is

Thanks for filling me in.
Unfortunately my archives don't go back far enough. Would you be able post a 
link or send them to me?

Thanks.

---

From: "Chris Olds" <[email-masked]>
Date: Wed, 26 Feb 2003 09:50:50 -0800

On Wed, 26 Feb 2003 09:26:18 -0700, Doug Royer wrote
> 	<xml version="1.0" encoding="one-you-do-not-expect">

XML documents in character sets that cannot be easily detected (based on the
assumption that the document is a legal XML document and the encoding
declaration) are expected to be very rare.

From the XML spec (Second edition[1], but I think the First edition text is
identical):

    Because the contents of the encoding declaration are restricted to
    characters from the ASCII repertoire (however encoded), a processor can
    reliably read the entire encoding declaration as soon as it has detected
    which family of encodings is in use. Since in practice, all widely used
    character encodings fall into one of the categories above, the XML
    encoding declaration allows reasonably reliable in-band labeling of
    character encodings, even when external sources of information at the
    operating-system or transport-protocol level are unreliable. Character
    encodings such as UTF-7 that make overloaded usage of ASCII-valued bytes
    may fail to be reliably detected.

Character sets that can be reliably detected by reading the encoding
decaration include UTF-16 (BE and LE), EBCDIC, and "any other 7-bit, 8-bit, or
mixed-width encoding which ensures that the characters of ASCII have their
normal positions, width, and values", which includes BIG5, JIS, UTF-8 and 8859
character sets.

     /cco

[1] http://www.w3.org/TR/REC-xml#sec-guessing

---

From: Doug Royer <[email-masked]>
Date: Wed, 26 Feb 2003 10:47:50 -0700

[email-masked] wrote:
> 
> The MIME header is the IMIP not the ICAL.  If you save the mime stream 
> it is not a .ics file;  It is an .eml file.

2445 is a registration of a MIME media type. And '.eml' is nowhere
in 2445, 2446, or 2447 . Perhaps that is a Lotus/IBM thing.
But it is not in iCalendar, iTIP, iMIP or CAP thing.

However in 2445:

3.10 File Extensions

<soapbox>
To transport it your going to need the charset defined (if not UTF-8)
agreed between the endpoints in advance of transfer. Not because it
is a mandate, but because it is a fact necessitated by the other
fact that the world does not use just one charset.

You can limit your sales (not that you are - but just a point)
and interoperability to latin-1 (or whatever) however this WG is
concerned about interoperability across vendors and across the
internet (world).

So I am *not* saying some do not do it that way. I am *not* saying
you can not do it that way. I am saying this WG is about interoperability
and for that you are going to have to transport MIME objects.

Someone talked about loading up .ics files directly into the web tool.
And that will work until the .ics file is not UTF-8 or whatever charset
they can parse. However that work is not within the scope of this WG.
That's one reason CAP exists - to solve these problems. Ignoring the
issue will only limit your sales and increase you bug count over time
as other charsets are sent in valid format to you customers.

</soapbox>

--

---

From: Doug Royer <[email-masked]>
Date: Wed, 26 Feb 2003 10:59:56 -0700

> interoperate.  Pretending that the entire world is MIME-capable doesn't 

True. But the topic is using the files and data transported from
one source to another by some means. So you can transport iCalendar
objects any way you want. But this WG is and has proposed iMIP
and CAP.

If somone has a proposal to tranport iCalendar objects other
than iMIP (MIME) or CAP (which defines the charst and language),
then we can discuss that. However we (WG) can not declare
that non-somehow-tagged files will work across vendors - because they
might not.

I am just trying to eliminate any belief that just because you
have access to a file that the human knows contains iCalendar data,
does not mean that a program can always parse it by guessing at the
charset. And I also want to (and I think we want to) discourage people
from thinking that ftp-ing (or whatever) .ics files will always
work - because it migh not and it does not mean that 2445/2446
are busted.

--

---

From: Doug Royer <[email-masked]>
Date: Wed, 26 Feb 2003 11:02:31 -0700

> Nonsense.  The RFC does not say this at all.  From 2445, 3.10:

and

Dan Winship wrote:
> That can't be right.

Your both correct - I was wrong - about .ics files.

--

---

From: John Stracke <[email-masked]>
Date: Wed, 26 Feb 2003 13:43:09 -0500

> they can parse. However that work is not within the scope of this WG.

Not so.  As I quoted from the charter, there is at least one channel 
(MUA-to-CUA) explicitly in our scope that does not include MIME headers.

-- 
/===============================================================\
|John Stracke      [email-masked]                        |
|Principal Engineer|http://www.centive.com                      |
|Centive           |My opinions are my own.                     |
|===============================================================|
|"I use emacs, which might be thought of as a thermonuclear word|
|processor." -- Neal Stephenson                                 |
\===============================================================/

---

From: Mark Swanson <[email-masked]>
Date: Wed, 26 Feb 2003 14:03:05 -0500

> objects any way you want. But this WG is and has proposed iMIP

To requote from Dan's email (quoted from 2445):
" the format in this memo is equally applicable
   for use outside of a MIME message content type."

Therefore it should be legal to state that any transport is equally applicable 
to iMIP and CAP; this includes ftp and http.

> work - because it migh not and it does not mean that 2445/2446

But the reason it can not work 100% of the time is exactly because 2445 does 
not specify a mandatory property type for the character set.

---

From: Doug Royer <[email-masked]>
Date: Wed, 26 Feb 2003 12:27:04 -0700

> to iMIP and CAP; this includes ftp and http.

Yes. Again it can be done. However you will not have a clue
about the charset.

> not specify a mandatory property type for the character set.

For the same reason I was wrong about the .ics file, you could
not read the charset property without first knowing the charset
of the data itself. It MUST BE external to the iCalendar
BEGIN/END tags.

--

---

From: "Chris Olds" <[email-masked]>
Date: Wed, 26 Feb 2003 11:53:32 -0800

On Wed, 26 Feb 2003 14:03:05 -0500, Mark Swanson wrote
> 
> But the reason it can not work 100% of the time is exactly because 
> 2445 does not specify a mandatory property type for the character set.

I'm (mostly) still in standard-quoting mode, but I should add that I agree
that 2445's punting of the character set question to MIME does cause problems
with stored objects.

    /cco

From RFC2445, p45:

4.3.11 Text

   Value Name: TEXT

   Purpose This value type is used to identify values that contain human
   readable text.

   Formal Definition: The character sets supported by this revision of
   iCalendar are UTF-8 and US ASCII thereof. The applicability to other
   character sets is for future work. The value type is defined by the
   following notation.

---

From: John Stracke <[email-masked]>
Date: Wed, 26 Feb 2003 15:04:44 -0500

Doug Royer wrote:

> of the data itself.

In general, this is true; however, in practice, taking an approach like 
XML's <?xml encoding="" ?> directive will work in the majority of cases. 
 iCalendar's current approach will fail in the majority of cases; the 
only plausible scenarios where it would work would be (a) an integrated 
CUA/MUA, where the CUA has privileged access to the MIME headers, (b) an 
integrated CS/MTA, where the MTA gives the CS access to the incoming 
iMIP (perhaps via IMAP?), and then the CUA accesses the CS via CAP.

(It's also theoretically possible that incoming iTIP might be coming in 
via CAP, but I don't consider that plausible--users don't want to have 
to keep track of CAP addresses in addition to email addresses, and 
admins don't want to open up another communications channel into the 
enterprise.)

-- 
/================================================================\
|John Stracke      [email-masked]                         |
|Principal Engineer|http://www.centive.com                       |
|Centive           |My opinions are my own.                      |
|================================================================|
|"The struggle is always worthwhile, if the end be worthwhile and|
|the means honorable; foreknowledge of defeat is not sufficient  |
|reason to withdraw from the contest." -- Adron e'Kieron, by     |
|Steven Brust                                                    |
\================================================================/

---

From: Mark Swanson <[email-masked]>
Date: Wed, 26 Feb 2003 15:12:50 -0500

On February 26, 2003 02:27 pm, Doug Royer wrote:
> > But the reason it can not work 100% of the time is exactly because 2445
> > does not specify a mandatory property type for the character set.
>

I read the spec differently: I take the formal definition of:
BEGIN:VCALENDAR 
to be ASCII only.

A CHARSET property would be ASCII encoded and the value would be from the IANA 
Charset Registry (which forces ASCII only)

http://www.iana.org/assignments/character-sets

---

From: [email-masked]
Date: Wed, 26 Feb 2003 15:31:21 -0500

Notes sets line after BEGIN: VCALENDAR with non-standard Lotus property 
that contains IANA Charset
_____________________
Note: new email address

Mark Swanson <[email-masked]> 
Sent by: [email-masked]
02/26/2003 03:12 PM

To

Subject

> > But the reason it can not work 100% of the time is exactly because 
2445

A CHARSET property would be ASCII encoded and the value would be from the 
IANA 
Charset Registry (which forces ASCII only)

---

From: Doug Royer <[email-masked]>
Date: Wed, 26 Feb 2003 13:53:28 -0700

> I read the spec differently: I take the formal definition of:

No:

  2.3 International Considerations

    In the rest of this document, descriptions of characters are of the
    form "character name (codepoint)", where "codepoint" is from the US-
    ASCII character set. The "character name" is the authoritative
    description; (codepoint) is a reference to that character in US-ASCII
    or US-ASCII compatible sets (for example the ISO-8859-x family, UTF-
    8, ISO-2022-xx, KOI8-R). If a non-US-ASCII compatible character set
    is used, appropriate code-point from that character set MUST be
    chosen instead. Use of non-US-ASCII-compatible character sets is NOT
    recommended.

So if you don't use a ASCII ~compatable~ character set  -> convert.

--

---

From: Mark Swanson <[email-masked]>
Date: Wed, 26 Feb 2003 22:29:45 -0500

> >>of the data itself. It MUST BE external to the iCalendar

No. Too many valid reasons have been posted why this should not be so.
How about this:

1. The charset property name is "CHARSET"

2. We may specify the value as "MUST be a  (ASCII) value from the IANA Charset 
Registry (http://www.iana.org/assignments/character-sets).

examples of mandating a value type are already in 2445: Section 4.3.3 
(Calendar User Address). IANA registered URI ... IANA registered Charset.

This way I can save the iCalendar object to disk, serve it up with a web 
server, have my MUA persist it to disk, and more.

> So if you don't use a ASCII ~compatable~ character set  -> convert.

What I meant was that the text:
BEGIN:VCALENDAR can only be ASCII. I wasn't talking in the general sense where 
any property value may be in any character set. I mentioned it as proof that 
you can read the CHARSET property because you always know the charset of 
ASCII.

Please help me understand paragraph 2.3:

A codepoint is US-ASCII.
A codepoint is US-ASCII or US-ASCII compatible.
A codepoint is an appropriate codepoint from a NON-US-ASCII character set.

The last one completely befuddles me!

---

From: "Chris Olds" <[email-masked]>
Date: Wed, 26 Feb 2003 21:30:36 -0800

On Wed, 26 Feb 2003 22:29:45 -0500, Mark Swanson wrote
> BEGIN:VCALENDAR can only be ASCII. I wasn't talking in the general 
> sense where any property value may be in any character set. I 
> mentioned it as proof that you can read the CHARSET property because 
> you always know the charset of ASCII.
> 

While paragraph 2.3 suggests rather strongly that UTF-8 or ASCII be used, 2445
doesn't mandate any particular character set; as a result, in theory, you
could conform to 2445 with EBCDIC (where the ordinal value of the letters
and essential punctuation are different from ASCII),  or UTF-16 data (16-bit
codes, so the ASCII values alternate with NUL bytes).  If you were to do such
a foolhardy thing, you should use the codepoint that corresponds to the
canonical ASCII value in your non-US-ASCII-compatible character set (e.g. an
EBCDIC code page).  Absent MIME headers, using anything other than UTF-8 for
2445 data objects seems risky; unfortunately, it sounds like people are doing
this already.

A CHARSET property for VCALENDAR objects seems useful, as does a CHARSET
parameter for properties with text values; neither will be of any use if
they're not present, so it doesn't fix anything, just suggests how to make
things better going forward.

    /cco

---

From: "Mike Higginbottom" <[email-masked]>
Date: Thu, 27 Feb 2003 12:50:08 -0000

Hello,

This may be a dumb-ass question but here goes:

Assuming I want to support some charset which does not have any
representation of US-ASCII characters at all.  I want a CUA to be able to
send me text using this charset in a property value.  Since the only source
of info for the charset I'm dealing with is the MIME headers, then this
charset is presumably also to be used to interpret the other property values
as well.  How do I parse BEGIN:VCALENDAR for instance in the context of this
charset which has no representation of those characters?

The thinking behind this is that in 2445 we have:

<quote>

4.1.4 Character Set

   The "charset" Content-Type parameter can be used in MIME transports
   to specify any other IANA registered character set.

</quote>

from which I infer that the MIME charset parameter should be applied to the
entire body part contents.  Am I missing something here?  From a practical
perspective it seems obvious that 4.1.4 does not apply to BEGIN:VCALENDAR
for example but I'm not at all clear how or where 2445 specifies the cases
where the charset parameter should be applied and where it should not.

Can someone with more experience than I clarify this?

Regards

Mike Higginbottom

P.S. As an aside, I think I missed a step in the discussion of why we want
to extend NON-US-ASCII to 0x80-0xFF.  UTF-8 specifically precludes use of
0xFE and 0xFF so on that basis we should extend only to 0x80-0xFD.  Are we
specifying 0xFF to cover the Latin-1 charset that appears to be in common
usage?

---

From: Mark Swanson <[email-masked]>
Date: Thu, 27 Feb 2003 08:52:08 -0500

On February 27, 2003 12:30 am, Chris Olds wrote:
> While paragraph 2.3 suggests rather strongly that UTF-8 or ASCII be used,
> 2445 doesn't mandate any particular character set; as a result, in theory,
> you could conform to 2445 with EBCDIC (where the ordinal value of the
> letters and essential punctuation are different from ASCII),  or UTF-16
> data (16-bit codes, so the ASCII values alternate with NUL bytes).  If you
> were to do such a foolhardy thing, you should use the codepoint that
> corresponds to the canonical ASCII value in your non-US-ASCII-compatible
> character set (e.g. an EBCDIC code page).  Absent MIME headers, using

That's about what I thought. I think it is very wrong that 2445 allows you to 
do this. This makes it impossible to build a parser that can parse all 
possible valid ICalendar. Someone could create a system that used UTF-16 
characters and conform perfectly with 2445 yet it would be impossible to 
interoperate with likely all other implementations.

> anything other than UTF-8 for 2445 data objects seems risky; unfortunately,
> it sounds like people are doing this already.

Yes.

> things better going forward.

True. Perhaps it should be modelled after PRODID where:

Conformance: The property MUST be specified once in an iCalendar
   object and it MUST be specified immediately after BEGIN:VCALENDAR.

---

From: Mark Swanson <[email-masked]>
Date: Thu, 27 Feb 2003 09:30:35 -0500

On February 27, 2003 07:50 am, Mike Higginbottom wrote:
> charset is presumably also to be used to interpret the other property
> values as well.  How do I parse BEGIN:VCALENDAR for instance in the context
> of this charset which has no representation of those characters?

Yep. Doug also brought up Section 2.3 which allows for BEGIN:VCALENDAR to be 
recoded in the Russian charset which would break all parsers that expect this 
to be in ASCII. Perhaps someone will give an explanation for this but atm I 
think it is a bad bug.

> Can someone with more experience than I clarify this?

Good point. It seems to be the entire document. This also seems to further 
support Section 2.3. 

Having a CHARSET property that states the default character set of the 
iCalendar document, and CHARSET parameters that can state the character set 
of individual property values would be the most flexible.

> specifying 0xFF to cover the Latin-1 charset that appears to be in common

I like it because it covers the Latin-1 charset, which appears to be in common 
use.

I just though of another reason why the definition of NON-US-ASCII is broken: 
if the MIME charset, or a CHARSET property/parameter, or even making use of 
Section 2.3 to use whatever charset/codepoints you like requires 0xf9-0xff 
then you are forced to create illegal iCalendar and will not be able to 
interoperate.

---

From: "Chris Olds" <[email-masked]>
Date: Thu, 27 Feb 2003 08:57:46 -0800

On Thu, 27 Feb 2003 09:30:35 -0500, Mark Swanson wrote
> I like it because it covers the Latin-1 charset, which appears to be 
> in common use.
> 
> I just though of another reason why the definition of NON-US-ASCII 
> is broken: if the MIME charset, or a CHARSET property/parameter, or 
> even making use of Section 2.3 to use whatever charset/codepoints 
> you like requires 0xf9-0xff then you are forced to create illegal 
> iCalendar and will not be able to interoperate.

If you think of the various productions that specify codepoints as abstract,
then they can be thought of as giving examples in the default (UTF-8)
character set, and as not binding on iCalendar objects in other character
sets.  However, I find little or no support in 2445 for such an
interpretation, which means NON-US-ASCII (and the exclusion of the C0 control
set (\0x00-\0x1F) plus DEL) could use some clarifying text.

It is my impression that the authors of 2445 found this as complex and
annoying an issue as we do, and decided to put it off for later.  My evidence
for this is in sec. 4.3.11 (quoted in part below); it appears that iCalendar
2.0 only supports latin-1 where it overlaps with 7-bit ASCII.

    /cco

4.3.11 Text

   Value Name: TEXT

---

From: "Mike Higginbottom" <[email-masked]>
Date: Thu, 27 Feb 2003 19:15:20 -0000

----- Original Message -----
From: "Chris Olds" <[email-masked]>
To: "Mark Swanson" <[email-masked]>; "[email-masked]"
<[email-masked]>
Sent: Thursday, February 27, 2003 4:57 PM
Subject: Re: UTF-8 iCalendar bug solution

> 2.0 only supports latin-1 where it overlaps with 7-bit ASCII.
I can well understand this viewpoint (looking at i18n in general is
something I've been studiously avoiding up until now) but I think we have
come to the stage now, not just in the context of 2445 and calendaring,
where the issues do need to be addressed fully and accurately.  The vast
majority of discussions on the 822 mailing list for example focus on these
very issues and they always seem to generate a lot of heat.  It's a huge
problem for them with the ubiquity of e-mail and the plethora of
non-conforming software.  It's not something that is easily retrofitted to
an existing installed base and the presence of the Latin-1 discussion serves
only to highlight this.  Even with the (relatively) small number of
calendaring apps about today we are already seeing problems emerging.

Since I have no expertise, I'm unfortunately able to offer little in the way
of a route forward but I suspect the fundamental issues are the usual ones:

1) Define what i18n support we want to provide in the ideal world situation.
2) Identify the scope of existing software which would become broken if we
were to implement the 'ideal' solution.
3) Come up with a compromise that comes as close to the ideal as possible
whilst not breaking too much of the installed codebase.
4) Make sure the discussions don't just generate heat without light -
there's always a tendency for the tricky threads to just peter out without
concensus or action.
5) Implement that compromise in the RFC's and drafts. And make sure we don't
duck the issues that have caused so much trouble in other application
domains.

Just my 2c anyway.

Mike Higginbottom

---

From: Doug Royer <[email-masked]>
Date: Thu, 27 Feb 2003 14:18:12 -0700

> charset which has no representation of those characters?

You can charset convert from and to any charset. So in UTF-8
  'BEGIN:VCALENDAR' has the following TEXT and OCTAL values:

UTF-8

     B   E   G   I   N   :   V   C   A   L   E   N   D   A   R
   102 105 107 111 116 072 126 103 101 114 105 116 104 101 122

And the octal values in UTF-16:

   177377 000102 000105 000107 000111 000116 000072 000126
   000103 000101 000114 000105 000116 000104 000101 000122

So in this case there is a 1:1 mapping to values (same number
of printable characters) and a 1:2 mapping to octet values
(each UTF-8 octet in THIS example is 1 octet and it results
in 2 octets in UTF-16) + 2 leading octets (The 2 lead octets
for UTF-16). Not all charsets are this easy, some a multiple
and a varying number of octets in length for each printable
character (multibyte).

So for any blob of data if you want to parse it in a charset
that is native to your program then you charset convert it from
the PROVIDED charset to your charset (See 'iconv' - in
previous e-mail), something like:

		convert(blob1 in X-CHARSET to blob2 in UTF-8)

At some level is the core blob of data. And many XML implementations
can guess the charset of MOST XML data, not even XML or its implementations
can guess all charsets that can be used to encode XML data. I have read
books that tell the reader ways to make that guess - however it only
works some to most of the time. And until the correct charset is guessed
and *tried* even XML in an unknown charset is a blob of data.

The way we solved this in iCalendar is to pass the problem
(the name of the charset) to the transport layer then 100%
of the time we know the correct charset and the only limitation
is that the endpoints must be able to handle the named charset.

--

---

From: "Mike Higginbottom" <[email-masked]>
Date: Thu, 27 Feb 2003 22:19:22 -0000

----- Original Message -----
From: "Doug Royer" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, February 27, 2003 9:18 PM
Subject: Re: Reading NON UTF-8

>      B   E   G   I   N   :   V   C   A   L   E   N   D   A   R
Ahhhhh.  I see! In other words it's the responsibility of the implementation
to produce the right octet stream regardless of local character
representation.  In that case, please ignore my next message which as far as
I can tell hasn't reached the list as yet.

Thanks for the clarification.

Mike Higginbottom