Subject: The charset issue

From: Doug Royer <[email-masked]>
Date: Thu, 27 Feb 2003 10:23:50 -0700

(This message sent in UTF-8 - assuming Netscape did not lie to me)

 >
 > Yep. Doug also brought up Section 2.3 which allows for BEGIN:VCALENDAR to be
 > recoded in the Russian charset which would break all parsers that
 > expect this to be in ASCII. Perhaps someone will give an explanation
 > for this but atm I think it is a bad bug.

Below I am attempting to describe why this is not an issue.

The entire world does not revolve around US-ASCII. Right now someone
can e-mail you Russian, Japanese, or Chinese e-mail and modern
e-mail tools (MUA) can display it - why?

     It is NOT because they send US-ASCII, UTF-8, or iso-8859-1.

It is because the e-mail headers specify the charset and there
have been worked out over the years procedures to charset convert
to something your MUA can display.

iMIP is the only proposed standard way to transport iCalendar
objects - and that is e-mail that the tools already know how to
process various charsets.

I think that CAP is very close to RFC status at this point. And it
also has a way to set the charset and language that will be used.

The charset issue is NOT an issue for iCalendar, its an issue
for implementations. The problem has been solved for iCalendar.

Adding CHARSET to iCalendar will not fix any problem as you
can not read 'C' 'H' 'A' 'R' 'S' 'E' 'T' if you do not already
know the charset. In e-mail (iMIP) that problem is solved.
In CAP that problem is solved.

The only area were that is not solved is when someone wants
to transport iCalendar objects over a protocol (ftp, http, whatever)
where the charset negotiation specifics have not been standardized.
And they have been standardized for http but some here I think want
to transport iCalendar objects of unknown charset over an http connection
without telling the other endpoint the charset of the data itself.

In iCalendar the entire object is 'in one charset'. It is NOT
that the string 'VERSION' is in UTF-8 or US-ASCII and the '2.0' in
another charset. In other words; iCalendar objects are
TEXT objects in some known charset.

A mini charset tutorial:

Given this iCalendar object in UTF-8 from 2445 section 4.4:

      BEGIN:VCALENDAR
      VERSION:2.0
      PRODID:-//hacksw/handcal//NONSGML v1.0//EN
      BEGIN:VEVENT
      DTSTART:19970714T170000Z
      DTEND:19970715T035959Z
      SUMMARY:Bastille Day Party
      END:VEVENT
      END:VCALENDAR

Now the same object in EBCDIC-US (attached) it is binary.

Using Linux (I am sure other os's have similar methods) You
can from the command line charset convert the EBCDIC-US to
UTF-8 so you can parse it with your UTF-8 iCalendar program:

	iconv -f EBCDIC-US -t UTF-8 input_file > output_file

And programmatically you use the standard iconv_open() set of
functions. All OS that I know of have these features.

If I were to add 'CHARSET:UTF-8' to the above UTF-8 versions of
the object and then charset convert it to EBCDIC for my IBM program
to understand - it would simply convert 'CHARSET:UTF-8' to
the sequence:

	 ÃÈÁÙâÅãzäãÆ`ø

Which would be utterly useless unless outside of the object
itself I knew that it was the EBCDIC-US rendering of the
UTF-8 string 'CHARSET:UTF-8'. But then it has no meaning because
what good is saying 'CHARSET:UTF-8' in an 'EBCDIC-US' object.

Books that I recommend are:

	Creating Worldwide Software
		Bill Tuthill
		David Smallberg
		Sun Microsystems / Prentice Hall
		ISBN 0-13-494493-3
And:

	Programming for the World
	A Guide to Internationalization
		Sandra Martin O'Donnel
		Prentice Hall
		ISBN 0-13-722190-8

The 2nd (I think) was out of print - but I got it from an online
site that will legally sell you copies of out of print
books. Sorry I do not remember the URL or name.

-- 

  Doug Royer                     |   http://INET-Consulting.com
  -------------------------------|-----------------------------
  [email-masked]                 | Office: (208)612-INET
  http://Royer.com/People/Doug   |    Fax: (866)594-8574
                                 |   Cell: (208)520-4044

                 We Do Standards - You Need Standards

---

From: Mark Swanson <[email-masked]>
Date: Thu, 27 Feb 2003 14:15:53 -0500

> process various charsets.

Doug, I really do understand your MIME argument.
But, as others have mentioned (and quoted from 2445) iCalendar objects MUST be 
able to work outside the MIME environment. I think you are trying to address 
this in the next paragraph...
<quote>
the format in this memo is equally applicable
   for use outside of a MIME message content type
</quote>

> without telling the other endpoint the charset of the data itself.

What I was trying to describe was how Apple seems to have set up their 
iCalendar client/server. They seem to have a web server (and other sites have 
started to mimick this in a big way) that is serving iCalendar content, and I 
described how the web server is not capable of setting the proper character 
set because all iCalendar files would have the same extension.

Apple gets by with a web server, and we get by with something different. I can 
see some major advantages to using the webserver approach and we may switch 
to it for some uses in the future.

I do not like the fact that iCalendar forces me to use CAP or MIME especially 
when the mandate fo 2445 was for it to work outside of MIME. Forcing me to 
use CAP or MIME seriously limits its usefulness.

>In CAP that problem is solved.

I understand this. I think 2445 is seriously broken in this regard.
XML solves this issue by forcing the charset encoding attribute to be in 
ASCII.
2445 should learn from this and do the same.

In order to interoperate, 2445 MUST be able to specify a CHARSET property and 
its value in ASCII only.

Perhaps we need to modify BEGIN:VCALENDAR to:

1. BEGIN:VCALENDAR,CHARSET="UTF-8"
2. allow this line to be in ASCII only
3. bump the VERSION from 2.0 to 2.1.

Cheers.

-- 
Schedule your world with ScheduleWorld.com
http://www.ScheduleWorld.com/
Java Web Start:
http://www.ScheduleWorld.com/sw/ScheduleWorld.jnlp

---

From: Mark Swanson <[email-masked]>
Date: Thu, 27 Feb 2003 14:22:32 -0500

> 3. bump the VERSION from 2.0 to 2.1.

OK, after I hit send I realized this isn't exactly a compatible upgrade.

What about:
1. BEGIN:VCALENDAR
2. CHARSET:UTF-8

Where these two lines MUST be ASCII only, and they MUST be the first two 
lines. The CHARSET value must be an IANA registered charset (ASCII).

---

From: Doug Royer <[email-masked]>
Date: Thu, 27 Feb 2003 13:08:26 -0700

> XML solves this issue by forcing the charset encoding attribute to be in 

No. Is what the XML spec says is that most are detectable because
the VALUE of the 'encoding' attribute to the 'xml' tag must be
in ASCII (however encoded) . However there are some charsets where
that is difficult and it can require a hit and miss testing to determine
the charset. (see Chris Olds 26-FEB where he quoted from the XML spec)

There is an XML-CAL proposal draft-ietf-calsch-many-xcal-02.txt
that is in the process of defining a mapping between iCAL format
and XML, but that will not come out until after CAP.

> 3. bump the VERSION from 2.0 to 2.1.

Way too late for that as implementations already exist.
We could add CHARSET  - but it would be usless because
the existing implementations charset convert the entire
object now.

--

---

From: "Mike Higginbottom" <[email-masked]>
Date: Thu, 27 Feb 2003 20:27:48 -0000

Doug,

Well, something is seriously flawed here.  Either it's my understanding of
how this is all supposed to fit together or it's the content of 2445.  I'm
equally prepared to accept either as the result of this discussion ;-)

----- Original Message -----
From: "Doug Royer" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, February 27, 2003 5:23 PM
Subject: The charset issue

 > (This message sent in UTF-8 - assuming Netscape did not lie to me)
That's what my MUA tells me it is.

>to something your MUA can display.
Yes, but you're talking about e-mail here rather than iCalendar and I would
anticipate that a Chinese e-mail would have content that was entirely
representable using the Chinese character set.  If a Chinese person wanted
to e-mail the text BEGIN:VCALENDAR how would he do that?  My understanding
is that he would have to specify a MIME part using a charset that *could*
represent those characters (with whatever code points were appropriate).
There's no way to mix charsets within a single MIME part and yet we are
apparently supposed to mix characters from one charset with characters from
another charset in the same body part

>process various charsets.
Fair enough, but the tools don't know how to process two mutually exclusive
charsets within the same object.

>also has a way to set the charset and language that will be used.
Yes, but I remain to be convinced that it has a universally applicable way
to set the charset and language that will be used.

>for implementations. The problem has been solved for iCalendar.
Still not convinced....

>In CAP that problem is solved.
Yes, but my understanding of this propsed scheme was that (for example) you
specify the charset in the MIME headers that is to be used for the parsing
of the iCalendar object as a whole.  It would have to be mandatory for that
charset to support a representation of the characters used in the
'structural' text specified in 2445 (e.g. BEGIN:VCALENDAR).  The CHARSET
parameter would be present only in those properties that explicitly
supported NON-US-ASCII text (e.g. the value of an ATTENDEE property) and
would be applied only to the NON-US-ASCII portion of the property text.  The
text 'CHARSET' would be represented in the MIME specified charset (US-ASCII
or UTF-8 for example) that could represent the characters C, H, A, R, S, E,
T correctly and unambigously.  It's ugly and kludgy but in the absence of
anything better I think it would work.

>without telling the other endpoint the charset of the data itself.
Agreed.

>Now the same object in EBCDIC-US (attached) it is binary.
Sure, that's fine, but how would you approach this in some Chinese charset
rather than EBCDIC?

I think I'm still holding on to the issues involved here but please feel
free to tell me I'm talking rubbish.......

Mike Higginbottom

---

From: Doug Royer <[email-masked]>
Date: Thu, 27 Feb 2003 14:29:14 -0700

> lines. The CHARSET value must be an IANA registered charset (ASCII).

The problems is that existing implementations already convert
the entire BEGIN/END VCALENDAR to the target charset. So if you
are going to interoperate with them you are going to have to do that
work anyway.

In iCalendar we already solved this problem by having the
charset name in a layer above the blob of data. So iCalendar
objects are not restricted to the XML limitation of having
only MOST charsets guessable.

CAP uses BEEP and iMIP uses e-mail. If you want to propose
another transport - you can submit one. But there is going to be
a HUGE amount of resistance in making incompatible iCalendar
objects. And you will have incompatible iCalendar objects
with the proposal above.

--

---

From: "Chris Olds" <[email-masked]>
Date: Thu, 27 Feb 2003 14:04:15 -0800

On Thu, 27 Feb 2003 14:22:32 -0500, Mark Swanson wrote
> Where these two lines MUST be ASCII only, and they MUST be the first 
> two lines. The CHARSET value must be an IANA registered charset 
> (ASCII).

This could be relaxed a bit without losing anything.  As long as the charset
used supports the invariant portion of the ISO 646 charset (as defined in
RFC1345[1]) all of the tokens (keywords, x-names, iana-names and punctuation)
used in 2445 will be intelligible.  Obviously, this leaves out EBCDIC charsets
and charsets with more than 8 bits in the default state, but those charsets
don't work with 2445 as it is now either (absent a MIME wrapper).

    /cco

[1] ftp://ftp.isi.edu/in-notes/rfc1345.txt
--
GPG Key Fingerprint: B375 A4E7 752B DB8C 4359  852E C3CF BF64 379A E9B2
Debian Project (http://www.debian.org)

---

From: Doug Royer <[email-masked]>
Date: Thu, 27 Feb 2003 17:40:01 -0700

> don't work with 2445 as it is now either (absent a MIME wrapper).

Multitbyte will work when you know the charset type - as in CAP or
when the endpoints have agreed to the charset in advance
of using the object outside of MIME.

--

---

From: Mark Swanson <[email-masked]>
Date: Thu, 27 Feb 2003 21:49:42 -0500

> are going to interoperate with them you are going to have to do that

If implementations stick to the default recommended UTF-8 charset this will 
never be an issue, and a very minor one if they don't.

Doug, please let me know if you think differently. I am pushing for a CHARSET 
proposal but I do not want to impose any burden on existing or future systems 
based on the MIME or CAP transport. I also do not want to propose anything 
that will break compatibility.

> only MOST charsets guessable.

Which prevents iCalendar objects from working 100% of the time outside of 
MIME/CAP environments.

> objects. And you will have incompatible iCalendar objects

2445 is broken. It should be fixed. If you want to keep things the way they 
are then I think it would be fair to say you should propose we remove all 
wording like, "the format in this memo is equally applicable for use outside 
of a MIME message content type"

Here is a further modified proposal:

The first two properties MUST be:
1. BEGIN:VCALENDAR
2. CHARSET:UTF-8

With the following conditions:

1. Both of these properties are in a character set encoding that is 
ASCII-compatible. I think that was what Chris was saying about "supports the 
invariant portion of the ISO 646 charset". Chris: please speak up if I 
misunderstood you.

2. CHARSET  is optional. There is a single default charset: UTF-8. Put another 
way: if CHARSET is not specified the character encoding MUST be UTF-8.

I believe this should be completely compatible with version 2.0 of ICalendar 
specification.
If there are no major problems with this I will expand this into a formal 
request which will include a list of all of the good points made by the 
people on this list over the past few days.

---

From: Mark Swanson <[email-masked]>
Date: Thu, 27 Feb 2003 22:02:12 -0500

> I believe this should be completely compatible with version 2.0 of
> ICalendar specification.
Yep, hit send too quickly.
Compatible as long as the CHARSET property isn't used... :-)

I don't think an X- property should be used for this though that wouldn't 
break any existing parsers...

---

From: "Chris Olds" <[email-masked]>
Date: Thu, 27 Feb 2003 22:54:46 -0800

On Thu, 27 Feb 2003 21:49:42 -0500, Mark Swanson wrote
> 
> 2445 is broken. It should be fixed. If you want to keep things the 
> way they are then I think it would be fair to say you should propose 
> we remove all wording like, "the format in this memo is equally 
> applicable for use outside of a MIME message content type"
> 
> ASCII-compatible. I think that was what Chris was saying about 
> "supports the invariant portion of the ISO 646 charset". Chris: 
> please speak up if I misunderstood you.

What I said is weaker than 'US-ASCII compatible', but your understanding is
correct. (and yet, here I am speaking up...)

> 2. CHARSET  is optional. There is a single default charset: UTF-8. 
> Put another way: if CHARSET is not specified the character encoding 
> MUST be UTF-8.
> 
> ICalendar specification. If there are no major problems with this I 
> will expand this into a formal request which will include a list of 
> all of the good points made by the people on this list over the past 
> few days.

I think this proposal is a good idea.  I'd lke to offer some suggestions:

1) 2445 already says that the first line of an iCalendar object MUST contain a
delimiter string (sec 4.4, p50), so all you need to say is that an iCalendar
object (equiv. icalbody) MAY contain a CHARSET property, and if it does, the
CHARSET property MUST immediately follow the opening delimiter string ("BEGIN"
":" "VCALENDAR" CRLF)

2) It may be clearer to specify the CHARSET property without using 'UTF-8' in
the example; something like: 
  the CHARSET property is defined as "CHARSET" ":" iana-token
and then note that the iana-token should be a charset name.  CHARSET should be
specified so that it MUST NOT have any parameters, I think.

3) There is already precedent for a property in the iCalendar object having to
match the MIME headers (METHOD: sec 3.2, sec 4.7.2), and I think that it would
be good to follow this pattern for CHARSET.  Sec 4.7.2 says, in part:

   Description: When used in a MIME message entity, the value of this
   property MUST be the same as the Content-Type "method" parameter
   value. This property can only appear once within the iCalendar
   object. If either the "METHOD" property or the Content-Type "method"
   parameter is specified, then the other MUST also be specified.

Obviously, it isn't possible to use such strong wording for CHARSET, but
something along the lines of:

   Description: When used in a MIME message entity, the value of this
   property MUST be the same as the Content-Type "charset" parameter
   value. This property can only appear once within the iCalendar
   object, and MUST be precede any other properties or components.
   If either the "CHARSET" property or the Content-Type "charset"
   parameter is specified, then the other SHOULD also be specified.

A suggestion that programs that store or transmit iCalendar objects without
MIMI-encapsulation should add a CHARSET property (if none is present and the
charset is known) seems in order too.

It's too bad that 2445 doesn't include the 'iana-prop' production (it looks
like CAP adds that); if it did, a CHARSET property on iCalendar objects would
be completely compatible as soon as it was written up.  CAP-aware software
might not understand CHARSET, but it certainly shouldn't choke on it.

+1 for Mark's proposal.

     /cco

---

From: Andrea Campi <[email-masked]>
Date: Fri, 28 Feb 2003 12:23:46 +0100

> iCalendar client/server. They seem to have a web server (and other sites have 

Since the discussion has reached (IMO) a dead end, I'll voice my opinion.
I am 1000% of Doug's opinion. Mark, you are presenting a non-compliant
implementation as reason for changing RFC2445. Like it or not, an http
uses MIME, so everything you know about MIME applies - including provisions
set forth in RFC2445.

Without getting into too many details, here's an example:

brian% telnet www.webcom.it 80
GET /test.ics HTTP/1.1
Host: www.webcom.it

HTTP/1.1 200 OK
Date: Fri, 28 Feb 2003 11:19:49 GMT
[...]
Content-Length: 30
Content-Type: text/calendar; charset=utf-8

BEGIN:VCALENDAR
END:VCALENDAR

If Apple or any other websites get this wrong, well, that's their
problem, and they have to either fix it or face the incompatibilities
that will arise. And as far as I know, it might take some time, but
they WILL fix it.

So if I have to cast a vote, I am completely against changing RFC2445.

Bye,
	Andrea

-- 
Andrea Campi                              mailto:[email-masked]
I.NET S.p.A. - BT Ignite                  http://www.inet.it
Technical Dept. - R&D			  phone: +39 02 32863 ext 1
v. Darwin, 85 - I-20019			  fax: +39 02 32863 ext 7705
Settimo Milanese (MI), Italy

---

From: Mark Swanson <[email-masked]>
Date: Fri, 28 Feb 2003 08:50:26 -0500

> Content-Type: text/calendar; charset=utf-8

Perhaps you missed the discussion about this.
There is a big problem with it: if test1.ics is in utf-8 and test2.ics is in 
latin1 the web server will serve both as latin1. This can not work.

---

From: Mark Swanson <[email-masked]>
Date: Fri, 28 Feb 2003 09:17:04 -0500

> implementation as reason for changing RFC2445. Like it or not, an http

No Andrea. This is not why I am doing this at all.
Perhaps a summary of reasons will help:

1. MIME is not the universe (to steal a phrase from John Stacke :-). 2445 
states iCalendar objects are to work fine outside the MIME environment. To 
quote from 2445, "the format in this memo is equally applicable for use 
outside of a MIME message content type.".

There are the obvious examples of using an iCalendar object outside the MIME 
environment: http/ftp/custom protocols - even copying the iCalendar object as 
presented by Outlook (in text format, which does not show the MIME headers as 
part of the iCalendar object) to the clipboard.  John Strake has stated that 
no MIME implementations (MUA or HTTP) include the MIME headers when it saves 
a MIME body-part.

I believe a lot more examples will surface as time goes on. 2445 will likely 
last for decades, why not fix it now?

Lastly, I believe 2445 MUST be fixed. If people do NOT agree with the CHARSET 
property then they MUST agree to change the wording of 2445 to something 
like, "the format of this memo is only applicable for use in a MIME message 
content type".

---

From: Lawrence Greenfield <[email-masked]>
Date: Fri, 28 Feb 2003 13:43:07 -0500

From: Mark Swanson <[email-masked]>
   Date: Fri, 28 Feb 2003 09:17:04 -0500

   There are the obvious examples of using an iCalendar object outside
   the MIME environment: http/ftp/custom protocols - even copying the
   iCalendar object as presented by Outlook (in text format, which
   does not show the MIME headers as part of the iCalendar object) to
   the clipboard.  John Strake has stated that no MIME implementations
   (MUA or HTTP) include the MIME headers when it saves a MIME
   body-part.

Of course not. They should save objects in the local charset, doing
conversions if necessary.

   I believe a lot more examples will surface as time goes on. 2445
   will likely last for decades, why not fix it now?

   Lastly, I believe 2445 MUST be fixed. If people do NOT agree with
   the CHARSET property then they MUST agree to change the wording of
   2445 to something like, "the format of this memo is only applicable
   for use in a MIME message content type".

Simply stating that an iCalendar parser must know the charset before
interpreting an iCalendar object would be fine. If you want to give
some guide to implementors, we can even say "Implementations may want
to interpret byte sequences matching legal UTF-8 sequences as UTF-8
and all other byte sequences as their local character set." or
something of the sort.

Larry

---

From: Mark Swanson <[email-masked]>
Date: Fri, 28 Feb 2003 14:19:07 -0500

> conversions if necessary.

This is impossible because you can not always convert to the local character 
set (F.E. UTF-8 -> 8859-1).

> interpreting an iCalendar object would be fine. If you want to give

You are basically stating my solution, which is basically:

an ICalendar parser must know the charset before interpreting an iCalendar 
object because of the existence of a CHARSET property.

> and all other byte sequences as their local character set." or

No. This ignores the fact that the local character set may not be able to 
represent the encoded iCalendar characters.