Subject: Re: The charset issue

From: Andrea Campi <[email-masked]>
Date: Sat, 1 Mar 2003 13:08:34 +0100

On Fri, Feb 28, 2003 at 08:50:26AM -0500, Mark Swanson wrote:
> Perhaps you missed the discussion about this.
> There is a big problem with it: if test1.ics is in utf-8 and test2.ics is in 
> latin1 the web server will serve both as latin1. This can not work.

Uhm... I see no reason why that should be the case. It's an
implementation detail really, but just for an example, CERN-style
meta files have been around for like 10 years; if using Apache you
can also use mod_asis, even though it's not optimal. Or even if
using static files you could pass all requests through a script
which determines the appropriate charset via your preferred
mechanism, and adds it to the MIME headers.

Bye,
	Andrea

-- 
Andrea Campi                              mailto:[email-masked]
I.NET S.p.A. - BT Ignite                  http://www.inet.it
Technical Dept. - R&D			  phone: +39 02 32863 ext 1
v. Darwin, 85 - I-20019			  fax: +39 02 32863 ext 7705
Settimo Milanese (MI), Italy

---

From: Andrea Campi <[email-masked]>
Date: Sat, 1 Mar 2003 13:20:30 +0100

On Fri, Feb 28, 2003 at 09:17:04AM -0500, Mark Swanson wrote:
> On February 28, 2003 06:23 am, Andrea Campi wrote:
> >
> > Since the discussion has reached (IMO) a dead end, I'll voice my opinion.
> > I am 1000% of Doug's opinion. Mark, you are presenting a non-compliant
> > implementation as reason for changing RFC2445. Like it or not, an http
> 
> No Andrea. This is not why I am doing this at all.
> Perhaps a summary of reasons will help:
> 
> 1. MIME is not the universe (to steal a phrase from John Stacke :-). 2445 
> states iCalendar objects are to work fine outside the MIME environment. To 
> quote from 2445, "the format in this memo is equally applicable for use 
> outside of a MIME message content type.".

This is exactly the reason why I think no change is necessary. If
you are in a MIME environment, use MIME. If you are not, you MUST
use UTF-8. I think this is quite reasonable in fact.

> There are the obvious examples of using an iCalendar object outside the MIME 
> environment: http/ftp/custom protocols - even copying the iCalendar object as 
> presented by Outlook (in text format, which does not show the MIME headers as 
> part of the iCalendar object) to the clipboard.  John Strake has stated that 
> no MIME implementations (MUA or HTTP) include the MIME headers when it saves 
> a MIME body-part.

You see - implementation errors. ;-) Even for clipboards, you do
have a choice - you can trancode to UTF-8 before sending the data
to the clipboard, i.e. before throwing away the charset information.

I agree with you that it would be nice to be universally compatible;
however, I don't think that should be done by compromising on the
present clean specification - expecially since I still haven't seen
a single example where implementations could not do The Right Thing (TM)
within the standard.

Bye,
	Andrea

---

From: Andrea Campi <[email-masked]>
Date: Sat, 1 Mar 2003 13:23:49 +0100

On Fri, Feb 28, 2003 at 01:43:07PM -0500, Lawrence Greenfield wrote:
>    There are the obvious examples of using an iCalendar object outside
>    the MIME environment: http/ftp/custom protocols - even copying the
>    iCalendar object as presented by Outlook (in text format, which
>    does not show the MIME headers as part of the iCalendar object) to
>    the clipboard.  John Strake has stated that no MIME implementations
>    (MUA or HTTP) include the MIME headers when it saves a MIME
>    body-part.
> 
> Of course not. They should save objects in the local charset, doing
> conversions if necessary.

Absolutely not, they should convert everything to UTF-8. I honestly
can't see the reason for NOT doing that - since any conforming
implementation MUST be able to handle at least UTF-8, why go to
the effort of using anything else as a native charset? I mean I
can see the need to import or even export data in different
charsets, but UTF-8 should be the main one.

Bye,
	Andrea

---

From: Mark Swanson <[email-masked]>
Date: Sun, 2 Mar 2003 22:14:28 -0500

On March 1, 2003 07:08 am, Andrea Campi wrote:
> > There is a big problem with it: if test1.ics is in utf-8 and test2.ics is
> > in latin1 the web server will serve both as latin1. This can not work.
>

My web server doesn't support CERN-style meta files.

> can also use mod_asis, even though it's not optimal. Or even if

mod_asis is not supported by my web browser, and even though I agree it would 
serve the purpose of providing the MIME headers I still do not like the fact 
that requiring MIME breaks the fundamental encapsulation of iCalendar.

> mechanism, and adds it to the MIME headers.

You could, but imposing that as a restriction to using iCalendar is not 
reasonable.

Again, the fundamental objection is: 2445 should work equally well outside of 
a MIME environment.

-- 
Schedule your world with ScheduleWorld.com
http://www.ScheduleWorld.com/
Java Web Start:
http://www.ScheduleWorld.com/sw/ScheduleWorld.jnlp

---

From: Mark Swanson <[email-masked]>
Date: Sun, 2 Mar 2003 22:25:14 -0500

On March 1, 2003 07:20 am, Andrea Campi wrote:
> > > Since the discussion has reached (IMO) a dead end, I'll voice my
> > > opinion. I am 1000% of Doug's opinion. Mark, you are presenting a
> > > non-compliant implementation as reason for changing RFC2445. Like it or
> > > not, an http
> >
> > states iCalendar objects are to work fine outside the MIME environment.
> > To quote from 2445, "the format in this memo is equally applicable for
> > use outside of a MIME message content type.".
>

I mostly quite like that.
Unfortunately 2445 does not agree with us. 
Instead of a CHARSET property would you agree to rewording 2445 so UTF-8 MUST 
be used if not in a MIME environment?

> > There are the obvious examples of using an iCalendar object outside the
> > MIME environment: http/ftp/custom protocols - even copying the iCalendar
> > object as presented by Outlook (in text format, which does not show the
> > MIME headers as part of the iCalendar object) to the clipboard.  John
> > Strake has stated that no MIME implementations (MUA or HTTP) include the
> > MIME headers when it saves a MIME body-part.
>

Oh? You are going to expect all past and future software to do this? 
Notepad.exe for example?

> a single example where implementations could not do The Right Thing (TM)

The examples are endless: FTP for example, notepad editing a 2445 document and 
pasting to clipboard for another. Perhaps more importantly the possibilities 
are endless because not all environments are MIME environments.

---

From: Doug Royer <[email-masked]>
Date: Mon, 03 Mar 2003 10:56:30 -0700

> be used if not in a MIME environment?

That is not going to happen ether as several asian countries charsets
are more efficient then UTF-8 for them. We did not pick UTF-8 and the
wording in 2445 at random. It was discussed over a few years and agreed
upon. Changing 2445 in an incompatible way is just not going to happen.
If we changed it to a MUST it would make currently compliant applications
non compliant. Do you think they are just going to ignore that issue?

 >> ...

> Oh? You are going to expect all past and future software to do this? 

It is *not* the job of an interoperability protocol to be view-ready.
It is the job of an interoperability protocol to be interoperable
with multiple venders across multiple countries. Your OS tool specific
example is not relevant. The point made was that you can write
code to do copy/paste, do it or not but do not expect every
vendor to agree that they have to change.

> are endless because not all environments are MIME environments.

Your point is not relevant to iCalendar as FTP and Notepad.exe were never
designed to be iCalendar aware as was an endless list of other unrelated
software.

Just because you do not want to write code to translate charset does
not mean the rest of the iCalendar compliant vendors should break
their code to make you happy.

If you have suggestions that can extend or make it more useful, great.
But making a new version of 2445 break or make non compliant existing
implementations is not going to happen.

-- 

  Doug Royer                     |   http://INET-Consulting.com
  -------------------------------|-----------------------------
  [email-masked]                 | Office: (208)612-INET
  http://Royer.com/People/Doug   |    Fax: (866)594-8574
                                 |   Cell: (208)520-4044

                 We Do Standards - You Need Standards

---

From: Mark Swanson <[email-masked]>
Date: Mon, 3 Mar 2003 14:01:35 -0500

On March 3, 2003 12:56 pm, Doug Royer wrote:
> > Instead of a CHARSET property would you agree to rewording 2445 so UTF-8
> > MUST be used if not in a MIME environment?
>

Others have pointed this out to me too. I agree we can't force UTF-8.

> vendor to agree that they have to change.

My opinion is that 2445 should encapsulate charset so we can work outside the 
MIME environment, as mandated by the spec.

> > The examples are endless: FTP for example, notepad editing a 2445
> > document and pasting to clipboard for another. Perhaps more importantly
> > the possibilities are endless because not all environments are MIME
> > environments.
>

It is relevant because iCalendar states that it should work outside of a MIME 
environment.

> their code to make you happy.

It is wrong of you to make this personal. It is wrong of you to publicly state 
that I want or do not want to do something when you have absolutely no idea 
what my reasons or intentions are.

Peace...

> implementations is not going to happen.

Perhaps it is me. Perhaps I just do not understand one single sentence in 
2445:

"However, the format in this memo is equally applicable
   for use outside of a MIME message content type."

Because of this, I maintain the spec is broken because the format will not 
work without a CHARSET property.

All arguments seem to be based around, "use MIME". But this is in conflict 
with the sentence above.

I see multiple organizations taking advantage of this to break 
interoperability in various ways (from calendar clients and servers to cell 
phones) and I feel it is only going to get worse.

There has not been a single better proposal than the CHARSET property for 
fixing the above statement.

---

From: Andrea Campi <[email-masked]>
Date: Tue, 4 Mar 2003 10:53:35 +0100

This will be my last post on this thread, sorry my time is short lately.
I feel this issue has been beaten to death, and Doug is voicing my
opinion as well so I don't have much to add. Just a summary:

> All arguments seem to be based around, "use MIME". But this is in conflict 

The format in the memo is perfectly applicable without MIME. However,
the fact that your application works without MIME doesn't allow you to
disregard charset issues; rather, it puts the burden on you to make sure
you know how to interpret iCalendar data properly. This I think is the
spirit of the above sentence, and thus the spec is doing its duty.

You've been repeatedly citing the example of passing iCalendar data around
with tools which are unaware of MIME. That's fine, you can either convert
the data to UTF-8 or pass around external metadata. For instance, if a MUA
saves a part from a MIME message and then calls an helper application to
read it, it can either convert from the original charset to UTF-8, or it
can pass the charset as a command line argument (for instance).

> phones) and I feel it is only going to get worse.

What makes you feel those organizations will update their applications to
this new iteration of the spec you are proposing?

> fixing the above statement. 

Sorry, I don't want to sound blunt, but since you're raising the issue
and advocating a change, the onus is on you to bring evidence for the
change, not the other way around. So far the hasn't been a single
example of something which is not working because of a flaw in the
spec instead of a vendor mistake.

Bye,
	Andrea

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 09:46:50 -0500

Lawrence Greenfield wrote:

>   John Strake has stated that no MIME implementations
What is the local charset? POSIX, for example, specifies that locale 
settings are stored in environment variables, which means that charsets 
are per-process, not per-user or per-machine.

-- 
/================================================================\
|John Stracke      [email-masked]                         |
|Principal Engineer|http://www.centive.com                       |
|Centive           |My opinions are my own.                      |
|================================================================|
|He wondered if Elli was going to buy that explanation. His taste|
|for heavily-armed girlfriends did have its drawbacks.           |
\================================================================/

---

From: [email-masked]
Date: Tue, 4 Mar 2003 10:42:45 -0500

Andre Campi  wrote
>spec instead of a vendor mistake.

I have already stated the example where this is happening.
Phones in Japan create iCalendars in Shift-JIS and attach them to email 
messages.  Because  by standard you are suppose to hand the attachment to 
the registered handler on an OPEN, the attachment processor no longer has 
any MIME charset information.

_____________________
Note: new email address

[email-masked]

Andrea Campi <[email-masked]> 
Sent by: [email-masked]
03/04/2003 04:53 AM

To
Mark Swanson <[email-masked]>
cc
"[email-masked]" <[email-masked]>
Subject
Re: The charset issue

Bye,
                 Andrea

-- 
Andrea Campi                              mailto:[email-masked]
I.NET S.p.A. - BT Ignite                  http://www.inet.it
Technical Dept. - R&D                                              phone: 
+39 02 32863 ext 1
v. Darwin, 85 - I-20019                                            fax: 
+39 02 32863 ext 7705
Settimo Milanese (MI), Italy

---

From: Andrea Campi <[email-masked]>
Date: Tue, 4 Mar 2003 17:08:54 +0100

> any MIME charset information.

Do they set the correct charset on the MIME part?
If no, then it's their fault, and they should fix it.
If yes, then it's the MUAs fault for not handling things properly.

4.1.4 Character Set

   There is not a property parameter to declare the character set used
   in a property value. The default character set for an iCalendar
   object is UTF-8 as defined in [RFC 2279].

   The "charset" Content-Type parameter can be used in MIME transports
   to specify any other IANA registered character set.

Which to me means: unless MIME header says otherwise (or your app knows
better) the native charset is UTF-8.

So to get back to my message you quoted: this example doesn't show
a fault in the spec (IMO).

Bye,
	Andrea

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 09:24:18 -0700

> are per-process, not per-user or per-machine.

That works if it works for the implementation. However there is no
such restriction as to the stored charset. Just because my application
is running in en_US.US-ASCII, does not mean that it can not read
in a binary blob of SHIFT-JIS and charset translate it to UTF-8
for storage. The user view and data storage do not have to be in the
same charset. Nether does the view and the transport charset have
to be the same.

--

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 09:33:24 -0700

[email-masked] wrote:
> messages.  Because  by standard you are suppose to hand the attachment 
> to the registered handler on an OPEN, the attachment processor no longer 
> has any MIME charset information.

I am not aware of any such 'standard' that says '...suppose to hand
the attachment to the registered handler on an OPEN, the attachment
processor no longer has any MIME charset information' If it exists
please post the standards body name and the name or ID of the standard.

So far it looks to me as if you are using an OS library that does
that for you. That however is your implementation decision and not
a MUA, MIME, or iCalendar restriction.

--

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 11:59:44 -0500

> please post the standards body name and the name or ID of the standard.

It is, however, a de facto standard, because that's what everybody does. 
 Nobody's told everybody to do it, but everybody has to accept the 
reality that everyone else does it.

-- 
/=================================================================\
|John Stracke      [email-masked]                          |
|Principal Engineer|http://www.centive.com                        |
|Centive           |My opinions are my own.                       |
|=================================================================|
|"Hastur was paranoid, which was simply a sensible...well-adjusted|
|reaction to living in Hell." --_Good Omens_                      |
\=================================================================/

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 12:04:20 -0500

Doug Royer wrote:

> Just because my application
> for storage.

Once again: MUAs don't do that.

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 12:04:14 -0500

>If yes, then it's the MUAs fault for not handling things properly.
Since no existing MUA does what you want, it is not legitimate to say 
that what you want is "handling things properly".  Any new standard that 
uses RFC-X needs to take into account the environment of existing 
implementations of RFC-X.  iCalendar has failed to do so.

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 10:52:55 -0700

John Stracke wrote:
> Since no existing MUA does what you want,

John - are you sure? :-)

 >  it is not legitimate to say

No. Implementations may have failed, however many work and
they do not do it by magic. It it because there are ways
to get the information, MIME headers, and iCalendar attachments.

--

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 11:00:05 -0700

> Once again: MUAs don't do that.

They don't have to do that. That's not their job. We are
talking about a calendar aware program which can also be an MUA.
You CAN send iCalendar attachments using any MUA that I have
seen. MUA's are not responsible for displaying video attachments
any more or less than iCalendar attachments.

No where in RFC 244[567] or CAP does it mandate or suggest
that MUAs be calendar aware. iMIP simply shows how to transport
iCalendar data using MIME. 'MUA' is only used once in 2447
and it is not talking about mandating that MUA be CUAs.

--

---

From: Andrea Campi <[email-masked]>
Date: Tue, 4 Mar 2003 19:09:40 +0100

> Since no existing MUA does what you want, it is not legitimate to say 

The one I use (mutt) does, as all Unix MUAs which use metamail. In fact
metamail has been handling this correctly for the best part of a
decade. Correctly, in this context, means it's able to (be configured to)
pass the charset as a command-line argument to any helper application
it calls.

That said, I'll readily agree that this is a niche application. Still
it show that SOME existing MUA chose to do it right.

I don't use Windows so I can't easily check it, but given that
Unicode has been the native format from Win NT 3.5 (or was that 4?),
it would surprise me if no MUA chose to convert from the incoming
charset to some sort of Unicode before passing the data to an
helper application.

Bye,
	andrea

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 11:02:06 -0700

> reality that everyone else does it.

Everybody does not do that. And my other point is still valid. If that
method does not work - don't use it.  There are methods that do work.

--

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 13:34:22 -0500

>> Since no existing MUA does what you want,

Um, no, not any more.

-- 
/=======================================================\
|John Stracke      [email-masked]                |
|Principal Engineer|http://www.centive.com              |
|Centive           |My opinions are my own.             |
|=======================================================|
|"Music is not a noun, it's a verb." --John Perry Barlow|
\=======================================================/

---

From: Mark Swanson <[email-masked]>
Date: Tue, 4 Mar 2003 13:56:03 -0500

> spirit of the above sentence, and thus the spec is doing its duty.

Would it be fair to say this: you feel it is up to non-MIME environments to 
come up with their own charset solution, while I feel this should be solved 
by having the document state its charset.

My reason is because there exists a number of applications that do not provide 
a charset solution, and I'm sure there will be future applications that do 
not provide on either. It would seem to me that 2445 would only benefit from 
encapsulating the charset.

> can pass the charset as a command line argument (for instance).

No. Someone in Japan can't use notepad to edit a SHIFT-JIS iCalendar document 
and ftp it to me because I will not know what charset the document is in.

> > interoperability in various ways (from calendar clients and servers to
> > cell phones) and I feel it is only going to get worse.
>

I believe people will follow the spec if they can. Also, if the IETF releases 
a new spec people will have no choice if they want to interoperate. It's the 
same reason people try to comply the best they can right now.

> spec instead of a vendor mistake.

There must be a disconnect as there are several posted by more than one 
person.

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 14:30:15 -0500

Mark Swanson wrote:

>No. Someone in Japan can't use notepad
>
(Or vi, or emacs.)

> to edit a SHIFT-JIS iCalendar document 

>same reason people try to comply the best they can right now.
Right.  And anybody that implements a Proposed Standard needs to know 
that they might have to change their implementation when it goes to 
Draft Standard--which, by definition, incorporates experience with 
interoperability problems.

-- 
/===============================================================\
|John Stracke      [email-masked]                        |
|Principal Engineer|http://www.centive.com                      |
|Centive           |My opinions are my own.                     |
|===============================================================|
|"I use emacs, which might be thought of as a thermonuclear word|
|processor." -- Neal Stephenson                                 |
\===============================================================/

---

From: [email-masked]
Date: Wed, 05 Mar 2003 17:47:38 -0800 (PST)

> pass the charset as a command-line argument to any helper application

The MUA I use does this as well, and it isn't based on metamail. (I've been
considering giving it access to content-disposition parameters as well.) And
the EMS callout interface in Eudora provides access to both content-type and
content-disposition parameters.

It is also worth pointing out that the mailcap format metamail and others use
that includes this ability to access content-type parameters is specified
in RFC 1343.

> it show that SOME existing MUA chose to do it right.

More than one, actually. FWIW.

				Ned

---

From: John Stracke <[email-masked]>
Date: Thu, 06 Mar 2003 09:07:51 -0500

>that includes this ability to access content-type parameters is specified
OK, I'm dropping this line of argument; I was grossly ignorant.

|"What now, Brain?" "We should flee in terror. Yes, that would be|
|the wisest course."                                             |