Subject: re: Charset

From: [email-masked]
Date: Mon, 3 Mar 2003 14:32:20 -0500

I agree with Mark Swanson, we should add the CHARSET property.  I had to 
modify Notes code to allow  the recipient to try different charsets on 
attached iCalendar objects.  The reason was that some phones in Japan can 
generate iCalendar.  However they are being generated in SHIFT-JIS.  They 
are sent as attachments to email.  As a consequence to this problem 
Notes/Domino 6.0 always inserts an X-LOTUS-CHARSET <inana-charset> as just 
below the BEGIN: VCALENDAR.  Thus, if it was generated as an attachment by 
Notes (i.e sending some sort of combined event such as conference), I 
absolutely know the charset.
_____________________
Note: new email address

[email-masked]

---

From: Doug Royer <[email-masked]>
Date: Mon, 03 Mar 2003 13:10:19 -0700

What is the MIME header for the attachment? What is the Content-Type
header in the E-mail?

-

[email-masked] wrote:
> attached iCalendar objects.  The reason was that some phones in Japan 
> can generate iCalendar.  However they are being generated in SHIFT-JIS. 
>  They are sent as attachments to email.  As a consequence to this 
> problem Notes/Domino 6.0 always inserts an X-LOTUS-CHARSET 
> <inana-charset> as just below the BEGIN: VCALENDAR.  Thus, if it was 
> generated as an attachment by Notes (i.e sending some sort of combined 
> event such as conference), I absolutely know the charset.
> _____________________
> Note: new email address
> 

-- 

  Doug Royer                     |   http://INET-Consulting.com
  -------------------------------|-----------------------------
  [email-masked]                 | Office: (208)612-INET
  http://Royer.com/People/Doug   |    Fax: (866)594-8574
                                 |   Cell: (208)520-4044

                 We Do Standards - You Need Standards

---

From: [email-masked]
Date: Mon, 3 Mar 2003 15:16:32 -0500

Does not matter since I have to look at attachment after it was 
de-attached.
_____________________
Note: new email address

Doug Royer <[email-masked]> 
Sent by: [email-masked]
03/03/2003 03:10 PM
Please respond to
"[email-masked]" <[email-masked]>

To
"[email-masked]" <[email-masked]>
cc

Subject
Re: Charset

-

--

---

From: Doug Royer <[email-masked]>
Date: Mon, 03 Mar 2003 13:57:06 -0700

> Does not matter since I have to look at attachment after it was 

Then that is the real problem not the fact that the object
does not have its original charset inside the object. You seem
to be talking about how you store the object in your application.

If you are electing to save it in its original charset then that
is fine. If you charset it to UTF-8, then you would not need it.

What do you do when you get a charset that you can not read without first 
knowing the charset - after ripping off the MIME header?

It looks to me as if MIME and iCalendar is working as documented.
Otherwise you would not know that it was SHIFT-JIS. And that
is what iCalendar + iMIP is about - it is already 100% portable - correct?

So despite the fact that you want to add CHARSET to the object,
your telling me that as an interoperability protocol - you did
not need it. How you store object internally (which is what I seem
to think you are saying) is out of scope for this working group.

--

---

From: [email-masked]
Date: Mon, 3 Mar 2003 16:16:23 -0500

When you try to OPEN an attachment, you must pass attachment to registered 
handler for the attachment.  So attachment is extracted and handed off. It 
is no longer in MIME envelope.
_____________________
Note: new email address

03/03/2003 03:57 PM

To

Subject
Re: Charset

--

---

From: Doug Royer <[email-masked]>
Date: Mon, 03 Mar 2003 15:31:17 -0700

[email-masked] wrote:
> 
> When you try to OPEN an attachment, you must pass attachment to 
> registered handler for the attachment.  So attachment is extracted and 
> handed off.  It is no longer in MIME envelope.

That is what your implementation does. It is not because
of iCalendar. Besides even if you are using some kind
of library or tool on your OS, I can't believe that
the MIME data is unavilable, if it were you would also
loose encoding and other information. Its your CUA,
and if not then intercept the MIME data prior to being
sent to the MUA and send it to your CUA.

Adding CHARSET to iCalendar objects will only make
it work some times for some implementations and only
when they try and guess the charset. Not a portable
solution.

--

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 09:46:45 -0500

Doug Royer wrote:

>> registered handler for the attachment.  So attachment is extracted 
>> and handed off.  It is no longer in MIME envelope.
>
> That is what your implementation does.

That is what EVERY implementation does!

Doug, you're resisting changing iCalendar because of the deployed base 
of CUAs; but what about the much larger deployed base of MUAs? If we 
want to be able to use iMIP, we cannot require that MUAs be modified to 
transport iCalendar safely.

-- 
/================================================================\
|John Stracke      [email-masked]                         |
|Principal Engineer|http://www.centive.com                       |
|Centive           |My opinions are my own.                      |
|================================================================|
|He wondered if Elli was going to buy that explanation. His taste|
|for heavily-armed girlfriends did have its drawbacks.           |
\================================================================/

---

From: Lawrence Greenfield <[email-masked]>
Date: Tue, 4 Mar 2003 10:24:47 -0500

Date: Tue, 04 Mar 2003 09:46:45 -0500
   From: John Stracke <[email-masked]>
[...]
   Doug, you're resisting changing iCalendar because of the deployed base 
   of CUAs; but what about the much larger deployed base of MUAs? If we 
   want to be able to use iMIP, we cannot require that MUAs be modified to 
   transport iCalendar safely.

Having redundant and potentially conflicting character set tagging is
harmful, too. Once the character set also exists inside the MIME
object, all tools that might try to do a character set transcoding must
be aware of the iCalendar format.

If you want to go ahead with iCalendar having its own character set
tagging system, you should choose a MIME type other than
text/calendar so that processing systems do not default it to
text/plain.

Larry

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 09:20:05 -0700

> transport iCalendar safely.

No modification is needed in a MUA, they work already as MUAs do
support MIME.

--

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 09:27:43 -0700

> be aware of the iCalendar format.

No, it means that they can do charset translation an never be aware
of the content.

> text/calendar so that processing systems do not default it to

I do not follow your logic on that point. I can charset translate
any text/plain to any other charset once I look at the MIME header
charset attribute and never know or care about the formatting
of the contents.

--

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 12:01:39 -0500

Doug Royer wrote:

>
>
> John Stracke wrote:
>
>> Doug, you're resisting changing iCalendar because of the deployed 
>> base of CUAs; but what about the much larger deployed base of MUAs? 
>> If we want to be able to use iMIP, we cannot require that MUAs be 
>> modified to transport iCalendar safely.
>

False.  Once again: the step of transporting the attachment from the 
MIME body to the helper application is not MIME-based.  The only cases 
where the CUA gets the MIME headers are those where the CUA is part of 
the MUA.

-- 
/=================================================================\
|John Stracke      [email-masked]                          |
|Principal Engineer|http://www.centive.com                        |
|Centive           |My opinions are my own.                       |
|=================================================================|
|"Hastur was paranoid, which was simply a sensible...well-adjusted|
|reaction to living in Hell." --_Good Omens_                      |
\=================================================================/

---

From: Lawrence Greenfield <[email-masked]>
Date: Tue, 4 Mar 2003 12:51:11 -0500

Date: Tue, 04 Mar 2003 09:27:43 -0700
   From: Doug Royer <[email-masked]>
[...]

Huh?

That's correct. However, if iCalendar objects ("text/calendar")
contained their _own_ charset information (via a CHARSET:utf-8
property or whatever) a naive conversion routine would not know to
update that property.

Larry

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 11:14:25 -0700

> MIME body to the helper application is not MIME-based.

No, the method that SOME use might work that way. If it does
not work for you or your application - use another method.

I can extract the MIME headers on Windows and Unix from e-mail.
It is an implementation decission - not a problem in the specification.

> The only cases where the CUA gets the MIME headers are those where the
 > CUA is part of the MUA.

In the area of calendaring - yes you need a CUA. Just like to view
video you need some kind of viewer. That is orthogonal to the MIME
header issue as there are methods that do work to get he MIME headers
with the attachements.

--

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 11:26:48 -0700

> property or whatever) a naive conversion routine would not know to

As that would only work some of the time - why should we consider that?

--

---

From: John Stracke <[email-masked]>
Date: Tue, 04 Mar 2003 14:06:08 -0500

Doug Royer wrote:

> That is orthogonal to the MIME

But there is no good method to do so without the cooperation of the MUA.

I suppose there are some *bad* methods.  If you know where the message 
came from (e.g., a local spool or an IMAP server), and have access to 
that source (might be tricky, if the IMAP server demands a certificate 
and the CUA doesn't have it, or if the IMAP server doesn't permit 
simultaneous logins--some don't--or if it's a nonstandard message 
store), you might be able to search for text/calendar attachments and 
read the MIME headers directly.  It'd be disgustingly slow, though.

-- 
/===============================================================\
|John Stracke      [email-masked]                        |
|Principal Engineer|http://www.centive.com                      |
|Centive           |My opinions are my own.                     |
|===============================================================|
|"I use emacs, which might be thought of as a thermonuclear word|
|processor." -- Neal Stephenson                                 |
\===============================================================/

---

From: "Chris Olds" <[email-masked]>
Date: Tue, 4 Mar 2003 12:42:14 -0800

On Tue, 04 Mar 2003 11:26:48 -0700, Doug Royer wrote
> As that would only work some of the time - why should we consider that?

Because MIME-encapsulation only works some of the time, and a CHARSET property
helps with that complimentary set.  

I cannot recall any MUA that transcodes attachments (I would be interested to
hear of an example), so I don't think that suggesting such transcoding is the
duty of the MUA is a useful resolution to this issue, nor do I think that the
potential problems with naive transcoding rule out a CHARSET property.

Some comments on other emails I recieved today:

The majority of non-EBCDIC character sets implement the invariant portions of
ISO 646, so the property would provide usable information in those cases.

Windows NT (and it heirs and assigns) are natively Unicode, but Unicode is not
commonly used as the storage format for text documents (in the US, at least);
the only UNICODE-capable editors than come to mind are TextPad and Emacs.

    /cco

--
GPG Key Fingerprint: B375 A4E7 752B DB8C 4359  852E C3CF BF64 379A E9B2
Debian Project (http://www.debian.org)

---

From: Doug Royer <[email-masked]>
Date: Tue, 04 Mar 2003 15:33:26 -0700

Chris Olds wrote:

 > Because MIME-encapsulation only works some of the time,

Until someone proposes and this WG agrees to a new iCalendar
transfer protocol. I do not know of any proposed inter vendor
issues with charset and iCalendar, iTIP, iMIP, or CAP. So
far all of the issues presented are implementation issues and
they are not protocol and transfer of information issues.

 > and a CHARSET  property helps with that complimentary set.

And makes some implementations objects in some charsets break.

> the only UNICODE-capable editors than come to mind are TextPad and Emacs.

This WG is concerted with transporting iCalendar objects between vendors
as an interoperability protocol. Why should we care if you can hand
edit an object or not? Copy/Paste and object or not?

---

From: Mark Swanson <[email-masked]>
Date: Wed, 5 Mar 2003 11:37:49 -0500

> they are not protocol and transfer of information issues.

I can not find anywhere in the charter or in 2445 that states only certain 
transfer protocols are allowed. To me, this means that the iCalendar object 
is encapsulated well enough that it does not matter what transfer protocol is 
used. Many statements in the rfc and charter back this up.

> edit an object or not? Copy/Paste and object or not?

I care about how people will use ICalendar and I would like to see it used in 
many flexible ways. I believe the spirit of ICalendar is to have an 
encapsulated format you can send with any transport mechanism you like - even 
if that transport mechanism is something as simple as the clipboard.

It seems to me the original authors of the spec (perhaps we could ask them) 
went through enormous pain in order to provide this flexibility and succeeded 
in every way except for the charset issue. As Chris mentioned, there is even 
a quote from 2445:

Formal Definition: The character sets supported by this revision of
   iCalendar are UTF-8 and US ASCII thereof. The applicability to other
   character sets is for future work. 

This leads me to believe that we are doing this future work right now, and can 
complete the spec as it was intended to work.

Comments welcome as always.

-- 
Schedule your world with ScheduleWorld.com
http://www.ScheduleWorld.com/
Java Web Start:
http://www.ScheduleWorld.com/sw/ScheduleWorld.jnlp

---

From: John Stracke <[email-masked]>
Date: Thu, 06 Mar 2003 09:14:08 -0500

Doug Royer wrote:

> Why should we care if you can [...]
> Copy/Paste and object or not?

Well, obviously, somebody cared when 2445 was written.  From the Abstract:

   In
   addition, the content type is useful as an object for interactions
   between desktop applications using the operating system clipboard,

Clipboards tend to be typed, but I'm not aware of any that uses MIME 
headers.  Mac doesn't; X doesn't; I'm pretty sure Windows doesn't (not 
surprising, since all of these predate MIME--RFC-1341, 1992).

Admittedly, it doesn't actually seem to be covered in the WG's charter.

|"What now, Brain?" "We should flee in terror. Yes, that would be|
|the wisest course."                                             |

---

From: John Stracke <[email-masked]>
Date: Thu, 06 Mar 2003 09:15:57 -0500

>complete the spec as it was intended to work.
Hear, hear.  In fact, given the tarpit that awaits us if people fall 
into the trap of placing non-UTF-8 iCalendar objects into non-MIME 
environments (as they are starting to do), I'd say it might be important 
enough to split our attention and work on it alongside CAP.

---

From: Mark Swanson <[email-masked]>
Date: Thu, 6 Mar 2003 10:00:22 -0500

>    between desktop applications using the operating system clipboard,

Excellent quote!