Subject: RE: iCalendar - "octets" question

From: "John Stracke" <[email-masked]>
Date: Tue, 27 Nov 2001 16:25:17 -0500

>Does this mean that in a multi-byte object you SHOULD wrap potentially in 
the middle of character?

Probably MAY, rather than SHOULD.

>Does this have any potential problems?

It shouldn't; folding and unfolding lines is defined to be irrelevant to 
the semantics of the lines, which means that the simplest way to do it is 
to do it below the level of the parser/lexer.  By the time the lines reach 
the parser, they've been unfolded.

One reason to make sure that splitting characters is explicitly permitted 
is that there are legitimate cases in which the code that folds the lines 
doesn't actually know it's using multibyte characters.  For example, the 
iCalendar implementation I did for eCal didn't contain any explicit 
character-set code; it was assumed that all the character sets we would 
support sending would be supersets of ASCII.  The code that did a MIME 
wrapper could tag the object as, say, UTF-8, but there was no need for the 
code that generated the lines to understand UTF-8; it just used whatever 
strings it was given.

/==========================================================\
|John Stracke                   |Principal Engineer        |
[email-masked]  |Incentive Systems, Inc.   |
|http://www.incentivesystems.com|My opinions are my own.   |
|==========================================================|
|A man's concepts should exceed his vocabulary, or what's a|
|metaphor?                                                 |
\==========================================================/

---

From: "Shannon J. Clark" <[email-masked]>
Date: Tue, 27 Nov 2001 15:58:26 -0600

John,

Thanks - that seems very reasonable, does anyone have proposed text to add
to this section to make this explicitly permitted?

(and perhaps to also define what an "octet" is...)

Shannon

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]]On Behalf Of John Stracke
Sent: Tuesday, November 27, 2001 3:25 PM
To: [email-masked]
Subject: RE: iCalendar - "octets" question

the middle of character?

---

From: Doug Royer <[email-masked]>
Date: Wed, 28 Nov 2001 16:37:18 -0700

> code that generated the lines to understand UTF-8; it just used whatever

Wrong:

For any character set, if your program knows that character set,
simply reading or writing a char in a I18N way will do the
correct thing. You can't even wrap a multibyte character because
you MUST NOT do OCTET counting, you MUST DO character counting.

If you don't know the character set, you have no business creating
the object.

If someone sends you an object, and you don't understand or
can not process the character set listed in the MIME header,
you have two choices:

	(1) Throw it away - because you would have no way
	    to know if it is single byte, double byte, or
	    a variable (multi) byte character set.

	(2) Try to parse it, if anything looks odd - go to step (1).

The only reason to wrap iCalendar object lines is for presentation
as it is *not* a must in 2445. If you can't present it because
you don't know the charset, don't process it.

If you were to accidentally wrap in after the first byte of
a two (or more) byte charset, the recipient would NEVER be able to
reassemble the object. Don't do OCTET counting.

---

From: Doug Royer <[email-masked]>
Date: Wed, 28 Nov 2001 17:09:50 -0700

> Probably MAY, rather than SHOULD.

NO!   -> MUST NOT <-

> >Does this have any potential problems?

BIG PROBLEMS!

	If you were to break apart the sequence 'AB' in this
	factious multi and variable byte charset:

	If the (factious) multi byte two octet sequence for the letter
	'A' (in hex):

		8A 0C

	And the (factious) 'B' were:

		8A 0D

	And you wrap (8A 0C 8A 0D) in the middle with a CR (OC) and a
	newline (OD) followed by a space (20), it would look like:

		8A 0C 0D 20 0C 8A 0D

	It would see the stream as:

		'A' newline space CR 'B'

	And NOT:

		'A' newline CR space 'B'

BUSTED!
	
> doesn't actually know it's using multibyte characters. ...

If you don't know the data format, don't muck with the data.
Most, but NOT all charsets are based on US-ASCII.

> ...For example, the
> support sending would be supersets of ASCII. ...

That will work much of the time for non-asian native charset.
But we MUST NOT support that - as there is NO way to predict
what the results would be.

>  ... The code that did a MIME

Did you ever test against any 8-bit UTF-8 strings that had
variable byte counts? I am guessing - no, or you were very lucky
that none of the wraps were in the middle of a variable byte 
sequence.