Subject: Re: 3028bis open issue #3: require 2047 decoding?

From: Michael Haardt <[email-masked]>
Date: Thu, 30 Jun 2005 11:34:13 +0200

> That is, support for RFC 2047 is only a SHOULD and not a MUST.  Do
> we want to leave that as is or should it be made stricter, with a
> MUST support RFC 2047, MUST support conversion of US-ASCII and UTF-8
> and SHOULD support conversion of ISO-8859-1 and the US-ASCII subset
> of ISO-8859-*?

I would prefer that, but:

>    If implementations fail to support the above behavior, they MUST
>    conform to the following:
>
>       No two strings can be considered equal if one contains octets
>       greater than 127.

To me, that states that if an implementation fails to convert a character
set to UTF-8, two strings can not be equal if one contains octets greater
than 127.  Assuming that all unknown character sets are one-byte character
sets with the lower 128 octects being US-ASCII is not sound.  But perhaps
that's not what was meant.

>    MIME parts identified as using charsets other than UTF-8 as
>    defined in [UTF-8] SHOULD be converted to UTF-8 prior to the match.

Shouldn't the implementation be free what to convert them to? It may
chose a different unicode representation.  Do we need to enforce UTF-8?

>    If an implementation does not support conversion of a given
>    charset to UTF-8, it MAY compare against the US-ASCII subset
>    of the transfer-decoded character data instead.  Characters from
>    documents tagged with charsets that the local implementation
>    cannot convert to UTF-8 and text from mistagged documents MAY
>    be omitted or processed according to local conventions.

That sounds more useful than RFC 3028 to me, but I slightly prefer to
match the raw transfer-encoded data.  Why bother decoding it, if you
can entirely unsure what it may be anyway?

Whatever the result is, I agree that comparisons for header and body
should be the same.

Michael

---

From: Tim Showalter <[email-masked]>
Date: Thu, 30 Jun 2005 13:19:21 -0700

> chose a different unicode representation.  Do we need to enforce UTF-8?

I realize the language in 3028 clearly implies implementations are 
required to convert to UTF-8 but if an implementation wants to use UCS-4 
or UCS-2 or UTF-7 internally, that must be allowed.  The specification 
has no power to specify behavior that can't be externally observed, and 
the text you cited is just wrong.

Perhaps the text should have been simply "MIME parts SHOULD be 
canonicalized to Unicode prior to the match".  I'm sure there's a way to 
phrase it in a way that's completely encoding-neutral, though.

Tim

---

From: Philip Guenther <[email-masked]>
Date: Thu, 30 Jun 2005 17:43:53 -0700

>the text you cited is just wrong.
...

For 3028bis, 2.7.2, paragraph 2, how about:

	Comparisons are performed in Unicode.  Implementations convert
	text from header fields in all charsets [HEADER-CHARSET] to
	Unicode as input to the comparator (see 2.7.3).  Implementations
	must be capable of decoding US-ASCII, ISO-8859-1, the US-ASCII
	subset of ISO-8859-* character sets, and UTF-8.

with the new normative reference:

[HEADER-CHARSET]	Moore, K., "MIME (Multipurpose Internet Mail
			Extensions) Part Three: Message Header
			Extensions for Non-ASCII Text", RFC 2047,
			November 1996

Hmm, I think the paragraph needs to also specify that text in unknown
charsets never matches, no?

(You can't just map them to U+FFFD (replacement character) because you
don't know how many characters are encoded!)

Philip Guenther

---

From: Tim Showalter <[email-masked]>
Date: Thu, 30 Jun 2005 19:18:35 -0700

> 	subset of ISO-8859-* character sets, and UTF-8.

Yeah, that's good.

> charsets never matches, no?

I think you're right.

Tim