Subject: Re: 3028bis open issue #3: require 2047 decoding?

From: Michael Haardt <[email-masked]>
Date: Fri, 1 Jul 2005 10:22:31 +0200

On Thu, Jun 30, 2005 at 05:43:53PM -0700, Philip Guenther wrote:
> 	Comparisons are performed in Unicode.  Implementations convert
> 	text from header fields in all charsets [HEADER-CHARSET] to
> 	Unicode as input to the comparator (see 2.7.3).  Implementations
> 	must be capable of decoding US-ASCII, ISO-8859-1, the US-ASCII
> 	subset of ISO-8859-* character sets, and UTF-8.

That sounds good.

> Hmm, I think the paragraph needs to also specify that text in unknown
> charsets never matches, no?

Either that or not decoding it, if it can not be converted.

But now that you cited the scope of 3028bis, can we do that?

   If implementations fail to support the above behavior, they MUST
   conform to the following:

      No two strings can be considered equal if one contains octets
      greater than 127.

If we agree that this only concers implementations that do not support
Unicode at all, of course we are free to specify what the result of
unicode supporting implementations is, if the conversion to it fails.
But I am not sure how exactly this is meant.

Michael

---

From: Philip Guenther <[email-masked]>
Date: Fri, 01 Jul 2005 13:56:21 -0700

>Either that or not decoding it, if it can not be converted.

So, if the implementation didn't understand the charset, it would
instead convert the raw ASCII (e.g., "=?charset?Q?blah?=") to Unicode
(an identity function, yes) and feed that to the comparator?  I guess I
can see _a_ use to that (scripts could match "=?charset?" to check for
use of particular charsets that the implementation doesn't support).

Hmm, I don't see any guidance in RFC 2047 or 2048 that could apply to
this.  Oh well...

>But now that you cited the scope of 3028bis, can we do that?

Good question.  RFC 3028 did not specify how an implementation should
handle charsets that it doesn't understand, effectively leaving it
implementation defined.  Is that causing interoperability problems?  If
so, then my understanding it that fixing it is in scope.  If not, then
we shouldn't add new constraints on implementations.  Is there consensus
that it's an interoperability problem with _and_ that we can obtain
consensus on how to resolve it?

Meanwhile, perhaps one of our chairs could speak on the procedural
point?

Philip Guenther

---

From: Ned Freed <[email-masked]>
Date: Fri, 01 Jul 2005 14:37:20 -0700 (PDT)

> Hmm, I don't see any guidance in RFC 2047 or 2048 that could apply to

I think leaving the encoded word intact in this case is slightly better. But
only slightly. The bottom line is that if there's a problem decoding (either
because you don't know the encoding or the encoded material is gronked somehow)
or converting (either because you don't recognize the charset or the material
doesn't actually match the charset), getting totally consistent and reasonable
match behavior is next to impossible. As such, any preference we express should
be just that: A preference. There's no best practice here as far as I can tell.

> implementation defined.  Is that causing interoperability problems?

I haven't seen actual problems in this space, but others may have
different experiences to report.

> If
> so, then my understanding it that fixing it is in scope.

It's in scope if a fix is possible. Aside from having a minimal set of charsets
you have to support (which I believe we already have) I don't see how to
fix this.

				Ned

---

From: Philip Guenther <[email-masked]>
Date: Fri, 01 Jul 2005 17:45:56 -0700

Ned Freed <[email-masked]> writes:
...
>I think leaving the encoded word intact in this case is slightly
>better. But only slightly. The bottom line is that if there's a problem
>decoding (either because you don't know the encoding or the encoded
>material is gronked somehow) or converting (either because you don't
>recognize the charset or the material doesn't actually match the
>charset), getting totally consistent and reasonable match behavior is
>next to impossible. As such, any preference we express should be just
>that: A preference. There's no best practice here as far as I can
>tell.

How about the following for the second paragraph of 2.7.2:
      Comparisons are performed in Unicode.  Implementations convert
      text from header fields in all charsets [HEADER-CHARSET] to
      Unicode as input to the comparator (see 2.7.3).  Implementations
      MUST be capable of converting US-ASCII, ISO-8859-1, the US-ASCII
      subset of ISO-8859-* character sets, and UTF-8.  Text that the
      implementation cannot convert to Unicode for any reason, MAY be
      omitted, treated as plain US-ASCII (including any [HEADER-CHARSET]
      syntax), or processed according to local conventions,

Philip Guenther

---

From: Ned Freed <[email-masked]>
Date: Sat, 02 Jul 2005 07:45:00 -0700 (PDT)

>       syntax), or processed according to local conventions,

Thought I sent a note about this, but cannot find it... Anyway, I think this
is fine, although I'd be tempted to out "treat as plain US-ASCII" first on the
list.

				Ned

---

From: Barry Leiba <[email-masked]>
Date: Sat, 02 Jul 2005 21:31:43 -0400

> is fine, although I'd be tempted to out "treat as plain US-ASCII" first on the

Definitely first on the list.  I actually think this *is* best practice, 
despite Ned's having said he doesn't think there is a best practice on
this, and, in particular, I'd like to eliminate "omitted".  Consider:

    Subject: =?bogus-charset?Q?Buy Viagra now!?=

Do you really want my rule that says
    if (header :contains ["subject"] ["viagra"]) {
       discard;
       stop;
    }
to be ignored because the spammer put in a bogus character set name
(perhaps purposefully, to screw up Sieve scripts)?

Barry

--
Barry Leiba, Pervasive Computing Technology  ([email-masked])
http://www.research.ibm.com/people/l/leiba
http://www.research.ibm.com/spam

---

From: Philip Guenther <[email-masked]>
Date: Sat, 02 Jul 2005 22:40:41 -0700

Barry Leiba <[email-masked]> writes:
>Ned Freed wrote:
>> Thought I sent a note about this, but cannot find it... Anyway, I
>> think this is fine, although I'd be tempted to out "treat as plain
>> US-ASCII" first on the list.
>
>Definitely first on the list.  <...>

Okay.  I have reordered the list so that the sentence now reads:

   Text that the implementation cannot convert to Unicode for any reason
   MAY be treated as plain US-ASCII (including any [HEADER-CHARSET]
   syntax), omitted, or processed according to local conventions.

Philip Guenther

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 04 Jul 2005 11:52:49 +0100

Philip Guenther wrote:

>Tim Showalter <[email-masked]> writes:
>  
>
>>Michael Haardt wrote:
>>    
>>
>>>Shouldn't the implementation be free what to convert them to? It may
>>>chose a different unicode representation.  Do we need to enforce UTF-8?
>>>      
>>>
>>I realize the language in 3028 clearly implies implementations are 
>>required to convert to UTF-8 but if an implementation wants to use UCS-4 
>>or UCS-2 or UTF-7 internally, that must be allowed.  The specification 
>>has no power to specify behavior that can't be externally observed, and 
>>the text you cited is just wrong.
>>    
>>
>...
>
>For 3028bis, 2.7.2, paragraph 2, how about:
>
Shouldn't this be MUST?

>with the new normative reference:
>
>[HEADER-CHARSET]	Moore, K., "MIME (Multipurpose Internet Mail
>			Extensions) Part Three: Message Header
>			Extensions for Non-ASCII Text", RFC 2047,
>			November 1996
>  
>
Sounds good to me.

>(You can't just map them to U+FFFD (replacement character) because you
>don't know how many characters are encoded!)
>  
>
Alexey

---

From: Ned Freed <[email-masked]>
Date: Thu, 07 Jul 2005 13:51:22 -0700 (PDT)

> this, and, in particular, I'd like to eliminate "omitted".

I certainly could live with omitting "omitted".

>  Consider:

> (perhaps purposefully, to screw up Sieve scripts)?

Good point.

				Ned

---

From: Philip Guenther <[email-masked]>
Date: Thu, 14 Jul 2005 19:42:10 -0700

Ned Freed <[email-masked]> writes:
><Barry wrote>

Since an implementation _could_ consider its "local convention" to be
"omit", removing the test doesn't really ban anything but does make the
idea less obvious.  I've removed it for the next rev.

Philip Guenther

---

From: Philip Guenther <[email-masked]>
Date: Thu, 14 Jul 2005 20:32:16 -0700

I wrote:
...
>idea less obvious.

That should say "removing the _text_", of course...