Subject: [EAI] silly states and header label

From: Chris Newman <[email-masked]>
Date: Mon, 20 Mar 2006 01:15:41 -0600

One thing I dislike more and more as time goes on is silly states: when two 
elements in a protocol provide contradictory information.  Whenever possible I 
prefer to design silly states out of the protocol.

Suppose we have no way to label a UTF-8 header message.  We now have three 
cases the code has to deal with:
1. The header is pure 7-bit.
2. The header is 8-bit and complies with UTF-8 syntax.
3. The header is 8-bit and does not comply with UTF-8 syntax.

We don't need to talk much about 3, but it does permit software to bounce 
certain non-standard behaviors or gateway them into something more reasonable. 
I assert the code to validate 2 need not be particularly expensive.  Indeed, 
I'll include a high-performance version of the code at the end of this message 
to prove my point.

The problem I have with a header to declare message type is that changes from 
three cases to six cases, at least two of which are silly states.  And since 
headers can be re-ordered it's still necessary to scan all the headers to find 
the label, so I don't see how the benefits (detection) can outweigh the costs 
(doubling the number of cases to deal with).  Were I writing code, I'd ignore 
the label, just as I ignore the MIME-Version header (which is only helpful as a 
hint if one supports RFC 1049).

If we must use a labeling mechanism, I suggest using a zero-width non-joiner 
UTF-8 character at the beginning of the message.  UTF-8 aware SMTP servers are 
required to insert Received and Return-Path after the zero-width non-joiner 
character.  That has some other nice features -- it permits instant detection 
of the encoding for certain out-of-band mechanisms (e.g. UTF-8 editors), it has 
no impact on display, and if the message gets to a non-8-bit system the failure 
is likely to be immediate and more obvious rather than later and more obscure 
(indeed it's so obvious that legacy systems can be patched to check for it 
easily).  I'm still not sure those benefits outweigh the cost of silly states, 
but it's closer to making the cut.

                - Chris
-----
UTF-8 detection:

/* a UTF8 lookup table */
#define EXT 0x20
#define BAD 0x40
#define SP  0x10
static unsigned char utf8_table[256] = {
    /* 0x00 */  BAD,    1,    1,    1,    1,    1,    1,    1,
    /* 0x08 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x10 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x18 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x20 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x28 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x30 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x38 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x40 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x48 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x50 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x58 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x60 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x68 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x70 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x78 */    1,    1,    1,    1,    1,    1,    1,    1,
    /* 0x80 */  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,
    /* 0x88 */  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,
    /* 0x90 */  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,
    /* 0x98 */  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,
    /* 0xA0 */  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,
    /* 0xA8 */  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,
    /* 0xB0 */  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,
    /* 0xB8 */  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,  EXT,
    /* 0xC0 */  BAD,  BAD,    2,    2,    2,    2,    2,    2,
    /* 0xC8 */    2,    2,    2,    2,    2,    2,    2,    2,
    /* 0xD0 */    2,    2,    2,    2,    2,    2,    2,    2,
    /* 0xD8 */    2,    2,    2,    2,    2,    2,    2,    2,
    /* 0xE0 */ SP|3,    3,    3,    3,    3,    3,    3,    3,
    /* 0xE8 */    3,    3,    3,    3,    3, SP|3,    3,    3,
    /* 0xF0 */ SP|4,    4,    4,    4, SP|4,  BAD,  BAD,  BAD,
    /* 0xF8 */  BAD,  BAD,  BAD,  BAD,  BAD,  BAD,  BAD,  BAD
};

/* verify a string is valid UTF-8 and doesn't contain NUL
 *  if len < 0, strlen(str) will be used.
 * returns -1 if not valid, 0 if valid
 */
int utf8verify(const char *str, int len)
{
    int state = 0, code;
    const char *end;
    unsigned char u1, u2;

    if (len < 0) len = strlen(str);
    end = str + len;
    while (str < end && (code = utf8_table[(unsigned char)*str]) != BAD) {
        if ((state == 0 && code == EXT) || (state > 0 && code != EXT)) {
            return (-1);
        }
        if ((state & SP) != 0) {
            state &= ~SP;
            u1 = (unsigned char) str[-1];
            u2 = (unsigned char) str[0];
            if ((u1 == 0xE0U && u2 < 0xA0U)
                || (u1 == 0xEDU && u2 > 0x9FU)
                || (u1 == 0xF0U && u2 < 0x90U)
                || (u1 == 0xF4U && u2 > 0x8FU)) {
                return (-1);
            }
        }
        if (code != EXT) {
            state = code;
        }
        --state;
        ++str;
    }

    return (str == end && state == 0 ? 0 : -1);
}
#undef SP

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Martin Duerst <[email-masked]>
Date: Tue, 21 Mar 2006 18:16:52 +0900

At 16:15 06/03/20, Chris Newman wrote:

 >The problem I have with a header to declare message type is that changes 
from three cases to six cases, at least two of which are silly states.  And 
since headers can be re-ordered it's still necessary to scan all the 
headers to find the label, so I don't see how the benefits (detection) can 
outweigh the costs (doubling the number of cases to deal with).  Were I 
writing code, I'd ignore the label, just as I ignore the MIME-Version 
header (which is only helpful as a hint if one supports RFC 1049).

I very much agree with the above.

 >If we must use a labeling mechanism, I suggest using a zero-width 
non-joiner UTF-8 character at the beginning of the message.  UTF-8 aware 
SMTP servers are required to insert Received and Return-Path after the 
zero-width non-joiner character.  That has some other nice features -- it 
permits instant detection of the encoding for certain out-of-band 
mechanisms (e.g. UTF-8 editors), it has no impact on display, and if the 
message gets to a non-8-bit system the failure is likely to be immediate 
and more obvious rather than later and more obscure (indeed it's so obvious 
that legacy systems can be patched to check for it easily).  I'm still not 
sure those benefits outweigh the cost of silly states, but it's closer to 
making the cut.

I'm not at all sure this is a good idea. It changes the overall message
format. Besides header fields and message body, separated by CRLFCRLF,
we now have something additional. Some tools that just pass through
the eigth bit (I'm more thinking about all kinds of secondary tools
rather than the main MUAs and MTAs,...) may choke.

Also, it's difficult to see, which makes debugging more difficult.
This also (together with what you wrote above) makes it difficult to drop.
And it would introduce a bad precedent of using

Regards,    Martin.