Subject: [EAI] Problems with empty group in downgraded headers

From: Ernie Dainow <[email-masked]>
Date: Tue, 12 Aug 2008 10:22:03 -0400

We've been doing some testing with Downgrade and encountered a case that 
does not work very well.

We send email to an ASCII address at a non-utf8 server. The message is 
downgraded and delivered. The CC: field has a utf8 address that does not 
have an alternate address. The CC: address is downgraded according to 
the Downgrade spec, using an empty group ":;" to note that a utf8 
address has been removed.  This is specified in Section 5 of 
draft-ietf-eai-downgrade-08 as
    [ Display-name ] "<" Utf8-addr-spec ">"
    Rewrite as
    [ Display-name ] "Internationalized Address " Encoded-word " Removed:;"

If the recipient of such a downgraded message attempts to Reply All, 
many email clients fail to send the message, complaining about an 
invalid email address. We tried four different email clients and all 
failed. (Three of the four were able to send a message if the group is 
not empty.)

An empty group is specified as valid in RFC822, so technically speaking 
there is nothing wrong with the Downgrade spec. However, from a 
practical point of view, when used in the broader email world, this will 
lead to the perception that Downgrade is broken.

What about using a comment instead of an empty group to provide the 
information about removed utf8 addresses? In my limited testing, 
comments seem to work much more reliably than empty groups.

Ernie

_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Tue, 12 Aug 2008 11:09:59 -0400

--On Tuesday, August 12, 2008 10:22 AM -0400 Ernie Dainow 
<[email-masked]> wrote:

> We've been doing some testing with Downgrade and encountered a
> case that does not work very well.
>
> We send email to an ASCII address at a non-utf8 server. The
> message is downgraded and delivered. The CC: field has a utf8
> address that does not have an alternate address. The CC:
> address is downgraded according to the Downgrade spec, using
> an empty group ":;" to note that a utf8 address has been
> removed.  This is specified in Section 5 of
> If the recipient of such a downgraded message attempts to
> Reply All, many email clients fail to send the message,
> complaining about an invalid email address. We tried four
> different email clients and all failed. (Three of the four
> were able to send a message if the group is not empty.)
>
> An empty group is specified as valid in RFC822, so technically
> speaking there is nothing wrong with the Downgrade spec.
> However, from a practical point of view, when used in the
> broader email world, this will lead to the perception that
> Downgrade is broken.
>
> What about using a comment instead of an empty group to
> provide the information about removed utf8 addresses? In my
> limited testing, comments seem to work much more reliably than
> empty groups.

Ernie,

You (and we) are in a bit of a bind here.   The empty group 
syntax has been around since at least 822, yet several popular 
MUAs don't recognize it.  On the other hand,
   cc: (comment)
with no other information present is simply not valid.  2822 says

    cc  =  "Cc:" address-list CRLF

and I can't find anything in the tree leading down from 
"address-list" that would permit having no addresses in it.

So, if I read things correctly, you are suggesting that we 
replace a valid construction that is not accepted by some broken 
MUAs with an invalid construction.  While it is just my opinion, 
somehow I don't think so.

More broadly, this is part of the "experiment" with Downgrade. 
If, in practice, it cannot work well enough to deliver a large 
percentage of messages without an unacceptable loss of 
information under normally-hostile conditions, then we need to 
reconsider either our criteria or whether it is worth the 
consider complexity that it adds to the i18n email environment.

     john

---

From: Ernie Dainow <[email-masked]>
Date: Tue, 12 Aug 2008 18:00:17 -0400

John C Klensin wrote:
> So, if I read things correctly, you are suggesting that we replace a 
> valid construction that is not accepted by some broken MUAs with an 
> invalid construction.  While it is just my opinion, somehow I don't 
> think so.
If a comment won't work, are there any other ways to handle this? One 
option is to not downgrading this type of header. It avoids the problem 
altogether but loses the information about other recipients who received 
the email.

I'm not sure how serious this problem will be; we tried mostly 
mainstream email clients. To put it another way, does anyone know of an 
email client that does handle empty groups?
> More broadly, this is part of the "experiment" with Downgrade. If, in 
> practice, it cannot work well enough to deliver a large percentage of 
> messages without an unacceptable loss of information under 
> normally-hostile conditions, then we need to reconsider either our 
> criteria or whether it is worth the consider complexity that it adds 
> to the i18n email environment.
>
>     john
Ernie

---

From: Pete Resnick <[email-masked]>
Date: Wed, 13 Aug 2008 16:46:56 -0500

On 8/12/08 at 6:00 PM -0400, Ernie Dainow wrote:

>mainstream email clients. To put it another way, does anyone know of 
>an email client that does handle empty groups?

Eudora does just fine. Thunderbird does not, but I can pretty much 
assure that *that* bug will be fixed quickly.

pr
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
Qualcomm Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102
_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: Pete Resnick <[email-masked]>
Date: Wed, 13 Aug 2008 17:14:12 -0500

On 8/13/08 at 4:46 PM -0500, Pete Resnick wrote:

>>mainstream email clients. To put it another way, does anyone know 
>>of an email client that does handle empty groups?
>

By the way: For what it's worth, Eudora generates group syntax all of 
the time for Bcc'ed recipients and has done so for years. So it's 
quite worrisome that there are clients out there that handle this 
badly.

(*grumble*)

pr

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 14 Aug 2008 10:51:20 +0100

On Tue, 12 Aug 2008 16:09:59 +0100, John C Klensin <[email-masked]> wrote:

> Ernie,
>
> You (and we) are in a bit of a bind here.   The empty group syntax has  
> been around since at least 822, yet several popular MUAs don't recognize  
> it.  On the other hand,
>    cc: (comment)

But you could put an empty group PLUS a comment.

We are dealing with broken MUAs that fail to handle what is in fact  
correct 2822 syntax. So presumably the MUA reports "message not sent  
because of yadda yadda", where "yadda yadda" might or might not be a  
helpful explanation of the problem. So what is then needed is something  
that will enable the recipient (or at least a more clueful recipient) to  
piece together manually some useful reply message that he can send. So a  
comment that indicates what the CC was meant to contain might assist that  
process.

But that should not really be necessary given that there will also be a
     Dongraded: Cc: ...
present, which should be enough for the recipient to figure out some  
alternative way to send the Cc, or to decide that it (or the offending  
part of it in the case of multiple Ccs) coule safely be omitted from the  
reply.

So all any comment would have to do is to say "(see Downgraded header)".

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5
_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: Tony Finch <[email-masked]>
Date: Thu, 14 Aug 2008 13:12:54 +0100

On Wed, 13 Aug 2008, Pete Resnick wrote:
>
> By the way: For what it's worth, Eudora generates group syntax all of the time
> for Bcc'ed recipients and has done so for years. So it's quite worrisome that
> there are clients out there that handle this badly.

Certain Microsoft MUAs try to generate an empty group for BCCed
recipients but cock it up in an amazingly cretinous manner, e.g.
	To: <Undisclosed-Recipient:;>

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
SOUTHEAST ICELAND: SOUTHWESTERLY 4 OR 5, OCCASIONALLY 6 LATER IN WEST.
MODERATE. MAINLY FAIR. GOOD.
_______________________________________________
IMA mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ima

---

From: Ernie Dainow <[email-masked]>
Date: Thu, 14 Aug 2008 11:33:23 -0400

Charles Lindsey wrote:
>> You (and we) are in a bit of a bind here.   The empty group syntax 
>> has been around since at least 822, yet several popular MUAs don't 
>> recognize it.  On the other hand,
>>    cc: (comment)
Good suggestion. I tried this with a few of the failing email clients. 
In one case, the client failed to send as before, complaining about 
invalid address. In the other case, the client did send the message but 
garbled the Cc: header. The recipient received the message with a Cc: 
containing comment-body <groupname;;>. I'm concluding that this approach 
will not provide much better results than an empty header alone.
> helpful explanation of the problem. So what is then needed is 
> something that will enable the recipient (or at least a more clueful 
> recipient) to piece together manually some useful reply message that 
> he can send. So a comment that indicates what the CC was meant to 
> contain might assist that process.
>
> part of it in the case of multiple Ccs) coule safely be omitted from 
> the reply.
>
Yes people on this mailing list and other literate email users can 
figure out the problem and just remove the empty group so that a Reply 
All can be sent. The vast majority of email users will have difficulty.

Here is the latest count. As suggested by Andrew S., Mutt works with 
empty groups. However I encountered failures with two more email client. 
The count is now 2/8. If Thunderbird is corrected (as promised by Pete 
R.) we will be 3/8, still a failing grade.

Here's a suggestion on how to downgrade these headers with comments 
instead of empty groups.

When downgrading the To: header, remove all addresses that are utf8 only 
(i.e. without alternate address) and list them (encoded) in a single 
comment like
    (Following internationalized addresses removed from To: display 
<addr> ... )
If the To: header is not empty, append the comment, otherwise place the 
comment on hold.

Repeat this for each of Cc:, Bcc: As soon as a non-empty header is 
encountered, append all comments that were put on hold. At least one of 
these headers must be non empty, as the Downgrade is only done when 
there is an ASCII email alternate address.

Not only will this work more reliably than the empty group construct, I 
think the resulting email is more readable. All the removed addresses 
are in a single list at the end of the header instead of being scattered 
throughout the header embedded in an empty group for each removed address.

Ernie

---

From: Pete Resnick <[email-masked]>
Date: Fri, 15 Aug 2008 14:07:57 -0500

On 8/12/08 at 10:22 AM -0400, Ernie Dainow wrote:

>failed. (Three of the four were able to send a message if the group 
>is not empty.)
>
>speaking there is nothing wrong with the Downgrade spec. However, 
>from a practical point of view, when used in the broader email 
>world, this will lead to the perception that Downgrade is broken.

So, having thought about this a while, I think the right answer is to 
leave things exactly as they are. Though I still believe that the 
e-mail clients in question are broken, they're not as broken as I 
first thought. Let me use a non-EAI example to explain:

Eudora currently uses the group construct when the sender types 
addresses into the Bcc: field but not into the To: field, or uses the 
group addressing features from the Address Book. So, I might receive 
the following message today:

     To: Joe's Friends:;
     From: Joe Blow <[email-masked]>
     Subject: testing

     This is a test.

If I do a reply-all to this message, my client might generate the following:

     To: Joe Blow <[email-masked]>
     From: Pete Resnick <[email-masked]>
     Cc: Joe's Friends:;

But actually there's something wrong with that. In the message I 
received, the To: field indicated that this message was sent to a 
group of people (whose addresses were not revealed) called "Joe's 
Friends". But if I send this reply, it is most certainly *not* going 
to a group of people called "Joe's Friends". I would claim that a 
reasonable client might say, "Hey, I don't know any group by the name 
of 'Joe's Friends'. Do you really want to send this message to an 
empty group?" and offer to remove it. I agree that it is terribly bad 
form for the client to say, "Invalid Cc" or "Cc must contain 
'user@host'", but I do think it's reasonable to say, "You're trying 
to reply to someone that you can't send any mail to."

In the case of EAI, I do think it's fair warning for a client to say, 
"You're trying to reply to someone that you can't send any mail to" 
and make the user go back and correct the recipients. EAI is, after 
all, an experimental protocol, and clients should get their act 
together to deal with cases that the user may try reply to addresses 
to which they cannot send.

Let's not change the algorithm.

pr

---

From: Joseph Yee <[email-masked]>
Date: Fri, 15 Aug 2008 16:53:08 -0400

Pete Resnick wrote:
>> speaking there is nothing wrong with the Downgrade spec. However, from 
>> a practical point of view, when used in the broader email world, this 
>> will lead to the perception that Downgrade is broken.
> 
> leave things exactly as they are. Though I still believe that the e-mail 
> clients in question are broken, they're not as broken as I first 
> thought. Let me use a non-EAI example to explain:
> 
> group addressing features from the Address Book. So, I might receive the 
> following message today:
> 
>     To: Joe's Friends:;
> received, the To: field indicated that this message was sent to a group 
> of people (whose addresses were not revealed) called "Joe's Friends". 
> But if I send this reply, it is most certainly *not* going to a group of 
> people called "Joe's Friends". I would claim that a reasonable client 
> might say, "Hey, I don't know any group by the name of 'Joe's Friends'. 
> Do you really want to send this message to an empty group?" and offer to 
> remove it. I agree that it is terribly bad form for the client to say, 
> "Invalid Cc" or "Cc must contain 'user@host'", but I do think it's 
> reasonable to say, "You're trying to reply to someone that you can't 
> send any mail to."
> 
> "You're trying to reply to someone that you can't send any mail to" and 
> make the user go back and correct the recipients. EAI is, after all, an 
> experimental protocol, and clients should get their act together to deal 
> with cases that the user may try reply to addresses to which they cannot 
> send.
> 

Agreed to Pete that to keep the current mechanism.  I have tested group 
and comment for various mail clients some time before, the results are 
same to what many had posted.  I personally don't think it's bad.  Many 
MUA fails with loud voice for attempting to send to empty group, sender 
will notice something.  This is better than sending to black hole and 
get sender thinking 'my group' receive the message.

When I first test it, I assume the downgrade doc's intention is to 
deliver the message with information of all recipients.  I personally 
believe this is very valuable.  I strongly recommend not to remove the 
information regards to IDN email address recipient(s).

Combining comment to empty group is just appending string.  We may just 
need a better message for the group name so that the recipient knows 
'reply-all' can't reach all.

Joseph