Subject: Re: [EAI] downgrade annotations and RCPT TO

From: "YAO Jiankang" <[email-masked]>
Date: Tue, 25 Jul 2006 18:38:39 +0800

utf8-capability server A--->non-utf8-capability server B (may relay server)--->utf8-capability server C

if the above situation is encountered, A may connect B first through DNS MX lookup

A sends B with ehlo-->A finds no EAI keyword in the response message-->A will use RCPT TO <alt-address> in the following operations. so alt-address will be used in rcpt to.

Yao Jiankang 

----- Original Message ----- 
From: "Tony Finch" <[email-masked]>
To: <[email-masked]>
Sent: Tuesday, July 25, 2006 5:44 PM
Subject: [EAI] downgrade annotations and RCPT TO

> As I understand it, the rough consensus of the group is that email between
> two UTF8 users should go through UTF8 infrastructure, and downgrading is
> only used when a UTF8 user is emailing an ASCII user.
> 
> This implies that downgrade information (ATOM^H^H^H^H DOWNGRADABLE and
> ALT_ADDRESS) on RCPT commands is not necessary.

not true.

> 
> This has the consequence that it is a configuration error to have a
> non-UTF8-capable MTA agent in the transmission path between two UTF8
> users, and if one is encountered the message must be bounced.
> 

may not be bounced.

> Tony.
> -- 
> f.a.n.finch  <[email-masked]>  http://dotat.at/
> FISHER NORTH GERMAN BIGHT: WEST VEERING NORTHWEST 4 OR 5, OCCASIONALLY 6 IN
> NORTHEAST FISHER. FAIR. MODERATE BECOMING GOOD.
> 
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/ima
> 

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: John C Klensin <[email-masked]>
Date: Tue, 25 Jul 2006 06:58:01 -0400

--On Tuesday, 25 July, 2006 18:38 +0800 YAO Jiankang
<[email-masked]> wrote:

> utf8-capability server A--->non-utf8-capability server B (may
> relay server)--->utf8-capability server C
> 

Sure.  However, this is precisely the "configuration error"
pointed to below.  It is in general stupid for the management of
"C", which supports and cares about utf-8 addresses, to
configure an MX backup or relay that does not support that
capability.  If we had a case in which no one who was
UTF-8-address-capable ever configured a lower-preference MX that
did not have that capability, then the case reduces to the
direct connnection (no relay) one and there would be no need for
a downgrade-on-the-wire capability at all: either A (presumably
the submission server) would get the rejection (absence of
authorizing keyword) on connection to C or would be able to send
without problems or C would be reluctant to advertise
UTF-8-based addresses.

There are some edge cases.  In particular, if C believes that
downgrading is widely supported (rather than having messages
bounce) and especially if it also believes that upgrading works
without loss of information, it might be tempted to configure a
low-capability MX relay to provide backup, get around an
incapable firewall, or out of general sloth or indifference.
But each of those cases can be turned into an argument that
downgrading capability is actually harmful in that it will
discourage full deployment of EAI facilities on the only basis
that will really work... radiating outward from sites that
believe the capability is important.

   john

> 
> A sends B with ehlo-->A finds no EAI keyword in the response
> message-->A will use RCPT TO <alt-address> in the following
> operations. so alt-address will be used in rcpt to.
> 
> 
> Yao Jiankang 
> 
>> As I understand it, the rough consensus of the group is that
>> email between two UTF8 users should go through UTF8
>> infrastructure, and downgrading is only used when a UTF8 user
>> is emailing an ASCII user.
>> 
>> This implies that downgrade information (ATOM^H^H^H^H
>> DOWNGRADABLE and ALT_ADDRESS) on RCPT commands is not
>> necessary.
> 
> not true.
> 
>> 
>> This has the consequence that it is a configuration error to
>> have a non-UTF8-capable MTA agent in the transmission path
>> between two UTF8 users, and if one is encountered the message
>> must be bounced.
>> 
> 
> may not be bounced.
> 
> 
>> Tony.
>> -- 
>> FISHER NORTH GERMAN BIGHT: WEST VEERING NORTHWEST 4 OR 5,
>> OCCASIONALLY 6 IN NORTHEAST FISHER. FAIR. MODERATE BECOMING
>> GOOD.
>>

---

From: Tony Finch <[email-masked]>
Date: Tue, 25 Jul 2006 12:09:45 +0100

> incapable firewall, or out of general sloth or indifference.

I was trying to argue that the current consensus (that email between UTF8
users must use entirely UTF8 infrastructure) implies that we should remove
the RCPT command downgrade extensions so that this kind of configuration
simply will not work (or will be unreliable).

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
THE MULL OF GALLOWAY TO MULL OF KINTYRE INCLUDING THE FIRTH OF CLYDE AND THE
NORTH CHANNEL: VARIABLE 2 TO 4. MAINLY FAIR, BUT RISK OF A SHOWER. MODERATE OR
GOOD. SLIGHT OR SMOOTH.

---

From: John C Klensin <[email-masked]>
Date: Tue, 25 Jul 2006 07:22:50 -0400

--On Tuesday, 25 July, 2006 12:09 +0100 Tony Finch
<[email-masked]> wrote:

>> bounce) and especially if it also believes that upgrading
>> works without loss of information, it might be tempted to
>> configure a low-capability MX relay to provide backup, get
>> around an incapable firewall, or out of general sloth or
>> indifference.
> 
> I was trying to argue that the current consensus (that email
> between UTF8 users must use entirely UTF8 infrastructure)
> implies that we should remove the RCPT command downgrade
> extensions so that this kind of configuration simply will not
> work (or will be unreliable).

That is equivalent, unless I misunderstand you, to removing
downgrade capability.  I am not yet ready to go on record as
believing that is the best solution.

      john

---

From: Tony Finch <[email-masked]>
Date: Tue, 25 Jul 2006 12:29:07 +0100

On Tue, 25 Jul 2006, John C Klensin wrote:
> On Tuesday, 25 July, 2006 12:09 +0100 Tony Finch wrote:
> >

No: there would still be downgrade information on the MAIL FROM command
and in the message header. Downgrading is only necessary when sending
email from a UTF8 user to ASCII users, in which case MAIL FROM needs
downgrading but not RCPT TO. For UTF8 recipients the message will go
through entirely UTF8 infrastructure so again the RCPT TO downgrade
information is redundant.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
SHANNON ROCKALL: CYCLONIC 3 OR 4, OCCASIONALLY 5 AT FIRST. SHOWERS. MODERATE
OR GOOD.

---

From: "Charles Lindsey" <[email-masked]>
Date: Tue, 25 Jul 2006 12:40:37 +0100

> believe the capability is important.

All that is true, and probably implies a poor configuration choice (except  
perhaps for the firewall case which may be totally impenetrable other than  
by ASCII). But we can be sure that such poor configurations _will_ happen,  
and we can't stop them. So we may as well let them work and not try to  
forbid them (obviously, our documents will point out that sensible  
configuration is much better).

But we don't want the situation where a message destined for an ASCII-only  
site gets through, but one addressed for a UTF-8 addrtess which follows  
the same route fails. That is just biting off your nose to spite your face.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John C Klensin <[email-masked]>
Date: Tue, 25 Jul 2006 07:54:29 -0400

--On Tuesday, 25 July, 2006 12:40 +0100 Charles Lindsey
<[email-masked]> wrote:

>> indifference. But each of those cases can be turned into an
>> argument that downgrading capability is actually harmful in
>> that it will discourage full deployment of EAI facilities on
>> the only basis that will really work... radiating outward
>> from sites that believe the capability is important.
> 
> All that is true, and probably implies a poor configuration
> choice (except  perhaps for the firewall case which may be
> totally impenetrable other than  by ASCII). But we can be sure
> that such poor configurations _will_ happen,  and we can't
> stop them. So we may as well let them work and not try to
> forbid them (obviously, our documents will point out that
> sensible  configuration is much better).
> 
> But we don't want the situation where a message destined for
> an ASCII-only  site gets through, but one addressed for a
> UTF-8 addrtess which follows  the same route fails. That is
> just biting off your nose to spite your face.

To make the problem clear, I am going to play devils advocate
below -- I am not necessarily convinced of all of it.

Charles, if there were just a matter of "let them work", I would
certainly agree with you.   But the issues associated with
downgrading comprise virtually all of the issues that make this 
effort complex.  It is not an accident that our discussions
about parameter formats, location of addresses in and out of
brackets, trick headers, and so on all relate to downgrading.
Most of the downgrade variants also imply burdens --for
recognition of options, headers, or header parameters-- on
fully-upgraded systems and will continue to impose those
requirements even after all, or virtually all, of the Internet
has been upgraded.  

If poor configurations happen (and I agree that they will) and
result in bouncing, then there will be complaints and incentives
to get them fixed.   If they 'just work', then there will be no
incentive to get them fixed and they may not even be noticed.
It is not clear that is the best tradeoff: perhaps poor
configurations should have obvious negative consequences so they
can be identified and corrected.

    john

---

From: Tony Finch <[email-masked]>
Date: Tue, 25 Jul 2006 13:27:20 +0100

> incentive to get them fixed and they may not even be noticed.

I agree.

The effect of allowing downgrading in the UTF8-to-UTF8 path will be, for
example, I send you a UTF8 message which gets downgraded; you reply to the
downgraded ASCII address, and the subsequent conversation ends up using
ASCII addresses instead of UTF8 addresses. This is contrary to the
requirements in the scenarios document.

I also anticipate that unexpected downgrading will be difficult for the
average postmaster to debug, moreso than a bounce with a clear error
message.

Tony.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 26 Jul 2006 21:47:17 +0100

> simply will not work (or will be unreliable).

But does "must" mean "MUST"? You point out that this same downgrade is  
needed for MAIL FROM, and it is certainly needed if the recipient is  
genuinely ASCII-only. So you woould be giving the appearance of a rather  
arbitrary rule.

Moreover, all these issues of whether to downgrade, and whether we need to  
specify (or imply) Atomic actually turn out to be more important in the  
case of From/MAIL FROM/Reply-To. You can probably ensure that your mail  
addressed to a UTF-8 recipient follows a "sensible" (UTF-8-capable) route.  
But you cannot assume the same when your recipient replies to you.

Moreover, there is still the case of a UTF-capable recipient sitting  
behind an ASCII firewall. Consider the case of an American Company whose  
top management insists that all company mail passes through its firewall  
in Denver (top management is renowned for giving orders that have no basis  
in the technical realities :-( ). And top management easily forgets that  
they have offices in China, Taiwan and Korea which might want to  
comminucate with each other. A stupid scenario, but stupider things have  
happened.

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 26 Jul 2006 21:53:44 +0100

> requirements even after all, or virtually all, of the Internet

Yes, but once UTF-8 capability has become the norm, newly written software  
will tend to bounce rather than downgrade.

How much current software actually bothers to downgrade to 7bit it it  
finds the following site does not do 8BITMIME?
> incentive to get them fixed and they may not even be noticed.

If they are "not even noticed", then they are working correctly (including  
upgrading at the proper moment). They ain't broke, and all that ...

---

From: John C Klensin <[email-masked]>
Date: Wed, 26 Jul 2006 17:16:09 -0400

--On Wednesday, 26 July, 2006 21:53 +0100 Charles Lindsey
<[email-masked]> wrote:

>> Charles, if there were just a matter of "let them work", I
>> would certainly agree with you.   But the issues associated
>> with downgrading comprise virtually all of the issues that
>> make this effort complex.  It is not an accident that our
>> discussions about parameter formats, location of addresses in
>> and out of brackets, trick headers, and so on all relate to
>> downgrading. Most of the downgrade variants also imply
>> burdens --for recognition of options, headers, or header
>> parameters-- on fully-upgraded systems and will continue to
>> impose those requirements even after all, or virtually all,
>> of the Internet has been upgraded.
> 
> Yes, but once UTF-8 capability has become the norm, newly
> written software  will tend to bounce rather than downgrade.

Exactly.  And earlier software will carry around code paths that
no one is using, which we have repeatedly proven is not a good
thing... since such paths become opportunities for subtle bugs
and security exploits.

> How much current software actually bothers to downgrade to
> 7bit it it  finds the following site does not do 8BITMIME?

Not a whole lot, except for packages that implemented
downgrading early-on (see above).  But "does not bother"
comprises two cases.  One is that the message is bounced, and
the other is that the absence of 8BITMIME is just ignored and
the message forwarded.  The latter still causes problems if the
path actually is not 8bit clean.  But 8BITMIME is so widely
implemented at this point that it is actually hard to find
non-artificial cases in which the option is not offered and the
question is relevant.

And that is exactly the reason I'm wondering how much trouble it
is worth going to in order to provide and support downgrading.

>> result in bouncing, then there will be complaints and
>> incentives to get them fixed.   If they 'just work', then
>> there will be no incentive to get them fixed and they may not
>> even be noticed.
> 
> If they are "not even noticed", then they are working
> correctly (including  upgrading at the proper moment). They
> ain't broke, and all that ...

Not necessarily.  The notion that upgrading will need to occur,
ever, depends on a series of target-site configuration decisions
that I find hard to predict.   For example, in many systems, if
a primary UTF-8 address and an alternate ASCII address are
supported, the most efficient way to support both may not be
tricky "upgrade" 
code but a simple system alias.  But that implies, as others
have pointed out, that replies will use the ASCII address path
and there will be no effective use of UTF-8 addressing until the
entire path is upgraded.   That isn't broken, but it also isn't
what we are trying to get accomplished.

>From your other message...

> But does "must" mean "MUST"? You point out that this same
> downgrade is  needed for MAIL FROM, and it is certainly needed
> if the recipient is  genuinely ASCII-only. So you woould be
> giving the appearance of a rather  arbitrary rule.

I don't know what makes the rule arbitrary.   I think it is
clear that, whether downgrading exists and is supported or not,
there are going to be some rough edges when UTF-8-capable
systems, relays, and users communicate with ASCII-only systems,
relays, and users.  Our job, it seems to me, is to sort through
those cases and balance the pain level associated with those
rough edges with the pain level associated with softening or
eliminating them, keeping in mind where we want to end up.  I
believe that the latter involves carrying around as little
legacy clutter as possible from the transition period... and
making that transition period as short as is sensibly possible.

> Moreover, all these issues of whether to downgrade, and
> whether we need to  specify (or imply) Atomic actually turn
> out to be more important in the  case of From/MAIL
> FROM/Reply-To. You can probably ensure that your mail
> addressed to a UTF-8 recipient follows a "sensible"
> (UTF-8-capable) route.  But you cannot assume the same when
> your recipient replies to you.

I [now] think that is probably correct.

> Moreover, there is still the case of a UTF-capable recipient
> sitting  behind an ASCII firewall. Consider the case of an
> American Company whose  top management insists that all
> company mail passes through its firewall  in Denver (top
> management is renowned for giving orders that have no basis
> in the technical realities :-( ). And top management easily
> forgets that  they have offices in China, Taiwan and Korea
> which might want to  comminucate with each other. A stupid
> scenario, but stupider things have  happened.

Yes, but, in the general case, this has no solution.  Said top
management is as likely to insist that everyone in the company
use the same (UTF-8-incapable) MUA or that everyone use
centrally-assigned ASCII addresses all the time so that the
management (or its data auditors) can easily determine the
recipient or sender of any message.  None of this prevents
anyone from communicating with anyone else; it just prevents
them from using non-ASCII addresses or UTF-8 headers.   We can't
solve the problem.  If anything, we probably help to get it
solved by providing as little support for "try to work around
your stupid management" as possible so that the need to permit
people in branch offices to communicate with addresses in their
own scripts hits the bottom line as clearly as possible.

      john

---

From: Tony Finch <[email-masked]>
Date: Wed, 26 Jul 2006 23:47:26 +0100

On Wed, 26 Jul 2006, John C Klensin wrote:
> downgrading early-on (see above).

Sendmail does.

> But 8BITMIME is so widely implemented at this point that it is actually
> hard to find non-artificial cases in which the option is not offered and
> the question is relevant.

Dumb application-level proxy firewalls are the example I see most
frequently :-(

> is worth going to in order to provide and support downgrading.

Without downgrading you'll never get a critical mass of users.

> > Moreover, all these issues of whether to downgrade, and whether we
> > need to specify (or imply) Atomic actually turn out to be more
> > important in the case of From/MAIL FROM/Reply-To. You can probably
> > ensure that your mail addressed to a UTF-8 recipient follows a
> > "sensible" (UTF-8-capable) route.  But you cannot assume the same when

I agree. However in this thread I wanted to concentrate on recipient
addresses and to point out that we can significantly simplify the SMTP
extension if we follow some decisions through to their logical
conclusions.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FISHER: WEST OR NORTHWEST 4 OR 5 BECOMING VARIABLE 3 OR 4. FAIR. MODERATE OR
GOOD.

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 27 Jul 2006 17:58:11 +0100

> Without downgrading you'll never get a critical mass of users.

Yes, I think that is the real reason to have it, and to have it work.  
Hopefully, when it is seen to work, adoption of UTF8smtp will become more  
common, and then downgrading will become unnecessary. Then there will be  
lots of downgrading code not being used, for people to exploit, as John  
said. But I think that is inevitable, and a price we have to pay.
>> > important in the case of From/MAIL FROM/Reply-To....
>>

Once the downgrade code is in the server, and being used for Reply-To and  
for genuine ASCII-only recipients, then you may as well let it be used  
wherever the next hop is ASCII-only (which should be rare - stupid  
management firewalls maybe), but not worth the trouble of switching off or  
forbidding in in that case.

Whether Atomic is or is not needed as part of the downgrade process is an  
orthogonal issue AFAICS.