Subject: [EAI] ATOMIC parameter

From: Chris Newman <[email-masked]>
Date: Mon, 20 Mar 2006 02:22:01 -0600

Although I'm sympathetic to the motivations behind the ATOMIC parameter, I 
believe it is an incorrect bias of protocol purity over operational robustness.

As a matter of operational robustness it is almost always better to move 
questionable email material forward than to bounce it.  If the recipient gets 
something that appears to be corrupt, they will contact the sender to resend 
the material.  If the message is important, the recipient will probably have 
out-of-band mechanisms to contact the sender that are more likely to work than 
the MAIL FROM address (e.g. a cell phone number, etc).

On the flip side, it is quite easy to mis-configure the MAIL FROM address and 
the problem probably won't be noticed unless the person who did so gets a 
complaint from a colleague which will only happen if the message moves forward 
and the  colleague's attempt to reply fails.  We also have widely deployed 
systems that do inappropriate things with MAIL FROM that can cause loops or 
other badness when a bounce happens.  And in the new world of Spam blowback, 
many people simply ignore almost all bounces (myself included, unfortunately).

This is very unfortunate for those of us who care a great deal about protocol 
purity.  It's really painful to pass non-interoperable junk forward but the 
outcome for the users involved is more likely to be better, so it's usually the 
right thing to do.

Which brings us to the ATOMIC parameter.  We all like the rule which forbids 
local part interpretation by remote systems and I respect the ATOMIC parameter 
approach as a clever way to preserve that rule.  Unfortunately, ATOMIC=no 
leaves MTAs in an untenable situation: either ignore the standard, downconvert 
and pass forward or bounce.  Unfortunately, robustness suggests the right 
answer is to ignore the standard.  Were I implementing a system, I'd probably 
ignore the standard for this case by default and include an option to follow 
the standard.  But having "do the right thing for users" and "comply with the 
standard" as two separate settings is not something I like (how often do people 
use the "strict ANSI" option on their C compiler?).

Use of a UTF-8 local part should implicitly grant permission to use the 
7-bit-encoded alias of that local part.  This is not significantly different 
from the rule that [email-masked] implies the existence of 
[email-masked] and the converse is true.  Furthermore, if 
someone wants to use structured UTF-8 addresses for special processing and do 
so while interacting with a 7-bit-only system, I believe it's reasonable to 
insist they upgrade their special processing system to remove the 7-bit 
encoding prior to processing.  And as much as I like sub-addresses, their use 
is simply not widespread enough to merit additional design complexity simply to 
avoid requiring up-conversion at that point in the system.

Finally, down-conversion is a non-negotiable feature for some implementers 
(myself included).  I have to be able to down-convert the UTF-8 local parts in 
the headers, even if an alternate address is unavailable (and it doesn't change 
things if the spec requires an alternate address be available, as we still have 
to deal with the non-compliant case in quality software).  So I suggest 
ATOMIC=yes should be implicit from the use of UTF-8 and thus not necessary to 
include in the protocol specification.

                - Chris

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Tony Hansen <[email-masked]>
Date: Mon, 20 Mar 2006 11:18:46 -0500

+1

Chris Newman wrote:
> Although I'm sympathetic to the motivations behind the ATOMIC parameter,
> I believe it is an incorrect bias of protocol purity over operational
> robustness.
> ...
> 
> 7-bit-encoded alias of that local part.  ...
> 
> Finally, down-conversion is a non-negotiable feature for some
> implementers (myself included).  I have to be able to down-convert the
> UTF-8 local parts in the headers, even if an alternate address is
> unavailable (and it doesn't change things if the spec requires an
> alternate address be available, as we still have to deal with the
> non-compliant case in quality software).  So I suggest ATOMIC=yes should
> be implicit from the use of UTF-8 and thus not necessary to include in
> the protocol specification.

---

From: Yangwoo Ko <[email-masked]>
Date: Tue, 21 Mar 2006 01:19:55 +0900

Dear Chris Newman,

We should consider how much standards can constraint real 
implementations. If we can safely assume that all implementations will 
execute some procedures in the exactly same order that authors of a 
specification had in their minds, it might  be quite wonderful. However, 
as you pointed out, developers of quality software will intentionally 
deviate from standard. (I would like to call it deviation rather than 
ignorance.) And that will invalidate the previous assumption. In that 
sense, it is painful to accept this though, standard is just a median 
point from which implementations can deviate from and provides the least 
common ground when two different deviations want to communication with 
each other.

Chris Newman wrote:
> Which brings us to the ATOMIC parameter.  We all like the rule which 
> forbids local part interpretation by remote systems and I respect the 
> ATOMIC parameter approach as a clever way to preserve that rule.  
> Unfortunately, ATOMIC=no leaves MTAs in an untenable situation: either 
> ignore the standard, downconvert and pass forward or bounce.  
> Unfortunately, robustness suggests the right answer is to ignore the 
> standard.  Were I implementing a system, I'd probably ignore the 
> standard for this case by default and include an option to follow the 
> standard.  But having "do the right thing for users" and "comply with 
> the standard" as two separate settings is not something I like (how 
> often do people use the "strict ANSI" option on their C compiler?).
>
> Use of a UTF-8 local part should implicitly grant permission to use 
> the 7-bit-encoded alias of that local part.  This is not significantly 
> different from the rule that [email-masked] implies the 
> existence of [email-masked] and the converse is true.  
> Furthermore, if someone wants to use structured UTF-8 addresses for 
> special processing and do so while interacting with a 7-bit-only 
> system, I believe it's reasonable to insist they upgrade their special 
> processing system to remove the 7-bit encoding prior to processing.  
> And as much as I like sub-addresses, their use is simply not 
> widespread enough to merit additional design complexity simply to

---

From: Harald Alvestrand <[email-masked]>
Date: Mon, 20 Mar 2006 10:26:17 -0600

Yangwoo Ko wrote:
> specification had in their minds, it might  be quite wonderful. 
> However, as you pointed out, developers of quality software will 
> intentionally deviate from standard. (I would like to call it 
> deviation rather than ignorance.) And that will invalidate the 
> previous assumption. In that sense, it is painful to accept this 
> though, standard is just a median point from which implementations can 
> deviate from and provides the least common ground when two different 
> deviations want communication with each other. 
actually the standard does slightly more than this; if the standard 
"works", it gives assurance that all parties following the standard are 
able to communicate in providing the given function.
thus, when someone deviates from the standard, they take on the 
responsibility of proving that their deviation does not harm 
interoperability; if interoperability is harmed, it is the 
responsibility of the implementation deviating from the standard.

People remember this, and may consider it when doing buying decisions 
later - the hope is that people's buying decisions will eventually cause 
the "deviants" to converge towards standard adherence. But of course, 
having that hope requires that one is an optimist :-)

Our job as standards writers is to get to the first base: Specifying a 
standard that allows useful communication between conformant entities.
Unless we have that, it's useless to ask people to conform to it.

                       Harald

---

From: Chris Newman <[email-masked]>
Date: Mon, 20 Mar 2006 14:50:14 -0600

Yangwoo Ko wrote on 3/21/06 1:19 +0900:
> We should consider how much standards can constraint real implementations. If
> we can safely assume that all implementations will execute some procedures in
> the exactly same order that authors of a specification had in their minds, it
> might  be quite wonderful. However, as you pointed out, developers of quality
> software will intentionally deviate from standard. (I would like to call it
> deviation rather than ignorance.) And that will invalidate the previous
> assumption. In that sense, it is painful to accept this though, standard is
> just a median point from which implementations can deviate from and provides
> the least common ground when two different deviations want to communication
> with each other.

My argument was worded a bit more subtle than it should have been.  If a 
responsible implementer feels it is necessary to willfully violate the standard 
to provide better service to end users, that suggests there is a technical flaw 
in the standard.

As Harald said, the only mandatory feature of a standard is that compliant 
implementations will interoperate to provide a useful service.  But a standard 
is more successful if it is designed in such a way that more implementers will 
choose to comply by default.

                - Chris

---

From: Yangwoo Ko <[email-masked]>
Date: Tue, 21 Mar 2006 06:20:01 +0900

The first impression when I heard the atomic idea is that most 
implementations and their users will tag email addresses as atomic=yes 
by default. If this is the case, atomic=no is very clear indication that 
local part should not processed at intermediate servers. Unless 
alternative address is specified, not trying to down-convert seems far 
more robust than trying to down-convert while expecting that final 
delivery MTA will up-convert before doing anything.

Chris Newman wrote:
>> implementations. If
> My argument was worded a bit more subtle than it should have been.  If 
> a responsible implementer feels it is necessary to willfully violate 
> the standard to provide better service to end users, that suggests 
> there is a technical flaw in the standard.
>
> As Harald said, the only mandatory feature of a standard is that 
> compliant implementations will interoperate to provide a useful 
> service.  But a standard is more successful if it is designed in such 
> a way that more implementers will choose to comply by default.
>
>                - Chris
>
>
>

---

From: John C Klensin <[email-masked]>
Date: Mon, 20 Mar 2006 18:23:45 -0500

Chris,

I've read your note several times to see if I can figure out 
where we disagree.    I think it is the nature of the things 
that are encoded in local parts and, ultimately, the reason why 
that "no local-part interpretation by senders or relays" rule is 
there.

Given that rule, if one is sending UTF-8 address material to a 
current, RFC821-compliant-only final receiver (the worst case as 
far as I know), I think that one needs to know exactly how that 
receiver interprets and decodes addresses, and the order in 
which it does so, in order to downgrade the addresses. 
Certainly downgrading other header, or even body, material is 
not problematic.   This many not be problematic for POP3 or IMAP 
because any mailbox routing or other transformations that will 
be applied by the delivery SMTP server will presumably be 
applied before anything gets to the mail store.

But, for in-transit downgrading in the general case, I think 
that any downgrade/recoding mechanism we can devise is a shot in 
the dark, one that risks a sender eventually getting a bounce 
message about an address he or she has never seen before. 
Personally, I think _that_ is the worst possible case: the 
message gets bounced rather than delivered, and the error 
message is unintelligible. If the message is going to bounce, 
better to have it bounce from a point at which a plausible error 
report can be generated.

Now, a way to read my analysis and yours together is that we 
need to drastically narrow the cases for which ATOMIC is 
relevant.  If, for example, we have a message with UTF-8 header 
material but all-ASCII addresses, then one clearly can, and 
should, downgrade the headers (using encoded words or the 
equivalent).  (I think this makes the case for no UTF-8 in 
header _names_ even stronger.)   It is only when the addresses 
(actually the address local-parts because we can "downgrade" 
domain names to punycode) themselves are in UTF-8 and there is 
no next-hop support are we in serious trouble.  And it was that 
case for which ATOMIC was designed.

That still leaves an interesting case in which the message can 
be accurately delivered because the local parts in all of the 
forward-pointing addresses are ASCII but the backward-pointing 
address(es) are not.  And that is the old argument about whether 
a message should be delivered even if it is known that it cannot 
be replied to.  You would, I think, say "yes", as would I.  The 
antispam community generally says "no".  And I hope that we can 
avoid having to resolve it here.

     john

--On Monday, March 20, 2006 02:22 -0600 Chris Newman 
<[email-masked]> wrote:

> Although I'm sympathetic to the motivations behind the ATOMIC
> parameter, I believe it is an incorrect bias of protocol
> purity over operational robustness.
>
> As a matter of operational robustness it is almost always
> better to move questionable email material forward than to
> bounce it.  If the recipient gets something that appears to be
> corrupt, they will contact the sender to resend the material.
> If the message is important, the recipient will probably have
> out-of-band mechanisms to contact the sender that are more
> likely to work than the MAIL FROM address (e.g. a cell phone
> number, etc).
>
> On the flip side, it is quite easy to mis-configure the MAIL
> FROM address and the problem probably won't be noticed unless
> the person who did so gets a complaint from a colleague which
> will only happen if the message moves forward and the
> colleague's attempt to reply fails.  We also have widely
> deployed systems that do inappropriate things with MAIL FROM
> that can cause loops or other badness when a bounce happens.
> And in the new world of Spam blowback, many people simply
> ignore almost all bounces (myself included, unfortunately).
>
> This is very unfortunate for those of us who care a great deal
> about protocol purity.  It's really painful to pass
> non-interoperable junk forward but the outcome for the users
> involved is more likely to be better, so it's usually the
> right thing to do.
>
> Which brings us to the ATOMIC parameter.  We all like the rule
> which forbids local part interpretation by remote systems and
> I respect the ATOMIC parameter approach as a clever way to
> preserve that rule.  Unfortunately, ATOMIC=no leaves MTAs in
> an untenable situation: either ignore the standard,
> downconvert and pass forward or bounce.  Unfortunately,
> robustness suggests the right answer is to ignore the
> standard.  Were I implementing a system, I'd probably ignore
> the standard for this case by default and include an option to
> follow the standard.  But having "do the right thing for
> users" and "comply with the standard" as two separate settings
> is not something I like (how often do people use the "strict
> ANSI" option on their C compiler?).
>
> Use of a UTF-8 local part should implicitly grant permission
> to use the 7-bit-encoded alias of that local part.  This is
> not significantly different from the rule that
> [email-masked] implies the existence of
> [email-masked] and the converse is true.
> Furthermore, if someone wants to use structured UTF-8
> addresses for special processing and do so while interacting
> with a 7-bit-only system, I believe it's reasonable to insist
> they upgrade their special processing system to remove the
> 7-bit encoding prior to processing.  And as much as I like
> sub-addresses, their use is simply not widespread enough to
> merit additional design complexity simply to avoid requiring
> up-conversion at that point in the system.
>
> implementers (myself included).  I have to be able to
> down-convert the UTF-8 local parts in the headers, even if an
> alternate address is unavailable (and it doesn't change things
> if the spec requires an alternate address be available, as we
> still have to deal with the non-compliant case in quality
> software).  So I suggest ATOMIC=yes should be implicit from
> the use of UTF-8 and thus not necessary to include in the
> protocol specification.
>
>                 - Chris
>
>

---

From: Chris Newman <[email-masked]>
Date: Wed, 22 Mar 2006 01:13:42 -0600

I need to break the contentious use case into a two sub-cases (neither case has 
an alternate address):

1. MAIL FROM:<ascii@ascii>
   RCPT TO:<ascii@ascii>
   ...
   From: name <utf8@utf8>

2. MAIL FROM:<utf8@utf8>
   RCPT TO:<ascii@ascii>
   ...
   From: name <utf8@utf8>

If I understand your position, it is always acceptable to pass case 1 forward, 
even if the headers must be down-converted.  And I observe that if the 
recipient replies to this message after down-conversion, the RCPT TO in the 
reply will use the down-converted address from the From header.

Furthermore, in case 2, you believe it is better to bounce rather than 
downgrade because once the envelope is downgraded any later bounce will be more 
confusing to the originator.

Do I correctly understand your position?

My position is that if we define a 7-bit down-conversion for utf-8 local parts, 
there is no way to prevent the originator's mail system from seeing that 7-bit 
down-conversion in some cases.  As a result, it will at least be a best 
practice (if not a SHOULD or MUST) for the mail system which supports utf-8 
local parts to also support the 7-bit down-conversion.  As long as the ACE is 
easy to recognize and 100% reversible, this seems like a simple problem for the 
mail system to solve.

And given that, the principle that we should bounce as a near last resort 
trumps any concern that the down-converted return address might cause problems. 
(And I'd like to experiment with putting the xtext-encoded UTF-8 original 
recipient in the ORCPT parameter to mitigate your concern further).

So I view "ATOMIC=N" as an explicit statement that the originator is 
deliberately violating best practices and has no wish to communicate with 
anyone who can't preserve the address in its pure form.  If we want to minimize 
partitioning problems, then I believe we should not make the protocol more 
complex simply to cater to this case.  Now as I'm an engineer rather than a 
politician, I'm willing to accept that I may be ignorant of political realities 
in this case and would be willing to compromise if ATOMIC had a clearer name 
(e.g. BOUNCE-BEFORE-7BIT-GATEWAY=Y).  But I do not see a valid engineering 
argument to ever do anything other than an implicit "ATOMIC=Y".

I also see no problem with giving systems that embed processing instructions in 
the local part a simple choice: either use 7-bit only, or support ACE upgrade 
as well as UTF-8.  If such systems find our present local-part ACE too complex, 
perhaps we should choose a simpler one (e.g., base 32 with a prefix).

                - Chris

John C Klensin wrote on 3/20/06 18:23 -0500:

> Chris,
>
> I've read your note several times to see if I can figure out where we
> disagree.    I think it is the nature of the things that are encoded in local
> parts and, ultimately, the reason why that "no local-part interpretation by
> senders or relays" rule is there.
>
> Given that rule, if one is sending UTF-8 address material to a current,
> RFC821-compliant-only final receiver (the worst case as far as I know), I
> think that one needs to know exactly how that receiver interprets and decodes
> addresses, and the order in which it does so, in order to downgrade the
> addresses. Certainly downgrading other header, or even body, material is not
> problematic.   This many not be problematic for POP3 or IMAP because any
> mailbox routing or other transformations that will be applied by the delivery
> SMTP server will presumably be applied before anything gets to the mail store.
>
> But, for in-transit downgrading in the general case, I think that any
> downgrade/recoding mechanism we can devise is a shot in the dark, one that
> risks a sender eventually getting a bounce message about an address he or she
> has never seen before. Personally, I think _that_ is the worst possible case:
> the message gets bounced rather than delivered, and the error message is
> unintelligible. If the message is going to bounce, better to have it bounce
> from a point at which a plausible error report can be generated.
>
> Now, a way to read my analysis and yours together is that we need to
> drastically narrow the cases for which ATOMIC is relevant.  If, for example,
> we have a message with UTF-8 header material but all-ASCII addresses, then
> one clearly can, and should, downgrade the headers (using encoded words or
> the equivalent).  (I think this makes the case for no UTF-8 in header _names_
> even stronger.)   It is only when the addresses (actually the address
> local-parts because we can "downgrade" domain names to punycode) themselves
> are in UTF-8 and there is no next-hop support are we in serious trouble.  And
> it was that case for which ATOMIC was designed.
>
> That still leaves an interesting case in which the message can be accurately
> delivered because the local parts in all of the forward-pointing addresses
> are ASCII but the backward-pointing address(es) are not.  And that is the old
> argument about whether a message should be delivered even if it is known that
> it cannot be replied to.  You would, I think, say "yes", as would I.  The
> antispam community generally says "no".  And I hope that we can avoid having
> to resolve it here.
>
>      john
>
>
> --On Monday, March 20, 2006 02:22 -0600 Chris Newman <[email-masked]>
> wrote:
>