Subject: [EAI] utf8 reply in RCPT Commands

From: "Jiankang YAO" <[email-masked]>
Date: Fri, 7 Jan 2011 17:29:22 +0800

----- Original Message ----- 
From: "Dave CROCKER" <[email-masked]>
To: "Apps Discuss" <[email-masked]>; <[email-masked]>; <[email-masked]>; "SM" <[email-masked]>; <[email-masked]>
Sent: Thursday, December 23, 2010 2:04 AM
Subject: [EAI] (private) draft review of: draft-ietf-eai-rfc5336bis-07.txt(v3)

...

>> 3.6.4.  UTF-8 Strings in Replies
>>
>> 3.6.4.1.  RCPT Commands
>>
>> If an SMTP client follows this specification and sends any RCPT commands
>> containing non-ASCII addresses, the SMTP server is permitted to use UTF-8
>> characters in the email address associated with 251 and 551 response codes,
>> and the client MUST be able to accept and process them.
> 
> { I assume that "follows this specification" means that the UTF8SMTPbis option 
> is in force.  If so, then say that, because it is simpler and more precise. But, 
> then, it does require declaring /use/ of the option...
> 
> However the meaning of the text here is odd.  It implies that the server is not 
> permitted to use UTF-8 unless it has already received UTF-8, even though the 
> option is in force.  I suspect that that is not the specification that is 
> intended.  Or, at least, I hope it is not.
> 
> If it actually /is/ what is intended, it means that the enhanced environment is 
> going to have all sorts of /additional/ conditional code, to check on whether a 
> specific context in the state machine is allowed to act in one way or another.
> 
> The simple and more reasonable model is that when the extension is in force, 
> UTF-8 is allowed.  Period.  So... }
> 
>      If an SMTP session is using this extension, then the server is permitted 
> to use UTF-8 characters in the email address associated with 251 and 551 
> reply-codes, and the client MUST be able to accept and process them.
> 
>

There are two kinds of cases when the client receives the utf8smtp ehlo word and builds a smtp session:

1) eai-smtp-client---->eai-smtp-server
2) non-eai-smtp-client---->eai-smtp-server

for the case 1), both the original text and your suggested text above are ok.
for the case 2), the eai-smtp-server must not send the text to non-eai-smtp-client.

the original text intentionally excludes the case 2).

your suggested text seems to still include the case 2).

but for the case 2), the client can not accept the utf8 message since it is non-eai-smtp.

what is your view?

thanks a lot.

Jiankang Yao

>
>
> { I chose "SMTP session" to avoid the more complex discussion of the 
> client/server 'negotiation' about using the extension.  So, here it is either in 
> force or it isn't and if it is in force, it is for client AND server. }
> 
> 
>>      If a given RCPT
>> command does not include a non-ASCII envelope address, the server MUST NOT
>> return a 251 or 551 response containing a non-ASCII mailbox.  Instead, it
>> MUST transform such responses into 250 or 550 responses that do not contain
>> non-ASCII addresses.
> 
> {  See above.  I very strongly disagree with having the complexity this kind of 
> conditional requirement creates.  Or else there is a basic issue involved here 
> that the document does not discuss but needs to, to justify the complexity. }
> 
>

---

From: Dave CROCKER <[email-masked]>
Date: Sat, 08 Jan 2011 13:40:46 -0800

> your suggested text seems to still include the case 2).

Jiankang,

Thank you for the follow-up.  I certainly admit to my having been confused about 
this part of the specification.  Yours and other's comments have clarified this.

To start with a basic point of agreement:

    An EAI server must not send UTF-8 data to a non-EAI client.

Any discussion therefore needs to be based on this absolute requirement, and I 
definitely did not mean to suggest that the rule be violated.

(There is also the requirement that an EAI client must not send UTF-8 data to a 
non-EAI server, but the proposed SMTP extension keyword solves that problem is 
the simple, usual way.)

The current specification is based on the idea that the server can treat a 
client as supporting UTF-8 if the server receives UTF-8 message or address data. 
  This requires inspecting address strings, for VRFY and EXPN, for example. 
(The client is not permitted to send UTF-8 unless is sees that the server can 
support it.)

Hence, the client/server interaction moves from ASCII to UTF-8 as an artifact of 
some address data representation.  This kind of protocol characteristic often 
has limitations and side-effects that are undesirable.

The alternative that has been suggested is to have the client add an explicit 
parameter to declare use of UTF-8.  This is a much more clean and common 
approach for switching "modes" in a protocol exchange.

I think the relatively appropriate approach is the simple and usual method of 
adding a parameter to MAIL, EXPN and VRFY.

d/
-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: John C Klensin <[email-masked]>
Date: Sat, 08 Jan 2011 23:28:20 -0500

--On Saturday, January 08, 2011 13:40 -0800 Dave CROCKER
<[email-masked]> wrote:

>> There are two kinds of cases when the client receives the
>> utf8smtp ehlo word and builds a smtp session:
>...
> Any discussion therefore needs to be based on this absolute
> requirement, and I definitely did not mean to suggest that the
> rule be violated.
>...
> The alternative that has been suggested is to have the client
> add an explicit parameter to declare use of UTF-8.  This is a
> much more clean and common approach for switching "modes" in a
> protocol exchange.
> 
> I think the relatively appropriate approach is the simple and
> usual method of adding a parameter to MAIL, EXPN and VRFY.

In the interest of avoiding confusion and focusing the questions:

* A parameter for VRFY/EXPN was specified in RFC 5336, so the
availability of that parameter is not an proposed addition here.

	There is, IMO, a problem with both the 5336 and 5336bis
	text, which is that it is not clear whether the client
	is required to supply the parameter if the string
	argument contains characters outside the ASCII range.
	I think we are headed toward "yes", even though a test
	on that string is lots less complex than tests on
	message content as would be required for the MAIL
	command if no parameter is required there.

* Neither 5336 nor 5336bis-07 specify a parameter for the MAIL
command.  That issue is already on Joseph's list of questions.

Is that a correct summary?

    john

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 09 Jan 2011 13:36:38 -0800

> Is that a correct summary?

Almost.  However it is reasonable to supply the VRFY/EXPN parameter when the 
string argument contains characters INSIDE the ASCII range, also.

I'll repeat that the working group is showing a tendency to use phrases like 
"outside the ASCII range" or "non-ASCII" as synonyms for Unicode or UTF-8.  Yet 
ASCII is also part of Unicode and the UTF-8 encoding of it.

There are some very specific cases in which it is essential to refer to the 
specific subset UTF-8 characters that are above x07F but I believe these cases 
are far fewer than occur in the working group.

In the example, here, there is no reason to prohibit use of the parameter, even 
when the string argument is "ASCII".  For one thing, ASCII is part of Unicode.

For another, it signals that the client can accept responses containing UTF-8. 
Such responses might, of course, occur for string arguments that are all-ASCII.

I suspect everyone in the working group already understands these points.  So my 
reason for flagging the issue is that this is an issue that is easily confused 
and the working group needs to be particularly careful in the way it discusses 
and documents it.

d/
-- 

   Dave Crocker

---

From: John C Klensin <[email-masked]>
Date: Sun, 09 Jan 2011 18:46:56 -0500

--On Sunday, January 09, 2011 13:36 -0800 Dave CROCKER
<[email-masked]> wrote:

> Almost.  However it is reasonable to supply the VRFY/EXPN
> parameter when the string argument contains characters INSIDE
> the ASCII range, also.

That is what I said, I think.   I agree with you that it is
reasonable.  Whether it is necessary or not is, IMO, an
unresolved question although I'm personally strongly leaning in
the direction of permitting the parameter always and requiring
if either the string argument to the comment contains Unicode
characters outside the ASCII range (see below) or if such
characters are permitted on replies.

> I'll repeat that the working group is showing a tendency to
> use phrases like "outside the ASCII range" or "non-ASCII" as
> synonyms for Unicode or UTF-8.  Yet ASCII is also part of
> Unicode and the UTF-8 encoding of it.

Careful, please.   I agree with you, at least in retrospect,
that "non-ASCII" is sloppy terminology and that we should get
rid of it even though I suggest that the meaning is clear to
almost every person on earth who works regularly with Unicode or
internationalization more generally.

However, especially since the EAI specs are, I think, already
quite clear that no CCS other than Unicode is permitted on the
wire, nor is any encoding of Unicode other than UTF-8, "outside
the ASCII range" or "outside the ASCII repertoire" are
completely and very precisely well-defined and, for all
practical purposes, synonymous.  The difference between the two
is only relevant if a constraint to Unicode (with or without a
UTF-8 encoding requirement) has not already been specified:
"ASCII repertoire" is still well-defined but "ASCII range" might
not so well-defined on another code space (although, in
practice, unless one is using either a very old CCS (e.g., BCD)
or a non-ISO CCS like EBDCIC, "ASCII range" is still
well-defined.   The only way to be more precise about what is
intended would be to say something like "Unicode code points
outside the range U+0000 through U+007F inclusive", but, in
addition to being pedantic, that is actually less clear about
what is intended than terminology based on ASCII, at least to
the less-sophisticated reader.

> There are some very specific cases in which it is essential to
> refer to the specific subset UTF-8 characters that are above
> x07F but I believe these cases are far fewer than occur in the
> working group.

Maybe I still don't understand the point you are trying to make,
but, as you would be the first to point out in other contexts,
there is an installed base out there and the commands and
headers of that installed base are defined entirely in terms of
"ASCII characters".  Exceptions arise for message content: you
will recall that one of the, if not the, primary original goal
for what became MIME was internationalization of textual
content.  It is also worth noting that MIME "charset" parameters
deliberately and explicitly conflate coded character sets with
their encodings.  We claimed for many years that was a strength
to reduce confusion so, while I agree with you that it may be
time to untangle the two, relative to IETF precedents in the
email area, we are swimming upstream (together).   Faulting the
EAI WG and is document for creating this situation is, IMO, a
little bit unreasonable.

More broadly and again unless I misunderstand, I think you are
contradicting your own arguments.

Because of that installed base, the WG has been extremely
careful to avoid requiring EAI behavior or parameters when
whatever both client and server are intending to do is strictly
conformant with 5321/5322.  There are some tricky balances there
and we may not have gotten them right.   But it seems to me
that, if someone sends, e.g., VRFY using exactly the syntax
specified by 5321 and its predecessors, they should be able to
expect to get back exactly what 5321 specifies... and that is
NVT ASCII in and NVT ASCII out.  Similarly, _especially_ if we
provide a parameter to the MAIL command to indicate that EAI
features are needed, it may be ok for a client to supply that
parameter and send NVT ASCII only, but, if the client sends NVT
ASCII without a parameter, both client and server should expect
to behave exactly as 5321 (and any other extensions) specify.

Another example of this is message/rfc822.  It is clearly
specified (Section 5.2.2 of RFC 2045 and probably elsewhere) as
supporting all-ASCII headers in the embedded messages.  It has
no provision for a charset parameter or other indication that
headers in something other than ASCII are being transmitted.
Rejection of the idea of message/global is a threat to the
installed base because it would force changing the semantics of
a well-defined and well-established concept (and
content-type)... unless, of course, one is willing to say that
no EAI-conformant message that does not conform to the
restrictions of 2046 (e.g., one that contains characters outside
the ASCII repertoire in UTF-8 encoding rather than RFC 2047
("encoded word") encoding) may ever be encapsulated for
forwarding or other purposes.

> In the example, here, there is no reason to prohibit use of
> the parameter, even when the string argument is "ASCII".  For
> one thing, ASCII is part of Unicode.

Sure.  There is probably no reason to prohibit it and no reason
why an EAI-compliant shouldn't include it, other than giving the
server a little bit more information... _unless_ the client is
sending ASCII and unable to accept a reply that contains
anything but ASCII.

I distinctly remember MTAs that, because various lookup systems
were required that were not used elsewhere in their systems,
branched off into completely separate code modules if VRFY or
EXPN were received, pushing down or otherwise remembering
session state if EHLO had been received first.    I don't know
if such systems are still in active use but, if they are, one
could easily imagine a different implementation schedule for
VRFY/EXPN than for actual message transmission.   I assume we
would not want to delay deployment of i18n mail for such an
implementation until VRFY/EXPN caught up, especially for vendors
whose customer experience was that no one enabled either
operationally anyway.
 
> For another, it signals that the client can accept responses
> containing UTF-8. Such responses might, of course, occur for
> string arguments that are all-ASCII.

And that was the case where 5336 (and the current text of
5336bis) absolutely require the parameter.  So I don't know who
you think you have to persuade.

> I suspect everyone in the working group already understands
> these points.  So my reason for flagging the issue is that
> this is an issue that is easily confused and the working group
> needs to be particularly careful in the way it discusses and
> documents it.

No disagreement there.

    john