Subject: List of open issues with Sieve reject draft (draft-ietf-sieve-refuse-reject-02.txt)

From: Alexey Melnikov <[email-masked]>
Date: Sun, 09 Jul 2006 17:56:28 +0100

1). “SHOULD be incompatible with other actions” is too strong, for 
example there are good reasons to do reject+fileinto.

I tend to agree that reject should be made compatible with other actions 
(except for vacation), so I would like to at least downgrade SHOULD to MAY.

2). Non ASCII text in rejection string - should it cause creation of 
DSN/MDN, runtime error or stripping of non-ASCII content?
Should we add a tagged argument to control this?
Or maybe we need another capability to enable UTF-8 clean rejection?

I am leaning toward having an extra capability to enable UTF-8 rejection 
text over SMTP/LMTP protocol.

3). Arnt has requested to allow for reject+redirect to be treated as 
just reject. I am not sure I like that. Opinions?

Comments?

Alexey

---

From: "Nigel Swinson" <[email-masked]>
Date: Mon, 10 Jul 2006 12:04:59 +0100

> (except for vacation), so I would like to at least downgrade SHOULD to MAY.

I agree, I deliberately ignored the SHOULD anyway

> just reject. I am not sure I like that. Opinions?

I don't like that either, I can't see any reason why we'd want that.

Nigel

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Mon, 10 Jul 2006 14:25:52 +0200

> example there are good reasons to do reject+fileinto.

please state such reasons.  I do not think it is a good idea to allow
the recipient system to outright lie about the success of the delivery.

> Or maybe we need another capability to enable UTF-8 clean rejection?

that capability needs to be added to SMTP, right?

> text over SMTP/LMTP protocol.

fine with me.  I don't think Sieve needs to address this point
normatively.  the implementation can do its best within the limits of
the current standards.

> just reject. I am not sure I like that. Opinions?

I don't like it either.  silently ignoring actions is bad.
-- 
Kjetil T.

---

From: Ned Freed <[email-masked]>
Date: Mon, 10 Jul 2006 06:43:06 -0700 (PDT)

> the recipient system to outright lie about the success of the delivery.

I have to agree with Kjetil here. Of course a user could forge a failure
DSN, but that's no reason to make it easy to lie about the status of email.
Joe-jobs have caused enough damage to the way DSNs work already; we really
should do more.

> that capability needs to be added to SMTP, right?

I would assume UTF-8 error responses in SMTP are somewhere in the EAI
to-do list, but we shouldn't have a dependency on such work here..

> normatively.  the implementation can do its best within the limits of

Agreed.

> I don't like it either.  silently ignoring actions is bad.

Agreed as well.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 10 Jul 2006 00:02:25 +0200

> just reject. I am not sure I like that. Opinions?

My reason:

If an implementation takes some trouble (e.g. using the postfix content 
filter interface), it can evaluate some clauses of a sieve script based 
only on the information in MAIL FROM and/or RCPT TO. After RCPT TO, 
such an implementation may know that one clause in the script directs 
the interpreter to reject the message, but it does not know whether 
other actions are to be run, because other tests cannot yet be 
evaluated.

As I read the current draft, such an interpreter is not allowed to 
reject the RCPT TO command, because it does not yet know whether the 
script is valid: The script may also try to execute some action which 
is incompatible with reject.

Arnt

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 11 Jul 2006 13:27:08 +0200

> is incompatible with reject.

I don't think this is needed.

my take on it: a script which is run after RCPT TO needs to give a
unambiguous answer based on the information available.  if the script
only does envelope tests, takes an action and then stops, that action is
valid.  if a test which accesses headers or other unavailable
information is encountered before the explicit "stop", the result of the
script is "don't know", and the implementation must accept the RCPT TO.
after DATA, the script can be rerun with all tests.

to illustrate with an example:

  MAIL FROM:<>
  RCPT TO:<[email-masked]>

if my script is:

  require ["envelope", "subaddress"];
  if envelope :detail "To" "ietf" {
    fileinto "INBOX.ietf"; stop;
  } elsif not envelope :localpart "To" "kjetilho" {
    reject "Unknown subaddress"; stop;
  }
  if header :contains "Subject" "[birdwatch]" {
    fileinto "INBOX.lists.birdwatch"; stop;
  }
  ...

the first test yields true, the execution stops, and the fileinto means
accept the RCPT TO.  for RCPT TO:<[email-masked]>, reject
is taken and RCPT TO is denied.  for any other RCPT TO, execution goes
on to the header test, which can't be performed until after DATA.
therefore the result is "don't know" and the RCPT TO must be accepted.

-- 
Kjetil T.

---

From: "Nigel Swinson" <[email-masked]>
Date: Tue, 11 Jul 2006 13:07:53 +0100

> the recipient system to outright lie about the success of the delivery.

To create a record of what has been rejected so that the accuracy of the test in the script can be verified, or to store for forward to an abuse@ mailbox maintained by the sys admin.

Nigel

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 11 Jul 2006 14:33:07 +0200

On Tue, 2006-07-11 at 13:07 +0100, Nigel Swinson wrote:
> To create a record of what has been rejected so that the accuracy of
> the test in the script can be verified,

there are no logging actions in Sieve, but most implementations can
provide such logs without it being triggered by explicit script actions.

(a hypothetical logging action should not be incompatible with any other
actions, IMO.)

> or to store for forward to an abuse@ mailbox maintained by the sys
> admin.

I don't think this is sufficient reason to allow an e-mail system to
misrepresent the delivery status.
-- 
Kjetil T.

---

From: "Nigel Swinson" <[email-masked]>
Date: Tue, 11 Jul 2006 13:59:29 +0100

> provide such logs without it being triggered by explicit script actions.

Well our implementation doesn't provide any end user accesible operational logs, so using fileinto seems like an excellent low cost implementation.

Besides, what is an end user going to want to see in those logs?  They will want access to the message data, and a list of what was rejected, which is exactly what you get with fileinto+reject.  Our product has run that way for years and the only complaints I've ever had about it is from you guys ;o)

> misrepresent the delivery status.

I'm quite happy to interpret reject as "I never want to see any mail like that ever again".  If the server carries on delivering/processing that message then I think that's up to the mail server.

Nigel

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 11 Jul 2006 15:59:51 +0200

On Tue, 2006-07-11 at 13:59 +0100, Nigel Swinson wrote:
> Well our implementation doesn't provide any end user accesible
> operational logs, so using fileinto seems like an excellent low cost
> implementation.

what do you gain by doing reject?  no spammer ever washes their lists
based on response codes, anyway.  (*perhaps* after negative response to
RCPT TO, for DATA, no way.)

> I'm quite happy to interpret reject as "I never want to see any mail
> like that ever again".  If the server carries on delivering/processing
> that message then I think that's up to the mail server.

RFC 2821, section 4.2.5:

   When an SMTP server returns a permanent error status (5yz) code after
   the DATA command is completed with <CRLF>.<CRLF>, it MUST NOT make
   any subsequent attempt to deliver that message. 

I don't think that's open to interpretation.
-- 
Kjetil T.

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 11 Jul 2006 11:25:41 -0400

> is incompatible with reject.

This looks like that old question: does the prohibition on
"reject+redirect" (or any other combination of actions defined as
incompatible) produce an invalid script, or an invalid action?
Per RFC3028 this is an implementation choice.  With the latter
choice, a script:

    reject "go away";    	# is executed
    redirect "mem@somewhere";   # gives an execution error

and if they are reversed, the redirect is executed but not the reject.
If an implementation chooses the former (wait until the end to execute
actions, doing none of them if there are conflicts) then you might have
to address the question you raise, but I don't think that should be in
the spec.

As for reject + fileinto: the one thing I could see it useful for has
already been mentioned, and that is administrative logging in order to
be able to debug script design decisions.  It's tempting to just say
"solve that problem another way, outside of sieve" but I can certainly
see how using fileinto is convenient for this.  I suppose an
implementation could have an vendor-specific capability to enable this,
not condoned by the spec.

Yours,
mm

---

From: Dave Cridland <[email-masked]>
Date: Tue, 11 Jul 2006 16:28:27 +0100

On Tue Jul 11 12:27:08 2006, Kjetil Torgrim Homme wrote:
> > > 3). Arnt has requested to allow for reject+redirect to be 
> treated as > > just reject. I am not sure I like that. Opinions?
> > > My reason:
> > > If an implementation takes some trouble (e.g. using the postfix 
> content > filter interface), it can evaluate some clauses of a 
> sieve script based > only on the information in MAIL FROM and/or 
> RCPT TO. After RCPT TO, > such an implementation may know that one 
> clause in the script directs > the interpreter to reject the 
> message, but it does not know whether > other actions are to be 
> run, because other tests cannot yet be > evaluated.
> >
> > As I read the current draft, such an interpreter is not allowed 
> to > reject the RCPT TO command, because it does not yet know 
> whether the > script is valid: The script may also try to execute 
> some action which > is incompatible with reject.
> 

You may recall that I do not, and have never, implemented Sieve. I'm 
just an ex-sysadmin with a ManageSieve client implementation and a 
handful of Sieve scripts I use a lot, so you can feel a little 
justified in disagreeing with me, given that I merely keep half an 
eye on the discussion, and I admit to not reading the drafts much.

But I think this is really needed.

Otherwise, you're telling users to write their Sieve filters in a 
manner which is beneficial for optimization on the server. This ain't 
gonna happen, as they say.

Basically, if a sysadmin wants to reject the maximal amount of 
messages before accepting the DATA - this being a sensible and 
desirable thing to do - Arnt's suggestion allows the sysadmin to 
select an implementation and/or suitable options which allows this to 
happen, in the case where the user or sieve generator has chosen to 
reject the message. All the tweaks required to change the server 
behaviour are purely on the server-side. And you argue in another 
message that rejection at RCPT stage is more likely to stop spammers, 
too, so I'm sure you agree that this is the desirable state.

Your suggestion means that for best performance, a very careful 
choice of Sieve generator and server implementation is required. Bear 
in mind that one of the more common Sieve generators is the user - I 
know plenty of sysadmins who'd argue for more choice in their users, 
but I can't see this happening. Whilst your proposal is certainly 
bending the rules less, I don't think it's providing any benefit 
aside from purism.

To put it another way, I'm perfectly happy for some actions to have 
an implicit stop as a side-effect. In fact, until I skimmed this 
thread, I sort of assumed that reject *did* have an implicit stop - 
maybe it just sounds like a final decision.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 11 Jul 2006 18:08:47 +0200

> gonna happen, as they say.

possibly not, but how do you suggest the server should handle it?
flatly ignore all tests which require unavailable information?  if so,
if a user writes a script with lots of explicit tests and an
unconditional "reject" at the end, you get disaster.

one other option is to make it explicit with different scripts for the
two situations.

> but I can't see this happening.

I just checked the numbers at our site, and of our 62804 users, 3156
have installed Sieve filters.  31 of these are handwritten, the rest are
generated by avelsieve (Squirrelmail plugin).  I convinced it's possible
to write a good UI which promotes putting envelope tests at the top.

> Whilst your proposal is certainly 

I don't see it as purism as much as correctness and determinism :-)

> maybe it just sounds like a final decision.

I don't disagree with this, a quite common error for our users is to
omit the "stop" after the "fileinto", so they get several copies of some
messages.  it's much too late to change fileinto/keep/etc now, though.
-- 
Kjetil T.

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 11 Jul 2006 12:38:44 -0400

> one other option is to make it explicit with different scripts for the

Or..  a good use for that "mode" variable, so the script can tell the
circumstance under which it is being executed.

Or.. a draft specifying how 'header' et al (anything requiring the
message to be present) should behave if the message isn't yet there.

Or.. a test that will see if the message is available for examination.
(e.g. via some read-only variable.)

But really, a problem is that support for RCPT-TO (and other) scripting
probably needs more fleshing out anyway, since RCPT-TO wants to see more
states and more information returned from a script than simply "accept"
or "reject."  To me, this suggests having a separate scripts, or at
least script fragments, for each mode.

mm

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 11 Jul 2006 19:01:10 +0200

> circumstance under which it is being executed.

I think this is less likely to be made usable in UI editors than my
suggestion (which just requires special ordering of statements, and as
such doesn't _require_ UI support at all).

> message to be present) should behave if the message isn't yet there.

yes, I think this needs to made explicit somewhere.

> (e.g. via some read-only variable.)

this is just a slight variation of your #1, isn't it?

> least script fragments, for each mode.

what more information?  you want scripts to be able to return 4xx as
well as 2xx and 5xx?

-- 
Kjetil T.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 11 Jul 2006 18:58:08 +0200

Mark E. Mallett writes:
> But really, a problem is that support for RCPT-TO (and other) 
> scripting probably needs more fleshing out anyway, since RCPT-TO 
> wants to see more states and more information returned from a script 
> than simply "accept" or "reject." To me, this suggests having a 
> separate scripts, or at least script fragments, for each mode.

Perhaps I'd better describe the specific idea that led me to this.

I want my code to send fewer bounces, as we all do, right? So I want to 
reject using SMTP, not by sending a DSN/MDN.

It occured to me that if I provide a postffix content filter interface 
to the sieve interpreter, then postfix can do SMTP rejections. The 
sieve interpreter must run the user's regular active script, skip any 
clauses that can't be evaluated (and don't contain stop), and see 
whether a reject is executed. (In my code, this happens to be fairly 
simple.) Done.

The only problem is that I cannot with good conscience execute a 
possibly invalid action, can I?

Arnt

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Tue, 11 Jul 2006 19:14:25 +0200

> reject using SMTP, not by sending a DSN/MDN.

I think we all want this to be possible.

> whether a reject is executed. (In my code, this happens to be fairly 

thanks for the explanation.  so your implementation will actually stop
at the first occurence of stop, even if it is inside a test block?
consider this script:

        require "fileinto";
        if header "Subject" "foo" {
           fileinto "INBOX.foo";
           stop;
        }
        reject "go away";

if I read your explanation correctly, this will never reject SMTP time,
correct?

this may give more expressibility than my suggestion, but I haven't
found an example yet, unless you allow reject after a fileinto to be a
no-op, as you say.  how do you figure the following script should be
handled?

        require "fileinto";
        if header "Subject" "foo" {
           fileinto "INBOX.foo";
        } else {
           keep;
        }
        reject "go away";

-- 
Kjetil T.

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 11 Jul 2006 13:37:01 -0400

> this is just a slight variation of your #1, isn't it?

Yes, but perhaps easier to get drafted, since it's one thing to
introduce a general mode variable with open-ended meanings,
and another to propose a boolean variable with only one
narrow purpose.  It's
   if 'mode == "rcpt_to"'
vs
   if '! have_message'

> what more information?  you want scripts to be able to return 4xx as

That sort of thing, yep.  Plus extended result codes, which I suppose
could be encoded into the reject/refuse text.

But I was also thinking this:  I've found that I have need for results
that are "maybe"s.  For example, one return might be "I reject this, but
other logic in the implementation can override my decision" vs "I reject
this, and I really mean it."  Or contrarywise, "I'm OK with this, but
the implementation can override me" and other extrapoloations.  There are
other shades in there somewhere too.  There are also things like "I
reject this, but give a 250 now and defer rejecting until DATA time."

A lot of these things could simply be handled by returning more than one
piece of information, but that's one of those things that, as I say,
need fleshing out.  Other people will likely have had other experiences
and other needs as well.  I certainly have more things that I've thought
of but haven't actually needed or implemented yet.

Some (all?) of these may be things that ordinary users aren't going to
want, but scripts aren't always end-user things.

mm

---

From: "Mark E. Mallett" <[email-masked]>
Date: Tue, 11 Jul 2006 14:01:53 -0400

> possibly invalid action, can I?

No you can't :-)

But I consider this just part of the general problem of what to do with
execution errors.  You have to deal with them, whether or not you can
identify right now the specific things that can go wrong in execution.
What do you do with:

    if <some test that succeeds> {
        reject "you aren't mem";
    }
    if <some other test that succeeds> {
        keep;
    }

(substitute something other than "keep" if that isn't problematic enough.)

Perhaps in the context of a rcpt-to milter you can be a little fast and
loose and stop execution whenever some explicit action is taken.  But
that has problems of its own, not the least of which is that you won't
discover incorrect scripts very easily.

Maybe I'm missing the point though.  

mm

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 13 Jul 2006 11:04:56 +0200

Kjetil Torgrim Homme writes:
>>  I want my code to send fewer bounces, as we all do, right? So I want 
>>  to reject using SMTP, not by sending a DSN/MDN.
>
>>  It occured to me that if I provide a postffix content filter 
>>  interface to the sieve interpreter, then postfix can do SMTP 
>>  rejections. The sieve interpreter must run the user's regular 
>>  active script, skip any clauses that can't be evaluated (and don't 
>>  contain stop), and see whether a reject is executed. (In my code, 
>>  this happens to be fairly simple.) Done.
>
> if I read your explanation correctly, this will never reject SMTP 
> time, correct?

Never at RCPT TO time. (DATA ... CRLF.CRLF time is another question; I 
haven't looked at whether I could do that yet, but another 
implementation might be able to.)

> found an example yet, unless you allow reject after a fileinto to be 
> a no-op, as you say. how do you figure the following script should be 
> handled?
>
>         require "fileinto";

My code (such as it is) would not be able to make a decision before 
DATA. I'll withhold opinion about whether it should be possible at all 
in this case.

IMO, the important bit is that a script which checks the envelope and 
then does other checks works gracefully: Fail at RCPT TO if the 
implementation can do that, at CRLF.CRLF if ditto and all recipients 
agree, and sends a DSN/MDN/whatever if all else fails.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 13 Jul 2006 12:38:28 +0200

> agree, and sends a DSN/MDN/whatever if all else fails.

Ick. Forget it. Thinko. A script doing this would of course use an 
explicit stop after the reject, and then the issue is moot.

Arnt

---

From: Aaron Stone <[email-masked]>
Date: Wed, 12 Jul 2006 14:54:18 -0700

> RCPT TO, for DATA, no way.)

I'm with Nigel on this. I would never make mail logs available to an end
user -- they are there for the system administrator to monitor the
health of the system, for compliance with company policy (e.g. the
company prohibits excessive personal use of its email system) and laws
(e.g. US federal requirements that banks log all email).

If it can't be used to deter spammers, such is life.

> I don't think that's open to interpretation.

The spec was written for a less hostile environment. In the case of the
US federal law above, even spam would probably have to be logged. I
could easily see an issue coming to course and requiring some evidence
that an email was not received because it was caught by a spam filter.
So even an email rejected with a 5yz code would still need to be logged.
Is that a form of delivery? It is different if the system does it vs.
the user doing it? Does this script undermine email in some evil way:

    fileinto "Rejected Mail";
    reject "I think not.";

Aaron

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Thu, 13 Jul 2006 22:52:12 +0200

On Wed, 2006-07-12 at 14:54 -0700, Aaron Stone wrote:
> I'm with Nigel on this.

but you don't answer my question, what is gained by doing reject?

> I would never make mail logs available to an end

in many (most?) European countries, the user is entitled to seeing _any_
logs and database entries pertaining to him/her.  in practice, when a
user makes such a request, a lot of manual work must be done to collect
all the log excerpts, but this is mostly due to such requests being rare
(we get perhaps one or two per year here at the University of Oslo), so
it's not worth the effort to automate it.

but all this is very much beside the point.

> If it can't be used to deter spammers, such is life.

so the reject is useless?

> US federal law above, even spam would probably have to be logged.

according to Norwegian law, official contact addresses in the public
service can NOT be filtered automatically, every inquiry must be judged
on its own merits by a human.  however, our mail system doesn't lie and
say that it was rejected.  it would be a bizarre if the applicant got a
rejection message and his missive still ended up in the archive for
posterity.  such archival without his knowledge could very well be in
breach of the transparancy laws.

> I
> Is that a form of delivery?

it's a delivery if the message itself is stored.

> It is different if the system does it vs.

it might.  did you never see a fetchmail going bananas?  this is just
the sort of thing which can confuse such software -- and it's entitled
to.
-- 
Kjetil T.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 14 Jul 2006 11:21:30 +0200

> but you don't answer my question, what is gained by doing reject?

SMTP rejection is the good way to let unlucky senders know about false 
positives.

If you only reject (ie. you don't secretly store), reject sometimes also 
lets you out of storage rules. For example, here in Germany HGB§257 may 
(AFAIK there have been no test cases) require businesses to store 
certain categories of email for a number of years. The safe ways to 
deal with that are a) to store all mail for that long, even the spam, 
b) have some human pick out the false positives from the spam and 
determine what must be stored, and c) reject the presumed spam at SMTP 
time, so it's not received in the first place.

Arnt

---

From: Kjetil Torgrim Homme <[email-masked]>
Date: Fri, 14 Jul 2006 12:08:02 +0200

> time, so it's not received in the first place.

in case it wasn't clear, I fully agree with this.  we really want to do
reject as much as possible, it's the secret store I object to.

-- 
Kjetil T.

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 15 Jul 2006 15:16:57 +0100

Kjetil Torgrim Homme wrote:

>That capability needs to be added to SMTP, right?
>  
>
Yes, that would be an SMTP extension for allowing of UTF-8 human 
readable response text.

---

From: Aaron Stone <[email-masked]>
Date: Fri, 14 Jul 2006 03:09:34 -0700

> least script fragments, for each mode.

There is a significant portion of the LEMONADE work involving Sieve
scripts that are activated by various IMAP actions. Could this be
extended to SMTP actions as well?

Aaron

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 18 Jul 2006 14:20:45 +0100

>extended to SMTP actions as well?
If there is enough interest to do this, I would encourage people to make 
this a separate document:
1). It is not clear if Lemonade WG end up using Sieve and it is not 
clear when this work will complete;
2). Profiling Sieve for different SMTP environments seems to be an 
independent (and hopefully easier) task, even though it might intersect 
with IMAP Sieve work (e.g. "environment extension").

---

From: Aaron Stone <[email-masked]>
Date: Mon, 17 Jul 2006 06:04:31 -0700

> time, so it's not received in the first place.

Would it be worth someone's getting a legal review of this issue? A few
folks on this list are with companies that have general counsels, and it
might be worth asking the question, "What if we get sued for not being
aware of something that came in an email that got protocol rejected?"

My *guess* is that there is probably some existing case law about what
happens when a letter is lost by the post office, or misdirected, or
otherwise not delivered as usual. Some of this case law may even be
hundreds of years old in countries with common-law systems.

Aaron

---

From: Aaron Stone <[email-masked]>
Date: Mon, 17 Jul 2006 05:51:25 -0700

> but you don't answer my question, what is gained by doing reject?

If, as you claim, spammers don't wash their lists based on SMTP
rejections (really!?) then perhaps what's needed, for those looking at
this as a portion of an overall anti-spam policy, is a hook to some
lower-level defense measures, such as OpenBSD's spamd proxy, which
monkeys with protocol timings to muck up the spammer's outbound system. 

But this is a reaction to a truly hostile Internet, not the happy go
lucky friendly Internet that email was first designed for :-\

> logs and database entries pertaining to him/her.
[snip]

I am not aware of such laws in the US. (hmm. grumble.)

> so the reject is useless?

I thought that quite a few good reasons for reject were already given.
Is this even a topic that's still at-issue?

> breach of the transparancy laws.

Interesting.

> it's a delivery if the message itself is stored.

Fair enough; there'd be an SMTP log that a message of some kind was
rejected, though not the text of the message itself. So it covers half
of the base that I'm positing -- maybe well enough not to worry.

> the sort of thing which can confuse such software -- and it's entitled

I dunno. But have you ever seen/heard ESR himself going bananas? ;-)

Aaron

---

From: Aaron Stone <[email-masked]>
Date: Mon, 17 Jul 2006 06:21:27 -0700

> Yes, that would be an SMTP extension for allowing of UTF-8 human 

I'd rather like to see such a UTF-8 extension, instead of the
workarounds being proposed to shoehorn messages into US-ASCII.

Aaron

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 18 Jul 2006 17:23:00 +0100

>workarounds being proposed to shoehorn messages into US-ASCII.
Well, I will be working on one, but it is not clear when the document 
will be done and how long it would take to deploy it.
We would need some advice in the Sieve reject draft to address short 
term problems.

---

From: Kristin Hubner <[email-masked]>
Date: Tue, 18 Jul 2006 13:41:52 -0700

On Jul 18, 2006, at 9:23 AM, Alexey Melnikov wrote:

>>>>> 2). Non ASCII text in rejection string - should it cause creation 
>>>>> of DSN/MDN, runtime error or stripping of non-ASCII content?
> will be done and how long it would
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I would recommend working on that with the IETF EAI Working Group, if
you aren't already doing so.

Regards,

Kristin

> take to deploy it.

---

From: Kristin Hubner <[email-masked]>
Date: Tue, 18 Jul 2006 13:36:40 -0700

> workarounds being proposed to shoehorn messages into US-ASCII.

Of course it would be nice to be able to issue other sorts of text
in SMTP responses.  But such proposals have been going nowhere for
years.  However, the IETF EAI (Email Address Internationalization)
Working Group work may finally kick this into happening.  (After
all, currently SMTP responses often include a domain name such as on
the banner line, or an e-mail address such as in a response to a
RCPT TO: or EXPN: command.  So internationalizing email addresses
leads right into at least some internationalization of SMTP sessions.)
Note that although that although the EAI focus is on
internationalization of email addresses, their charter is a more
general examination of internationalization of the email environment,
as internationalized email addresses rapidly bring up other issues.

Quoting from Section 1.2, Problem Statement, of

            Overview and Framework for Internationalized Email

http://www.ietf.org/internet-drafts/draft-ietf-eai-framework-01.txt

    Internationalization of email addresses is not merely a matter of
    changing the SMTP envelope, or of modifying the From, To, and Cc
    headers, or of permitting upgraded mail user agents (MUAs) to decode
    a special coding and display local characters.  To be perceived as
    usable by end users, the addresses must be internationalized, and
    handled consistently, in all of the contexts in which they occur.
    That requirement has far-reaching implications: collections of
    patches and workarounds are not adequate.  Even if they were
    adequate, that approach risks an assortment of implementations with
    different sets of patches and workarounds having been applied with
    consequent user confusion about what is actually be run and
    supported.  Instead, we need to build a fully internationalized email
    environment, focusing on permitting efficient communication among
    those who share a language or other community (see [I18Nemail-
    constraints] for an extended discussion of this optimization).

And take a look at the mentioned I18Nemail-constraints:

  Internationalization in Internet Applications: Issues, Tradeoffs, and
                             Email Addresses

http://www.ietf.org/internet-drafts/draft-klensin-ima-constraints-00.txt

especially

3.3.  Communication across languages and cultures

    All of this implies that those who communicate across language and
    cultural groups will be required to learn, if they do not understand
    already, to be quite self-aware about the use of internationalized
    identifiers, as well as other examples of characters or languages,
    across those boundaries.  There will be a lower level of demands on
    those who communicate only in a single language and within a single
    culture.  This is, of course, not an issue that originated with the
    introduction of the Internet: it has been this way since languages
    and scripts started to differentiate from each other and since
    different cultures came into contact.  As we internationalize the
    network, a user of a given language that cannot be fully expressed in
    ASCII will always be faced with a choice between insisting on the
    purism of an email address local part and domain name in the script
    associated with the local language and maximizing the number of
    people who can communicate with her conveniently.  In some cases, the
    right answer will be "local language", in others, it will be "ASCII",
    and in still others it will be "maintain two addresses".  We are not
    required, and should not try, to make that choice for users: the
    users should make the best choices for their own needs, preferably
    after understanding the consequences of the choices.  As a community,
    we will need to be very clever about user interfaces.  As an example
    much more general than email, if someone with no ability to read
    Chinese characters sees a domain name written in those characters and
    decides she wants to copy and paste it somewhere, the copy mechanism
    is probably going to need to provide for both "copy the Chinese" and
    "convert quietly to punycode and copy that".  Either choice, by
    itself, will be wrong sometimes.  Users who both want to use Chinese-
    script domain names and communicate outside that language or script
    or culture are going to either learn to understand the difference and
    relationship, or develop some good rituals that work, or the network
    will keep slapping them in the head with failed lookups or bounced
    mail until they do learn.  Of course, substantially any language or
    script could be substituted for "Chinese" in that example.

Substitute in the words "error response text" where the above discussion
talks about email addresses, and I think it still applies pretty well.

But in any case, take a look at the EAI planned timeline, which is at
this point merely for _experimental_ RFCs (to start testing out ideas
and implementations), and then keep in mind that even supposing an RFC
comes out with an SMTP extension for non-US-ASCII text in SMTP dialogue
(in particular, non-US-ASCII text in SMTP responses), there will be a
long (long!) time before one can forget about the old software that 
doesn't
support such.  So as far as SMTP responses are concerned, either 
sticking
to US-ASCII, or being able to downgrade from non-US-ASCII to US-ASCII 
when
dealing with pre-extension SMTP software is going to be necessary for a
long, long time.

So as far as what we can do _now_ for rejection text: sticking with 
US-ASCII
rejection text has an efficiency advantage (allows SMTP protocol level
rejection) for those who are willing to accept the restriction of 
US-ASCII
text, but some users and user communities will prefer (demand!) to use 
their
"own language", as they can with the "old" (original) reject behavior, 
even
though that means the cost of generating a notification message (DSN or 
MDN)
instead of being able to do SMTP protocol level rejection.  And I would
consider that completely appropriate for them to do in non-spam 
rejections,
though I might hope/encourage them to stick with US-ASCII and hence make
SMTP protocol level rejection possible for believed-to-be-spam message
rejections.

I would take a quote from above regarding email addresses (local 
language
vs. US-ASCII vs. providing both forms of email address) and apply it to
rejection text:

    We are not
    required, and should not try, to make that choice for users: the
    users should make the best choices for their own needs, preferably
    after understanding the consequences of the choices.

That is, I believe that we need to allow users -- when they wish -- to
choose which behavior they want.  And I do not believe that it is 
possible
to avoid at least some user education/training, at least in the form of
a "smart" user interface, especially for users who normally operate in
languages not written using US-ASCII.  While Western European language
users may be able to remain happily oblivious of the difference between
SMTP protocol rejection and rejection in the form of a notification 
message,
oblivious of any (desired) difference in rejecting spam vs. other 
rejections,
users of languages that use/need a different charset _will_ need to 
have at
least some awareness (or the interface that generates Sieve filters on 
their
behalf will need some awareness) that it is much preferable to reject
believed-to-be-spam messages with the "stick to US-ASCII-only" option,
whereas more "personal" rejections for other reasons can, if the user 
wishes,
be rejected using more personalized text in their own language.

Regards,

Kristin

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 19 Jul 2006 10:32:01 +0200

Aaron Stone writes:
> Would it be worth someone's getting a legal review of this issue? A 
> few folks on this list are with companies that have general counsels,

I'd expect most are, or retain the services of an external lawyer for 
the same purpose. (In the case of my employer, an external lawyer. Nice 
chap.)

> and it might be worth asking the question, "What if we get sued for 
> not being aware of something that came in an email that got protocol 
> rejected?"

But who will turn around and repeat the answer to the general public? Not I.

Arnt

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 19 Jul 2006 13:47:55 +0100

>> will be done and how long it would

Yes, Chris Newman has suggested to submit the draft to EAI WG.