Subject: Re: [Asrg] Are we allowed to extend the SMTP protocol?

From: [email-masked]
Date: Mon, 10 May 2004 16:56:19 -0500

> Date: Sun, 09 May 2004 16:12:00 -0700
From: Jeff Silverman <[email-masked]>
Reply-To: [email-masked]
To: [email-masked]
Subject: [Asrg] Are we allowed to extend the SMTP protocol?

> People,

>    I was wandering through the SMTP RFC 
(http://www.faqs.org/rfcs/rfc2821.html and other places), and I got to 
section 4.2 which is the reply codes that the receiving MTA sends back 
to the sending MTA.  I think this is where we ought to focus our thinking.

>    Can we modify the SMTP to add an authenticating capability?  I think 
so.  

While we certainly COULD, I think it's not desirable for a whole variety of 
reasons.  Among them:

  1)  There are a LOT of systems around the world which handle E-mail, and many 
of them qualify as "legacy" systems where the original company having written 
the software may have orphaned it.  It's not safe to presume that it either can 
be changed, or even that it's reasonable to expect a user to change software 
that "has always worked fine."

  2)  Many older programs won't know how to handle (and may not even report) 
previously unknown response codes.

  3)  The fact that mail sending/receiving software is "authenticated" is 
TOTALLY IRRELEVANT to the issue of whether the E-mail being sent is spam or not. 
Before a zombie spambot got infected, no doubt it was trustworthy and legitimate 
(and it probably sends at least SOME legitimate and non-spam mail, even AFTER 
being infected).

I truly don't understand this continuing fascination with the question of 
authenticating/verifying/certification of users and mail servers.  I understand 
that it _sounds_ like it probably is a good idea, but it TRULY doesn't do very 
much to prevent (or intercept!) the sending of spam!

> In particular, I notice the following in the RFC:

> The list of codes that appears below MUST NOT be construed as
   permanent.  While the addition of new codes should be a rare and
   significant activity, with supplemental information in the textual
   part of the response being preferred, new codes may be added as the
   result of new Standards or Standards-track specifications.
   Consequently, a sender-SMTP MUST be prepared to handle codes not
   specified in this document and MUST do so by interpreting the first
   digit only.

That's okay, although (again) existing legacy software clearly won't be in a 
position to differentiate between old codes with the same first digit and new 
ones.

>  What if we added a code 320 or 453, I think, which means authenticate 
yourself to me?   Then sending MTA will then queue the message for 
retransmission later.

I don't much like in principle any schemes which give information to spammers... 
whether that be what they need to change to get messages through, or most 
anything else.

I also don't like schemes as a matter of principle which result in adding cost 
to the worldwide E-mail system, whether that be human customer service people, 
or bandwidth cost for retries, or worldwide inherent software complexity.  
Ultimately, that cost will be borne (one way or the other) by "normal users", 
folks like you and me.

>    The receiving MTA would then send a Turing-difficult test to the 
erstwhile sender of the message, such as a pointer to a graphic that the 
sending MTA has to read (e.g. 
http://www.commercialventvac.com/~jeffs/mail.jpg 
<http://www.commercialventvac.com/%7Ejeffs/mail.jpg>) or it could be all 
text that a human still has to parse for example

>   A         H    H     A      RRRR  DDDD     TTTTTTT EEEEEE     SSSSS 
TTTTTTT
  A A        H    H    A A     R   R D   D       T    E         S         T
 A   A       H    H   A   A    R   R D    D      T    E          SSSS     T
AAAAAAA      HHHHHH  AAAAAAA   RRRR  D    D      T    EEEE           S    T
A     A      H    H  A     A   R   R D   D       T    E        S    S     T
A     A      H    H  A     A   R   R DDDD        T    EEEEEEE   SSSS      T

> Or it could be a sound clip.

And what do you think that mailing list software (Yahoogroups, Majordomo, or 
whatever) is going to do with such stuff?

Or, as another example... one of my consulting clients sends (automated, direct 
from their back-end accounting system) E-mailed PDF file copies of invoices to 
their customers.  What's likely to happen to these kind of "human challenge" 
bouncebacks?

Answer... they're going to fall on the ground, much of the time.  I've gotten 
newsletters from several very unhappy publishers pointing out that they TRULY 
don't have time for such foolishness, and stating that they either ignore or 
unsubscribe users sending them back such nonsense.

As for your ASCII text thing above, that operates on several (likely invalid for 
some users) assumptions... one being that they have a 80-column (or more!) 
screen width, another that they're viewing the message with a fixed-width font, 
another being that their paragraphs don't automatically get "reformatted" (much 
the way the top of the "T" wrapped for me).

>  The sender of the message takes the test, and puts the answer in the 
subject field.  This message is sent to the receiving MTA.  The 
receiving MTA has to go through the message until it gets to the subject 
field so that it knows that this is a test and not a message.

Again, these schemes fall apart (badly) when you have large-scale (legitimate) 
mailers.  Yahoogroups for instance.

>  Once the sender has passed the test, then the receiver can add the 
sender to a white list which is under the control of the user.  I 
realize I am not very structured - this idea is somewhat half baked.  
The user would have three lists of addresses of correspondents: a white 
list, which can be passed without further testing; a blacklist, which 
can be rejected without further testing; and a graylist, which must be 
tested.  Whenever a new correspondent appears, they automatically are 
added to the graylist by the MTA.  If the user decides that this 
correspondent is a spammer, then they can add the user to the blacklist.

I don't much like that approach, because I don't think it's finely-grained 
enough.

I think what's important is that as a recipient, I expect and trust a given 
sender to send me a CERTAIN TYPE of stuff, and (maybe) nothing more exotic than 
that.  

I may subscribe to a newsletter that's sent in plain ASCII text, or (maybe) with 
limited HTML formatting. If I get an E-mail from (even the same From address, 
whether legitimate or spoofed) that sender that suddenly calls for Java 
scripting, or an ActiveX enclosure, or has a PIF or EXE or SCR file 
attachment... then I probably don't want to handle it the same way just because 
I know the sender and trust the stuff they USUALLY send to me.  I probably want 
to quarantine (or toss entirely) such questionable stuff.

And likewise, just because I got a (maybe spoofed) infected message from someone 
CLAIMING to be my trusted correspondent, doesn't mean that I want to change 
their status for the OTHER (legitimate) stuff they will continue to send me.

>   There are a couple of problems with this.  One is that the Turing 
difficult test is language dependent.  I am used to thinking in terms of 
a roman alphabet, but what if the receiver and the correspondent are 
Israeli or Arabic?    

Then presumably the test also works in the language the sender and recipient 
understand!!!  Obviously, if they're e-mailing each other, there presumably is 
SOME common language permitting communication.

> Another problem is that if the correspondent were 
to somehow learn that an address was on the receiver's white list, then 
the correspondent could send a message with that address.  

Yes, and some of these won't be a huge surprise, especially if the sender is a 
big sender and normally sends relatively exotic stuff.  To some degree, that's 
part of the cost of sending more exotic E-mails (even legitimately):  it 
increases the likelihood that they won't be reliably delivered.

But that's part of the reason why it's important to have a finely-grained 
"permissions list" approach rather than a (too!) simple white/grey/black list.

> The user response in that case would be to move that address from the white 
list back to the gray list.

If they can differentiate based on content, then there should be NO reason to 
NOT scan the "whitelisted" entries, just as for the "greylisted" ones.

>    The big advantage of this scheme is that it requires modification 
only to the receiving MTA and requires no changes to the UTAs.  My 
intuition tells me that there are far fewer MTAs than UTAs and since the 
MTAs are under the control of sysadmins and not users, it will be easier 
to update them.

That's very much a mixed bag, though.  The fact that the sysadmin is responsible 
means angry complaints by some po'd users, who will object to WHATEVER the 
settings are.  This is one of those cases, I think, where a user-based product 
is better, and allows them to set it up in JUST the way they prefer.
 
> So, I couple of questions:

> 1)  Does this sound like a good idea?

I've listed some of my comments on it.

> 2)  If it is a good idea, then what do we do next?

> 3) If it is a bad idea, then why is it a bad idea, and can it be fixed 
or is it hopeless?

It depends on how much "pounding" on it it takes while you're still taking that 
as "fixing" versus "redesign".  :-)

*I* think that a single-ended system, under the TOTAL control of the receiving 
user, is the way to go.  I like a finely-grained permission list where the 
recipient has a very narrow-mesh gauntlet that incoming mail must traverse (and 
without help from outside) in order to reach the Inbox, and where there's no 
feedback sent to the sender.

> I recognize that this is a variant on challenge/response, 

Right, and I consider that a non-starter in principle, since large scale mailers 
simply can't be bothered.

> but I missed most of the discussion of challenge/response and I am not sure I 
understand why it is not a good idea.

Hopefully some of this discussion has helped you to "get it".  :-)

Gordon Peterson                  http://personal.terabites.com/
1977-2002  Twenty-fifth anniversary year of Local Area Networking!
Support free and fair US elections!  http://stickers.defend-democracy.org
12/19/98: Partisan Republicans scornfully ignore the voters they "represent".
12/09/00: the date the Republican Party took down democracy in America.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Bill Cole <[email-masked]>
Date: Mon, 10 May 2004 19:55:29 -0400

At 4:12 PM -0700 5/9/04, Jeff Silverman wrote:
>[...]
>erstwhile sender of the message,

How do you expect to figure out who that is?

>  such as a pointer to a graphic that the sending MTA has to read 
>(e.g. 
><http://www.commercialventvac.com/%7Ejeffs/mail.jpg>http://www.commercialventvac.com/~jeffs/mail.jpg) 
>or it could be all text that a human still has to parse for example
>
>    A         H    H     A      RRRR  DDDD     TTTTTTT EEEEEE     SSSSS TTTTTTT

You are quite completely wrong in assuming that a human is needed to 
parse that.

As for the images that are used by some operations now (like Yahoo) 
to determine humanity, the spammers have already figured out how to 
harness humans to solve them unwittingly. Some people will do 
practically anything for pornography.

Of course, you also run right in to issues of what blind people would 
do. If you think there are no blind people using email, you are again 
quite completely wrong.

>Or it could be a sound clip.

An even more naive idea. To the point of qualifying for words like 'stupid.'

Anyone who can see and display a JPEG image can type back in a 
character string as long as it sticks to a limited character set like 
the mailsafe 64, but identifying a sound is impossible to people with 
bad hearing and probably to anyone who does not share a specific 
dialect with the test administrator.

>   The sender of the message takes the test, and puts the answer in 
>the subject field.  This message is sent to the receiving MTA.  The 
>receiving MTA has to go through the message until it gets to the 
>subject field so that it knows that this is a test and not a message.
>
>     Once the sender has passed the test, then the receiver can add 
>the sender to a white list which is under the control of the user. 
>I realize I am not very structured - this idea is somewhat half 
>baked.  The user would have three lists of addresses of 
>correspondents: a white list, which can be passed without further 
>testing; a blacklist, which can be rejected without further testing; 
>and a graylist, which must be tested.  Whenever a new correspondent 
>appears, they automatically are added to the graylist by the MTA. 
>If the user decides that this correspondent is a spammer, then they 
>can add the user to the blacklist.
>
>
>     There are a couple of problems with this.  One is that the 
>Turing difficult test is language dependent.  I am used to thinking 
>in terms of a roman alphabet, but what if the receiver and the 
>correspondent are Israeli or Arabic?    Another problem is that if 
>the correspondent were to somehow learn that an address was on the 
>receiver's white list, then the correspondent could send a message 
>with that address.  The user response in that case would be to move 
>that address from the white list back to the gray list.
>
>     The big advantage of this scheme is that it requires 
>modification only to the receiving MTA and requires no changes to 
>the UTAs.  My intuition tells me that there are far fewer MTAs than 
>UTAs and since the MTAs are under the control of sysadmins and not 
>users, it will be easier to update them.
>
>
>

It's an OLD idea which has proven in practice to be garbage.

What you are talking about is generally referred to as a 
'challenge/response authentication protocol' and the acronym is 
fitting. I have received dozens of those challenges in the past year 
for mail I never sent along with hundreds of normal bounces from mail 
systems which accepted mail with my address in the envelope

ONE such challenge to any mail hitting my private domain is adequate 
for me to never accept a second piece of mail from that system again. 
If such a mechanism became anything like a standard practice, I'd 
abandon email.

>2)  If it is a good idea, then what do we do next?

Drive a stake through its heart so it stops rising from the dead 
every time someone new starts thinking shallowly about how to stop 
spam.

>3) If it is a bad idea, then why is it a bad idea, and can it be 
>fixed or is it hopeless?

Hopeless.

>I recognize that this is a variant on challenge/response, but I 
>missed most of the discussion of challenge/response and I am not 
>sure I understand why it is not a good idea.

The archives are open. Years of discussion in comp.mail.* and 
news.admin.net-abuse.email are open through Google. Read them. If 
reading the prior discussions of challenge/response ideas and 
implementations fails to convince you that it is an unsalvageably bad 
idea, come back and explain how it can be implemented so as to NOT 
generate robotic, bulk, unsolicited email in my mailbox.

There are language dependencies, technology dependencies, and human 
ability dependencies. Each is fatal in itself. The recent past of 
people adopting these sorts of measures for other media like websites 
(where those flaws are  subject to some mitigation) has led to an 
attack which is intrinsically impossible to overcome: the entity 
seeking to automate the test merely hooks the tests up to a proxy 
system fronted by something that he can draw a lot of human eyes to, 
such as a free porn site. There's no clear way around this.

-- 
Bill Cole                                  
[email-masked]