Subject: Re: Why spam is a problem

From: [email-masked] (Greg Skinner)
Date: Tue, 20 Aug 2002 20:16:34 +0000 (GMT)

I haven't had a chance to follow this thread entirely, so I apologize
if this has already been mentioned.  I ran across this in a coworker's
blog.  Being a fan of statistical techniques that provide reasonable,
cost-effective (but not necessarily perfect) solutions to difficult
problems, it appealed to me. :)

http://www.garyarnold.com/projects.php#bayespam

---

From: [email-masked]
Date: Tue, 20 Aug 2002 14:04:27 -0700 (PDT)

As everyone I'm sure have seen on the last "why is spam a problem" and 
other similar threads on ietf as well as numerous similar threads on 
other lists and boards, there is a serious need to do something to limit 
amount of unsolicited email. While the roots maybe social issue I do not 
see why we can not work on it from technical point of view. In addition 
to that during last years, I'v seen real need for new features to be 
added into SMTP, such as ones for callback, delayed transmission, delivery
notification,secure communications, etc, etc and there are in fact 
several drafts available on some issues. As far as anti-spam  mechanisms I 
do not belive we should force some particular method on everyone but 
rather built several verification features into protocol and allow server 
operators to themselve choose if they want to use it. Where the features 
were use the email would be considered more secure and users can use that 
to sort out mail (as many do already with special filters).

I believe its time we start working within IETF on new version of SMTP 
that would have more features and be more secure. I'v tried to point this 
out several times before on nanog and ietf hoping that someone would take 
the initiave but as this did not happen, I'm willing to do it now. At this 
point I'm proposing creation of IETF working group that would look into 
ways to extend SMTP. I'v created website and mailing list to discuss 
charter of the proposed working group at http://www.smtpng.org

Those who agree with me, please subscribe to the mailing list and lets 
work on this futher in a kind-of BOF. I'm also looking for two co-chairs 
for the working group with at least one preferablly having been chair of 
ietf group before. I'm planning on sending final draft for working group
charter in about two weeks time and right now I'm going to be contacting 
several people who have expressed interest in working on SMTP protocol as 
well as contacting IETF area director on proceeding with this.

-- 
William Leibzon
[email-masked]

---

From: [email-masked]
Date: Tue, 20 Aug 2002 17:56:53 -0400

> 1.      General extensions on SMTP negotiation This group will work on
> extensions to general SMTP syntax with primary focus on mechanism for
> negotiation on type of verification and security for transmission of mail
> message between two mail servers.

EHLO - a done deal a decade ago.

> 2.      Callback and delayed transmission extensions
> This group will work on extensions to SMTP for email callback mechanism.
> Callback should allow for special type of negotiation on transmission where
> instead of immediately sending a message, one email server would request a
> callback and specify time when the email message can be picked up from the
> original server. Additional work should also be done on delayed transmission
> extensions as proposed in draft-vaudreuil-futuredelivery-00 draft.

ETRN could be extended, and you have a draft that should probably be handled
in the IETF-SMTP group.

> 3.      Secure SMTP transmission
> This group will focus on extensions allowing for additional security and
> encryption for SMTP transmission for situations where such functionality is
> desired by both parties.
> 4.      Certificate based verification
> This group will work on certificate-based verification. This functionality
> would involve incorporation of SSL-like functionality into SMTP to allow two
> mail servers to verify each other by exchanging certificates.

RFC2487.  Already shipped as part of Sendmail 8.11.0 over 2 years ago.
Only thing missing here is a good X.509 PKI, which is out-of-scope for
an SMTP working group anyhow.

> 5.      Callback verification
> This group will also work in conjunction with group#2 on how callback
> mechanisms can be used to verify authenticity of mail servers. Work would also
> be done on callback verification as proposed at draft-nunn-ssmtp-00.txt draft.

And this needs a new working group why, rather than doing it in the IETF-SMTP
group?

> 6.      User/Password and other authentication verification
> This group will focus on authentication verification that can be used in
> conjunction with SMTP. The primary use of authentication should be to allow
> end-users to relay messages through their ISP's mail server, but this group
> should also work on how the same methods can be used in ISP-ISP mail server
> verification. Work should also be done in conjunction with group#3 to make sure
> that clear text passwords are not sent across public networks.

The same methods can be used no problem.  There's no reason SMTP AUTH can't be
used between ISP's, except for the lack of a PKI, unless you want to manage
O(n**2) password/authentication token pairs.

> 7.      Signatory and other verification mechanisms
> This group will work on verification schemes involving special type of
> signatures that would accompany email and on other similar verification methods

We already have S/MIME and OpenPGP standards.

> 8.      Delivery confirmation and address verification
> This group will work on extensions allowing for delivery confirmation options
> to be incorporated into SMTP, the initial start can be
> draft-moore-rfc1891bis-01draft. Additional work should also be done on ways to
> use mail server verification mechanism in conjunction with email address
> verification (certain level of trust between mail servers is often desired
> before one mail server would allow another to know if certain subscriber
> exist).

Delivery confirmation is already a solved issue, with DSN and MDN support.
Address verification is going to be a can of worms - consider the reasons many
sites disable VRFY before you go much further.

> 9.      MIME extensions
> This group will work with all other groups on necessary additions and changes
> to MIME content types as used in email transmissions.

Too vague for a charter, I suspect.

> 10.  Email headers
> This group will work with other groups to define if necessary new email headers
> and work on clarification of use of existing email headers by mail transfer
> agent programs.

Similarly.

I see 2 "too vague", two "can of worms/scaling" problems (PKI and verification),
2 that should probably be handled in the current SMTP group, and a number of
already-solved problems.

I'm not sure what the working group would be charged with accomplishing.
-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech

---

From: [email-masked]
Date: Tue, 20 Aug 2002 15:27:18 -0700 (PDT)

> EHLO - a done deal a decade ago.
It it seriously not capable of handling what is needed, particularly any 
kind of verification, callback, exchange of certificates, etc, etc.
 
> ETRN could be extended, and you have a draft that should probably be handled
ETRN can be extended. Unfortunetly I do not see mail server operators 
adobpting new extensions unless they have more reason then just this.
The whole idea is to do it all together and make serious case for upgrade.
 
> an SMTP working group anyhow.
First of all I do not disagree that its out of scope. Second I'v actually 
done technical work on this issue and I have SMTP working over SSL tunnel 
as do many other but its not as simple as this and we need standards on 
how SMTP certificates would be issued, etc. The problem is that there is 
no standard on this and no wide use and only few MTAs have any kind of 
support for it. And if you notice I'm actually big advocate of using 
certificate for mail transmission & server verification and I know all 
about this - I do not see it being used in many places (and its a good 
way to stop SPAM BTW) until more work has been done on it.

> And this needs a new working group why, rather than doing it in the IETF-SMTP
Since you mention it again, I do not see IETF SMTP working group in the 
list of current working groups. And there was no organized effort to work 
on SMTP for quite some time except for S/MIME. There are some groups 
workin on unified messaging but this is not the same.

> O(n**2) password/authentication token pairs.
SMTP AUTH sends passwords in clear text for starters.
 
> We already have S/MIME and OpenPGP standards.
I meant to find a way to use exactly this standards in the more general
SMTP server verification mechanism. As alternative to PKI based 
verification. This has been raised on IETF before as being less secure, 
but this maybe sufficient for many.

> sites disable VRFY before you go much further.
Delivery confirmation will only be solved issue when everyone supports it.
Address verification in its current form is unsecure that is why everyone 
disables it. But it is usefull feature to have and we need to look into this.

> I'm not sure what the working group would be charged with accomplishing.

The current draft of the charter is in places vague and needs more work. I 
do however think that only what would can be new version of SMTP would be 
able to solve number of needed features, this question has been raised 
here on IETF numerous times and I know people are interested in technical 
resolution to number of outstading issues. I also think that if number
of features added into new version is substantial, it will provide serious
reason for users to upgrade where as gradual one feature additions do not 
seem to be working.

BTW - I fully understand that number of issues raised already have 
solution to a degree. I would however like to see official standards
and besides having solution to number of these issues will just make the
entire process go falster (and I'm already been told 2 years is too 
optimistic for working group like this..). So more to the point, those who 
have done work on real solutions are even more welcome then others :)

-- 
William Leibzon

---

From: Dave Crocker <[email-masked]>
Date: Tue, 20 Aug 2002 18:28:04 -0700

>kind of verification, callback, exchange of certificates, etc, etc.

No doubt you have a detailed list of functional requirements written 
somewhere, so that folks do not have to guess at what you are seeking to do 
(and why it needs to be done.)

So, please provide the URL to the document and then we might be able to 
consider your suggestion productively.

d/

----------
Dave Crocker <mailto:[email-masked]>
TribalWise, Inc. <http://www.tribalwise.com>
tel +1.408.246.8253; fax +1.408.850.1850

---

From: [email-masked]
Date: Tue, 20 Aug 2002 18:37:38 -0700 (PDT)

I do not have such a "detailed" list, I do have number of notes on the 
topic and if insist I'll share some of them. I do think most of this
should be discussed on te mailing list I created just for that very topic.

Basicly I'm looking to create a way for mail servers to negotiate with 
each other based on the particular transmission received (that means 
after TO and FROM have already been sent and not at EHLO level), the idea 
is that if possible email should be sent with maximum amount of "trust" 
between parties and that means if for example several protocol for trust 
are available, the servers would decide which to use. If certificate trust 
is supported and desired, the server would exchange and confirm certificates, 
if futhermore it is desired that callback is necessary (and is supported 
and agreed to by both mail servers) that would be done, if servers both 
want to encrypt the transmission, that would also be possible. The "trust" 
that was used in particular transmission is up to configuration of mail 
server operator (on both sides) and possibly depended on settings for 
particular user (both sending and receiving party if supported by software)

While I do not expect to see spam stopped all together, the trust 
mechanisms will provide an additional filter to determine when mail is or 
is not likely to have come from legitimate source. In addition for 
important emails, confirmation that email was received is necessary.

All this should lead to wider use of email where as now many see 
importance of email communication going down, largely due to amount of 
unwanted emails received and in part due to limitations of existing 
protocol.

---

From: John C Klensin <[email-masked]>
Date: Tue, 20 Aug 2002 23:16:49 -0400

--On Tuesday, August 20, 2002 3:27 PM -0700 [email-masked]
wrote:

>...
>> The same methods can be used no problem.  There's no reason
>> SMTP AUTH can't be used between ISP's, except for the lack of
>> a PKI, unless you want to manage O(n**2)
>> password/authentication token pairs.

Hmm.  I use SMTP AUTH fairly frequently, and have never used it
with a clear-text password.  Perhaps there is some disconnect
here (and in other parts of your note as well).

      john

---

From: John C Klensin <[email-masked]>
Date: Wed, 21 Aug 2002 00:19:47 -0400

William, I don't want to discourage you -- if you can find
others to explore these ideas with you, please go ahead and do
that.  But I share Dave Crocker's skepticism: because of some
"been there, tried to figure out how to really make it work,
couldn't" issues, I'm disinclined (as he is) to spend a lot of
time on something like this until there is a fairly detailed
proposal on the table.

Where previous analysis has led us is that strong trust
relationships are best handled end to end.  Given the store and
forward --and responsibility handoff-- model used by all
networked mail systems so far, that usually ends up meaning a
trust and authentication infrastructure as part of the message
payload, rather than server to server (which tends to get very
messy in relaying).  I'm not saying it can't be done in
transport, but I haven't yet seen a plausible argument for doing
it, and, at least as important, we've had fairly poor
operational experience with things that increase the amount of
handshaking (e.g., number of transaction turnarounds) or
calculations and contemplation during mail transport
transactions.

While SMTP is not my favorite protocol either, our experience
with deploying new protocols when something is out there that
sort of does the job has been just miserable.  Consequently, I'd
also encourage you to ask yourself the question of whether the
mechanisms you think you need could not be fitted into SMTP
extensions and, if not, for what reason.  To do that, you are
going to have to dig into the existing stuff sufficiently to
avoid comments like "SMTP AUTH uses clear-text passwords": they
are used in some situations, and may actually be reasonable for,
e.g., first-hop (submission) handling in many dialup
environments, but they are certainly not the only option and my
impression is that they are uncommon amount SMTP AUTH
implementations.  I don't know that I'd recommend it (i.e.,
please work through this, do the analysis, and tell us), but the
ESMTP machinery is flexible enough that a client could abandon
the attempt to send mail if the server didn't advertise the sort
of trust-negotiation extension you ask for, and the server could
reply to the first RCPT command, or even to DATA, with some sort
of "you need to exchange certifications and negotiate trust now"
if the client didn't send the relevant information when it was
expected.

   john

--On Tuesday, August 20, 2002 6:37 PM -0700 [email-masked]
wrote:

> I do not have such a "detailed" list, I do have number of
> notes on the  topic and if insist I'll share some of them. I
> do think most of this should be discussed on te mailing list I
> created just for that very topic.
> 
> Basicly I'm looking to create a way for mail servers to
> negotiate with  each other based on the particular
> transmission received (that means  after TO and FROM have
> already been sent and not at EHLO level), the idea  is that if
> possible email should be sent with maximum amount of "trust" 
> between parties and that means if for example several protocol
> for trust  are available, the servers would decide which to
> use. If certificate trust  is supported and desired, the
>...

---

From: [email-masked]
Date: Wed, 21 Aug 2002 01:57:36 -0400

> kind of verification, callback, exchange of certificates, etc, etc.

Those are *specific* extensions.  The general framework of EHLO has been
shown to work fine.

> The whole idea is to do it all together and make serious case for upgrade.

Given the number of things you'd have to upgrade, you had better show some
*really serioys* wins compared to what SMTP+EHLO can do before you're going
to see much chance of deploying.

> way to stop SPAM BTW) until more work has been done on it.

Actually, given that Sendmail 8.11 supports it, a *LOT* of MTAs have
support for it.  What you're seeing is the fact that few sites *enable*
the support.

I don't see that we need a standard for "how SMTP certificates are issued".
We already understand *very* well how to issue host certificates - if we
didn't, the https:// protocol would be quite DOA.

I'm not at all convinced that certificates for "mail transmission & server
verification" are as big a win as you might think.  What you probably *want*
here is more end-to-end verification.  Which do you care more about - that
the machine that submitted the mail to loki.ietf.org was authenticated,
or that it can be shown that I was the author of the mail?  Consider - the mail
I'm replying to had 9 Received: headers - only 2 of which were done by
machines that I fully trust, and another 2 by machines run by people I
can smack upside the head if they screw up. I suspect that at 4-of-9,
I have more control of the path than 99.999998% of the net's users.

> workin on unified messaging but this is not the same.

Odd, there was a lot of traffic on the list for discussing CONNEG.

You might also ask yourself what callback buys you security-wise on the
Internet - is there a realistic case where callback buys you anything?
Remember if they can hijack the 3-packet handshake for the *original*
connection, they can do it to the callback (and with even MORE opportunities
for fun and games - if I can make you do a callback I know when to drop
in my DNS cache poisoning or whatever).

Callback *partially* closed a number of problems with dialup modem access
(note however that most devices that did callback on the same line as the
inbound request were *painfully* easy to exploit, and even callback on a
second line still had race conditions).  I'm not convinced that said holes
are the same on the IP network as they were on the POTS network.

> SMTP AUTH sends passwords in clear text for starters.

It does?  CRAM-MD5 and DIGEST-MD5 are cleartext?   See RFC2554 and RFC2222.

> Delivery confirmation will only be solved issue when everyone supports it.

The point is that the *standard* is there, and most MTAs *do* support it to
some extent, but it's usually disabled for very good reasons - people seem
to have an innate need for plausible deniability.  We've *all* read our
e-mail at 4:50PM on a Friday and found a piece of mail we wish we hadn't
seen until 9:05AM Monday.  The last thing we want at that point is a
smoking-gun proof that we DID see it at 4:50 Friday.

> disables it. But it is usefull feature to have and we need to look into this.

Again, the standard is there.  The  standard is even flexible enough that
you can get away with answering VRFY/EXPN fully and accurately for queries
from some hosts (perhaps those in your administrative domain), issuing
partial answers for others (perhaps VRFY but not EXPN), and saying "Unsupported"
to other sites.

However, that's the proper realm of the programmer doing the code, and then
for the salescreature to brag up as a selling point.  The IETF has done its
part there, it's not our fault MTA developers don't provide better controls
on VRFY/EXPN.

Remember - some of us have been doing this for a LONG time, and there's a
lot of dragons with subtle camouflage but pointy sharp teeth.  There's a
very good chance that if you're asking yourself "why hasn't anybody done
this before?", that it HAS been thought of before, and that it's either
a lot harder to do correctly than it looks, or it's not addressing the
problem, or both.  Even the experts can get it wrong - this time last week,
I had 3 or 4 very recognizable names telling me exactly why an idea of mine
wasn't as generally applicable as I thought (thanks guys - turns out if
I had deployed it, I'd have broken my boss's email, for exactly the reasons
you cited it as a bad idea ;)

Step 1:  Ask if it's doable now with COTS software.  If it is, it's a
problem for your purchasing department, not the IETF.
Step 2:  Ask if it's supported by the RFCs but not shipping.  If it is,
it's a problem for your vendor to fix, not the IETF.
Step 3:  Ask if it's (a) reasonable and (b) addresses the problem.  If it's
not both, you need to reformulate the problem and/or solution.
Step 4:  Start thinking if an I-D is a good idea - *NOW* it's an IETF problem.

---

From: Martin Duerst <[email-masked]>
Date: Wed, 21 Aug 2002 14:35:16 +0900

Dear IETFers,

This is a question that I hope some of you can help me with.

The IETF uses the terms MUST/SHOULD/MAY very consistently,
according to http://www.ietf.org/rfc/rfc2026.txt.

I seem to remember that there is also some terminology
for implementations, in particular some terms for
1) implementations that do all the MUSTs, but not necessarily more
2) implementations that do all the MUSTs and all the SHOULDs

Can somebody point me to this terminology? Thanks in advance,    Martin.

#-#-#  Martin J. Du"rst, I18N Activity Lead, World Wide Web Consortium
#-#-#  mailto:[email-masked]   http://www.w3.org/People/D%C3%BCrst

---

From: Fred Baker <[email-masked]>
Date: Wed, 21 Aug 2002 09:52:26 -0700

>2) implementations that do all the MUSTs and all the SHOULDs

per RFC 1122:

          An implementation is not compliant if it fails to satisfy one
          or more of the MUST requirements for the protocols it
          implements.  An implementation that satisfies all the MUST and
          all the SHOULD requirements for its protocols is said to be
          "unconditionally compliant"; one that satisfies all the MUST
          requirements but not all the SHOULD requirements for its
          protocols is said to be "conditionally compliant".

---

From: "C. M. Heard" <[email-masked]>
Date: Thu, 22 Aug 2002 08:37:02 -0700 (PDT)

> according to http://www.ietf.org/rfc/rfc2026.txt.

I think you mean RFC 2119.

> 1) implementations that do all the MUSTs, but not necessarily more

RFC 1812 uses the term "conditionally compliant" for this.

> 2) implementations that do all the MUSTs and all the SHOULDs

RFC 1812 uses the term "unconditionally compliant" for this.

> Can somebody point me to this terminology?

From p. 10 of RFC 1812:

  An implementation is said to be conditionally compliant if it
  satisfies all the relevant MUST, MUST IMPLEMENT, and MUST NOT
  requirements.  An implementation is said to be unconditionally
  compliant if it is conditionally compliant and also satisfies all the
  Relevant SHOULD, SHOULD IMPLEMENT, and SHOULD NOT requirements.  An
  implementation is not compliant if it is not conditionally compliant
  (i.e., it fails to satisfy one or more of the Relevant MUST, MUST
  IMPLEMENT, or MUST NOT requirements).

I don't know of an "umbrella" document like RFC 2119 that defines
these terms;  someone else may, however.

Mike