Subject: RE: [Asrg] C/R Interworking Framework

From: "Peter Kay" <[email-masked]>
Date: Wed, 4 Jun 2003 15:14:38 -1000

I think its reasonable to think deployment would be rapid. CR is a
relatively new "protocol" if you will. We as a SMTP CR vendor would
comply.

> -----Original Message-----
> From: Vernon Schryver [mailto:[email-masked]] 
> Sent: Wednesday, June 04, 2003 2:01 PM
> To: [email-masked]
> Subject: Re: [Asrg] C/R Interworking Framework
> 
> 
> > From: Yakov Shafranovich <[email-masked]>
> 
> > ...
> > However, if we are defining an SMTP process for CRI 
> protocols, can we 
> > use
> > the VRFY command as a part of an ESMTP extension?
> 
> What about deployment and adoption rates?  Do you want it to 
> work even with MTAs not running your new protocol? In other 
> words, do you want it to work only in theory or to also have 
> a glimmer of a hope of working in the real Internet within 
> the next 20 years?
> 
> 
> Vernon Schryver    [email-masked]
> _______________________________________________
> Asrg mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/asrg
> 
> 
> 

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Yakov Shafranovich <[email-masked]>
Date: Wed, 04 Jun 2003 22:49:54 -0400

At 06:01 PM 6/4/2003 -0600, Vernon Schryver wrote:

> > However, if we are defining an SMTP process for CRI protocols, can we use
>What about deployment and adoption rates?  Do you want it to work
>even with MTAs not running your new protocol?
>In other words, do you want it to work only in theory or to also have a
>glimmer of a hope of working in the real Internet within the next 20 years?

As I pointed out before in a posting to Eric Dean, my vision is a 
multi-stage CRI protocol. The basic protocol would use RFC 2822 headers and 
human-readable email, which would be compatible with current MTAs. As C/R 
systems become more common, they can negotiate to use the other parts of 
the protocol such as an ESMTP extension or MIME fields. An extension 
mechanism would be in place where the protocol can be extended to support 
things like PKI, hash cash, etc. The VRFY command would be part of that 
ESMTP extension which will be used by C/R systems that want to, and not 
used by existing MTAs. Existing MTAs will start off by supporting the basic 
protocol or if they want the other parts.

Additionally, deployment and adoption rates is something that should be 
considered in more detail (this would be something that Eric Brunner can 
help us with). According to D. J. Berstein's SMTP statistics 
(http://cr.yp.to/surveys/smtpsoftware6.txt), the following breakdown of 
major MTAs was present on the Internet in 2001 (out of 958):

401/958 41.85%          UNIX Sendmail
176/958 18.37%          Windows Microsoft Exchange or IIS SMTP
167/958 17.43%  UNIX qmail
57/958  5.9%            Windows Ipswitch IMail

These four accounted for about 84% of all MTAs with the other MTAs were 1% 
or less. Of these, qmail and sendmail account for 59.28% of all MTAs, with 
the Windows ones accounting for the other 24.27%.

IF a CRI protocol is implemented and both qmail and sendmail support it, 
that would mean that a sizable majority of the Internet would support it. 
If Microsoft jumps in, that would increase the effect. The question 
remains, taking the impact of qmail and sendmail, and the propagation rates 
for admins installing the newest versions, how long would deployment take?

Yakov

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 4 Jun 2003 22:44:38 -0600 (MDT)

> that would mean that a sizable majority of the Internet would support it. 

There is a very large difference between the the current versions of
a package supporting something and installations of the package
supporting it.  It is not only that I suspect most SMTP servers are
more than a year behind in current releases, but that having facilities
available in the code is not at all the same as turning them on.

A good case study is SMTP-TLS.   Sendmail and other STMP implementations
have  supported SMTP-TLS for a year or two.  Turning on SMTP-TLS
in sendmail is easy, one you figure it out the OpenSSL documentation.
It is entirely upward compatable and a Good Thing(tm) for a bunch
of reasons.  However, as far as I can tell the vast majority of
the Internet does not support SMTP-TLS and practically none of the
minority that does has done the trivial additional work to make
available their certs.  It's only a slight exaggeration to say that
I see almost as many signs of SMTP-TLS in my logs from spammers as
from legitimate SMTP clients.  In truth I see little of either,
but some of both.

Before you say that everyone cares about spam but not about mail
confidentiality and security and so your protocol would be different,
please say why ISPs that would have to turn on your protocol have
not done what's necessary to find and crush spammers.  Why did
Earthlink reportedly spend months and millions of dollars of
technician and inside lawyer time chasing the Buffalo Spammer while
the outside lawyer pin him in days?

---

From: Yakov Shafranovich <[email-masked]>
Date: Thu, 05 Jun 2003 10:44:51 -0400

>available in the code is not at all the same as turning them on.

My original post ended off with:

"The question remains, taking the impact of qmail and sendmail, and the 
propagation rates for admins installing the newest versions, how long would 
deployment take?"

That is my question exactly, how long will it take for such feature for 
propagate if it were included in sendmail and qmail. Dave Crocker begun to 
address adoption issues in his draft but more work needs to be done. The 
adoption/propogation of features is an issue not just with CRI but with any 
anti-spam protocol or proposal. However, I believe that ISPs and end-users 
have a bigger incentive to turn-on such features due to the amount of spam 
today.

>the outside lawyer pin him in days?

I tend to disagree with SMTP-TLS example. Confidentiality of email via 
SMTP-TLS is something that most end-users or ISPs do not care about. Even 
though every single time that I send an email, the National Security Agency 
or some hacker who hacked a router on the way can listen in, that does not 
affect my every day life. Most users are not affected by SMTP-TLS presence 
or absence since you do not see the NSA or hackers publishing people's 
private email in the National Enquirer. Thus, there is not incentive for 
them to do anything which is not true with spam. Whatever is true for 
SMTP-TLS is also true for PGP and S/MIME, which have not become widespread. 
People do not go through the added burden of setting these things up 
because of lack of incentive.

As for the Earthlink example, I am not sure exactly what you mean. I was 
not aware that Earthlink used outside lawyers which successed in several 
days after spending money on doing it in-house. Please provide some links 
for the story.

Yakov

---------------------------------------------------------------------------------------------------
Yakov Shafranovich / <[email-masked]>
SolidMatrix Research, a division of SolidMatrix Technologies, Inc.
---------------------------------------------------------------------------------------------------
"One who watches the wind will never sow, and one who keeps his eyes on
the clouds will never reap" (Ecclesiastes 11:4)
---------------------------------------------------------------------------------------------------

---

From: Scott Nelson <[email-masked]>
Date: Fri, 06 Jun 2003 10:09:11 -0700

At 03:22 PM 6/5/03 -0400, Kee Hinckley wrote:
>At 10:52 AM -0400 6/5/03, Yakov Shafranovich wrote:
>>The "FROM" field is the one that will get C/R checked, since that is 
>>the mailbox that sent the email. Additionally, the "MAIL FROM" 
>>addresses that is used in SMTP is not intended to this purpose, 
>>rather it indicates a mailbox to which errors should be sent to. It 
>>is perfectly legal and sometimes even recommended in RFC 2821 to use 
>><> for the MAIL FROM.
>
>Are you advocating that C/R systems send mail to the From: address 
>rather than MAIL FROM?  That strikes me as exceedingly wrong.  MAIL 
>FROM is for notification of delivery problems--and that's exactly 
>what C/R is.  Furthermore, we've seen on this list what happens when 
>a C/R system sends to From: instead of MAIL FROM--everyone who posts 
>to the list gets challenged by the person who forgot to whitelist the 
>mailing list.  This is the "On Vacation" message problem all over 
>again.
>

There are basically two choices;
Envelope from a.k.a. MAIL FROM, and 
Reply-To: (or From: if there's no Reply-To:)

If you view a challenge as a DSN then envelope from is the proper
choice.  I'd even go so far as to format the challenge as an
RFC 1894 compliant DSN.  (This has the interesting side effect
that challenging a mailing list is likely to unsubscribe you
automatically)  And like any DSN, I recommend including the 
message-id in the In-Reply-To: and Reference: headers.

If however, you view a challenge as a new message, 
then sending it to the Reply-To: (or From: if there is no Reply-To:)
is the more logical choice.

Personally I feel that challenges are DSNs, so envelope from
is the right choice, but I can see the "new message" 
point of view too.

Scott Nelson <[email-masked]>

---

From: "Peter Kay" <[email-masked]>
Date: Fri, 6 Jun 2003 17:59:17 -1000

Thanks for the clarification. I'm talking about 4.1.1.2 of RFC 2821,
"MAIL FROM".  

"The reverse-path consists of the sender mailbox."  

What I'm saying is that because some CR systems do not set the MAIL FROM
to be the sender mailbox, some CR systems cannot interoperate because of
the reasons stated earlier in this thread. All I'm saying is that for
good CRI, the MAIL FROM should be that of the sender's mailbox.

> Careful perusal of the relevant RFCs shows that there are differences 
> between these fields. Section 3.6.2 of RFC 2822, 
> distinguishes between 
> various "sender" fields:
> 
> o "The "From:" field specifies the author(s) of the message, 
> that is, the 
> mailbox(es) of the person(s) or system(s) responsible for the 
> writing of 
> the message."
> o "The "Sender:" field specifies the mailbox of the agent 
> responsible for 
> the actual transmission of the message. "
> o "When the "Reply-To:" field is present, it indicates the 
> mailbox(es) to 
> which the author of the message suggests that replies be sent."
> 
> The "FROM" field is the one that will get C/R checked, since 
> that is the 
> mailbox that sent the email. Additionally, the "MAIL FROM" 
> addresses that 
> is used in SMTP is not intended to this purpose, rather it 
> indicates a 
> mailbox to which errors should be sent to. It is perfectly legal and 
> sometimes even recommended in RFC 2821 to use <> for the MAIL FROM.
> 
> Yakov
> 
> "One who watches the wind will never sow, and one who keeps 
> his eyes on the clouds will never reap" (Ecclesiastes 11:4)

---

From: "Peter Kay" <[email-masked]>
Date: Fri, 6 Jun 2003 18:24:52 -1000

I agree, a challenge can be seen as a DSN or a new message. My point is
that in either case, the MAIL FROM or REPLYTO should be consistent with
the challenger's email address, otherwise 2 C/R systems will challenge
each other's challenge.

> From: Scott Nelson [mailto:[email-masked]] 
> Sent: Friday, June 06, 2003 7:09 AM
> To: [email-masked]

---

From: Scott Nelson <[email-masked]>
Date: Sat, 07 Jun 2003 11:21:38 -0700

>the challenger's email address, otherwise 2 C/R systems will challenge

If you're sending a DSN, then the From: header could be your address,
but the MAIL FROM should probably be <>.
That prevents a lot of bad things from happening.

Setting the MAIL FROM to your address means that you could end up
challenging a non-existent address, then the mailer_daemon that
bounces the mail back to you.
If the C/R system was really broken, it could challenge bounces 
from a non-existent address endlessly, but in the normal case
this is only twice as much work as you need to do.

---

From: "Peter Kay" <[email-masked]>
Date: Mon, 9 Jun 2003 11:53:50 -1000

Rob's suggestion makes CRI more "I". I like it.

> From: Rob Cameron [mailto:[email-masked]] 
> Sent: Monday, June 09, 2003 10:33 AM
> To: [email-masked]
> It is hard to keep up with the volume of this list.   However,
> here are some of my thoughts on how C/R systems should
> use existing headers.
> 
> Although there are good arguments in favour of issuing 
> challenges to the MAIL FROM envelope header when it differs 
> from the RFC 2822 From: field, I do not think that C/R 
> systems should consider challenges to be delivery status 
> notifications (DSNs).
> 
> (1)  A DSN is logically a communication from the MTA,
> acting like a postal worker.   Some C/R systems will view a
> challenge as a communication from the MUA, acting like a 
> secretary. (In our research, we use the term e-secretary and 
> envisage a protocol to define the future interoperation of 
> e-secretaries to carry out a variety of low-level messaging 
> tasks on behalf of mailbox owners).
> 
> 2) A DSN is designed as a terminal communication, in which
> the MAIL FROM address <> is used to indicate that a reply 
> should not be made.   But the purpose of a challenge, to a
> legitimate sender, is to solicit a reply.
> 
> (3)  A challenge sent as a DSN is likely to be confusing
> to a human who receives it, depending on MUA behaviour.
> 
> (4)  To automatically reply to challenges as DSNs, software 
> that otherwise knows never to reply to a DSN will need to
> be modified to violate this principle.    
> 
> For the purpose of the Internetworking Framework, I don't
> think that we should assume that challenges/responses will only
> be generated to the MAIL FROM field.   Rather, the framework
> should be robust enough to handle challenges/responses to 
> either the envelope MAIL FROM or the From: message field.
> 
> Indeed, I think that the framework should make a statement 
> something like the following.
> 
> "In formulating a challenge message, a C/R system should use 
> the MAIL FROM envelope header and the from, reply-to
> and sender message headers for appropriate purposes.   However, 
> a C/R system must ensure that it is capable of receiving and 
> correctly processing a response sent to any address mentioned 
> in any of these fields."
> 
> Beyond this, I also think that C/R systems should be required 
> to provide full support for message-id, in-reply-to and 
> references headers.  That is, the framework should state that 
> challenges and responses must (not should) provide a unique 
> message-id and must (not should) properly form in-reply-to 
> and references
> headers from prior e-mails in the chain.    By implementing these
> RFC2822 recommendations, C/R systems will give each other 
> valuable information to address both looping and spoofing concerns.
>

---

From: Yakov Shafranovich <[email-masked]>
Date: Mon, 09 Jun 2003 18:04:22 -0400

At 01:32 PM 6/9/2003 -0700, Rob Cameron wrote:
>[..]
>challenges to the MAIL FROM envelope header when
>it differs from the RFC 2822 From: field, I do not think
>that C/R systems should consider challenges to be delivery
>status notifications (DSNs).

What about defining CRI within the "Multipart/Report" MIME type defined by 
RFC 3462 (as follows)?

" The Multipart/Report MIME content-type is a general "family" or 
"container" type for electronic mail reports of any kind."

>tasks on behalf of mailbox owners).

That is a valid point, DSNs are used exclusively for MTAs. However in many 
cases in systems such as Earthlink or Titan Key, the MTA is the one that is 
responsible for the C/R interaction BEFORE the email reaches the MUA. In 
TitanKey's case specifically, the C/R process begins at the SMTP level, 
before email reaches the MUA.

>to a human who receives it, depending on MUA behaviour.

Why would a challenge be sent to a DSN? The challenge itself is being sent 
as a DSN with no return MAIL FROM path, thus it cannot be challenged unless 
the sender starts interpreting the headers.

>be modified to violate this principle.

Correct that's why DSNs might not be the best solution, however we can use 
it as a basis for the CRI protocol.

>correctly processing a response sent to any address mentioned in
>any of these fields."

For systems such as Titan Key where negotiation is done on SMTP level, only 
the MAIL FROM field is used. If its empty, then the system fails. We should 
probably add guidelines that in case empty <> is specified, the C/R system 
should accept the message and use its "From" field.

>to provide full support for message-id, in-reply-to and references
>headers.  That is, the framework should state that challenges
>and responses must (not should) provide a unique message-id
>and must (not should) properly form in-reply-to and references

Agreed.

---

From: "Peter Kay" <[email-masked]>
Date: Mon, 9 Jun 2003 16:33:23 -1000

> From: Yakov Shafranovich [mailto:[email-masked]] 

> should accept the message and use its "From" field.

I'm in agreement w/ all that's being said so far on this thread. We can
also work w/ the above as well and accept empty MAIL FROM. We'll just
have to be really careful here as it's a hole that spammers can use to
get messages through.

---

From: Yakov Shafranovich <[email-masked]>
Date: Tue, 10 Jun 2003 12:40:20 -0400

At 04:21 PM 6/9/2003 -0600, Art Pollard wrote:
>[..]
>
>
>I think as has been mentioned previously in regards to CR systems in 
>general (and I don't remember if it was mentioned in the CRI case), that 
>what should happen is that the messages should be digitally signed by the 
>sender.  The CR system would filter based in the digital signature rather 
>than the FROM address.  Thus it would be quite possible for people to have 
>multiple clients with the same digital signature (one for each e-mail 
>address say) and they would only have to undergo the CR once -- even if 
>they switched ISPs.  Furthermore, it would virtually eliminate spoofing 
>since even if someone were able to obtain a previous copy of someone's 
>mail and a list of all their friends, they still would be unable to spoof 
>the digital signature.  When whitelisting occurred, it would whitelist a 
>particular person's signature rather than their e-mail address.

We discussed the possibility within the CRI framework. However, it is 
important to keep the CRI framework simple and things like 
digital  signatures and hashcash, would be implemented via an extension 
mechanism.

>I'm not sure if the CRI framework provides for this or not as I have a 
>hard time keeping up with things (just as many in this list apparently do).
>
>Is there a brief synopsis of the current state of the CRI framework so I 
>can refresh my memory on everything? (Which would be much better than 
>having to re-read all the CRI messages. ;-)

Hold on for Eric Dean's latest draft.

---

From: Yakov Shafranovich <[email-masked]>
Date: Sun, 15 Jun 2003 13:49:05 -0400

At 11:18 PM 6/14/2003 -0400, Eric D. Williams wrote:

>On Monday, June 09, 2003 6:21 PM, Art Pollard [SMTP:[email-masked]] 
>wrote:
>8<...>8
> > ... The CR system would filter based in the digital signature rather
> > than the FROM address.
>
>A signature that signs what? or do you mean a 'hash' produced using a 
>'senders'
>private key?

I think he meant a digital certificate issued by a third-part certificate 
authority, not a digital signature.

> > Thus it would be quite possible for people to have
> > they switched ISPs.
>Same private key?

Yep - Same certificate.

>8<...>8
> > ...When whitelisting occurred, it would whitelist a
>Caching of the public key?

Yep.

>-e

---

From: "Eric D. Williams" <[email-masked]>
Date: Wed, 18 Jun 2003 21:55:32 -0400

Got it.  Thanks Art.

-e

On Sunday, June 15, 2003 6:56 PM, Art Pollard [SMTP:[email-masked]] wrote:
> At 11:18 PM 6/14/2003 -0400, you wrote:
> >'senders'  private key?
> A digital signature uses a public / private key pair and a hash (typically 
> SHA).  Given the public key then the signature and message could be 
> verified.  The message would be signed with the private key as it went 
> out.  The message's header would contain:
> 
> 1) The digital signature (generated by the public/private key pair and the 
> message)
> 2) The public key.
> 
> The whitelisting would occur based not on the e-mail address but on the 
> public key.  Thus when a new message comes in, the public key would be 
> looked up in the whitelist to see if it is already there.  If it is there, 
> the message can be checked with the public key and the signature to ensure 
> that the proper public / private key pair actually was used to sign the 
> message and that the message has not been altered.
> 
> By whitelisting on the public key and not the e-mail address / 
> username/etc. the user can move between machines and accounts without new 
> challenges as long as they use the same public/private key pair to sign 
> their messages.
> 
> Yep. The public key would be cached and would be used in the whitelisting 
> process.
> 
> -Art
> 
> -- 
> Art Pollard
> http://www.lextek.com/
> Suppliers of High Performance Text Retrieval Engines.
>