Subject: Differences between CSV and Sender-ID

From: Andrew Newton <[email-masked]>
Date: Wed, 30 Jun 2004 10:28:41 -0400

I have been observing that the recent discussion on CSV in comparison 
to the SPF/Sender-ID solutions have been between a small, consistent 
subset of the participants of this mailing list.  I can only conclude 
that others either do not care about these issues or do not understand 
these issues.  After reading every single message over the past couple 
of threads, I must admit that I do not comprehend many of the points.

Therefore, I'd like to get feedback from others:
    Do you appreciate the difference between a HELO check vs. a 
MAILFROM/PRA/SUBMITTER check?
    Do you understand the semantic differences between a CSV check on 
HELO and an SPF/Sender-ID check on HELO?
    Is it clear to you that CSV has definite security advantages over 
SPF/Sender-ID?

I'm only asking these questions so that we can hopefully refine the 
conversation on this topic.

-andy

---

From: wayne <[email-masked]>
Date: Wed, 30 Jun 2004 11:44:10 -0500

>     MAILFROM/PRA/SUBMITTER check?

Yes.

>     HELO and an SPF/Sender-ID check on HELO?

I don't think there is a significant semantic difference.

>     Is it clear to you that CSV has definite security advantages over

No

> conversation on this topic.

Thanks, I think these questions are useful.

-wayne

---

From: Douglas Otis <[email-masked]>
Date: Wed, 30 Jun 2004 10:09:27 -0700

On Tue, 2004-60-29 at 21:47:42 -0400, Andrew Newton stated:

> I have in my notes an interesting tidbit from the MAAWG meeting in DC:
> Carl Hutzler of AOL said that >70% of spam comes through the mail 
> servers of ISPs.

This is not the case for AOL and is perhaps close to a figure of UCE
advertising major ISP servers and perhaps this percentage of ISPs not
using SAP.  It is also close to this figure with respect to the amount
of UCE emerging from major ISP networks but specifically from their
"servers"? (Although these are often spoofed.)   

On Wed, 2004-06-30 at 07:28, Andrew Newton asked:

> Difference between a HELO check vs. a MAILFROM/PRA/SUBMITTER check?

Checking a sending SMTP against 2822 identities may provide a
confirmation of permissions granted, via a sequence of DNS TXT records,
for a message with this identity to emerge from a particular server. 
These checks may improve effectiveness of filtering, until abusers adapt
or simply "take-out" these checks. Never the less, such confirmation
SHOULD NOT be trusted to have been generated by the identity indicated. 
The mail channel is NOT SECURE and checks at administrative boundaries
may not fully ensure mail had not been injected unchecked.  The methods
of injecting mail is legion and currently 70% of the ISPs do not use
SAP!

Checking a sending SMTP against the HELO 2821 identity (its own
identity) via a DNS record that establishes both authorization and
authenticity provides a reasonable level of assurance of the SMTP
identity.  If there is abuse seen from this server, the administration
of this server can be held accountable.  As a message MUST NEVER BE
TRUSTED without end-to-end checks, the SMTP server offers the only
identity where accreditation is applicable.  It is the administration of
the server that ensures mail is not injected by abusive individuals, and
if so, that such abuse is abated.

HELO/EHLO is the best method to stop abuse through the use of
accreditation of the SMTP administration of policy. MALFROM/PRA
/SUMBITTER offers no reliable identity for accreditation of the SMTP
administration of policy. Without proper SMTP administration of policy,
nothing is abated and there is no means to follow-up on abuse.

> Differences between a CSV check on HELO and an SPF/Sender-ID check on
> HELO?

The CSV check establishes the Authorization and Authentication of the
SMTP server identity with a single DNS record to allow follow-up with
the server administration (a single domain).

SPF/Sender-ID provides correlations between the domain of the message
and SMTP server.  The identity of the server is obfuscated by a large
and complex domain matrix spread over many DNS TXT records that must be
parsed, expanded, compiled, and executed (many domains).

> Is it clear to you that CSV has definite security advantages over 

CSV is needed to ward off an exponential rise in mail abuse.  CSV does
offer security through its ability to establish sender SMTP identity and
thereby provide a method of accountability.

SPF/Sender-ID offers a substantial security risk through miss use of
DNS. It obfuscates the sender SMTP identity and thus prevents any method
of accountability.  Claims regarding assurance of the 2822 identity are
irresponsible with the lack of security existing within the mail
channel.

-Doug

---

From: Greg Connor <[email-masked]>
Date: Wed, 30 Jun 2004 10:15:09 -0700

Thanks Andrew.  Let me try to answer by way of summarizing what I think the 
current areas of disagreement are.  I think that the differences are minor 
and that we will all be able to come to common ground soon.

> MAILFROM/PRA/SUBMITTER check?

My understanding of the issue:

A HELO check can catch some really obvious bad cases (like spam or viruses 
using the receiver's own name) and some obvious good cases (like people we 
want to whitelist).  Also, some folks would like to protect their own name 
from being used in HELO by other MTAs (so that it won't appear in Received: 
lines and cause them to get misdirected abuse reports) -- I think there is 
general agreement that this doesn't happen often and probably isn't core to 
MARID, but it's certainly something that some domain owners want.

CSV also uses HELO to tie a reputation to the sending MTA.  This seems to 
be based on the assumption that good mail comes from good MTAs, and bad 
mail comes from bad MTAs, which some have suggested is not well-supported.

My opinion:

I think checking the HELO *alone* is not an adequate solution to the 
problem set.  I don't think CSV alone is enough to be effective against 
spam coming from big ISPs, where good and bad mail may flow from the same 
MTA.

If the main thing we want out of MARID is to stop people forging mail 
apparently-from and bouncing-to our own domains, a MAILFROM/PRA check is 
going to be required.

> Do you understand the semantic
> differences between a CSV check on HELO and an SPF/Sender-ID check on
> HELO?

My understanding of the issue:

Mechanically, CSV and SPF are both capable of checking HELO.  Speaking only 
about the mechanisms themselves, here is a quick overview:
 - If the HELO name is different from the base domain people use for email 
addresses,
    - you need an SPF record for each MTA name (HELO name) in addition to 
the mail domain's SPF record.
    - CSV is similar - you need a SRV record for each full name used in 
HELO - though since it doesn't check any other identities, there won't be a 
CSV record for the mail domain (unless it is a negative record saying 
"nobody may use this domain for HELO")

 - If the MTA name is also used as a HELO name for one of the MTAs
      - In most cases the existing SPF record should be sufficient, since 
it probably includes that MTA.  (If not, for example if the machine calling 
itself "example.com" is a web server and not one of the MX mailers for the 
domain, then the bounces or notifications coming from this machine are 
already blocked by SPF).
      - If the same name is used for both HELO and email@domain, the SPF 
record should be a union of the two policies.  All machines able to use 
this name in HELO and all machines able to send outgoing mail from this 
domain should be listed.  It has been proposed that SPF add a macro that 
sites could use to post a different HELO policy than its PRA policy, but in 
practice most users will just go with the merged policy (if they are 
different at all)

Semantically, there is some difference in the understanding between what 
the CSV check means, and what the SPF+HELO check means.  The big difference 
between CSV and SPF+HELO is that CSV's supporters have attached a large 
amount of significance to the record, since it is ONLY used for HELO names, 
it seems a lot clearer that "These are the servers I control and I'm the 
best domain to address your problems to".  This is more about what you read 
into the result when you do the check, and not really a factor of the 
mechanism.

CSV doesn't have an "unknown" mode, and I think some CSV supporters have 
said this is a good thing.

My opinion on this:

In the case where you want to permit users to send outgoing mail from 
another domain, but you don't control their servers and you don't 
completely trust them, I would strongly recommend admins to use "unknown" 
status (like ?include or ?ptr).  If you add to your SPF record using 
+include:comcast.net, you are in effect saying "Anyone at comcast.net can 
claim to be me and the mail is guaranteed not forged".

It would be better to use ?include:comcast.net or ?ptr:comcast.net. That 
way the mail from those domains is still allowed, but not "guaranteed" to 
be from you.  If the result comes back unknown, you can't attach reputation 
or whitelisting to that transaction, you just have to proceed in "legacy" 
mode.

That is an important part of why I believe most domains can use the same 
SPF record for HELO purposes and for PRA/MAIL FROM.  I guess trying to 
mingle the two modes without explaining the ? vs. + is probably confusing 
to people :) but I still believe that the same tool can be used for both 
jobs.

If people are really worried about others forging their name in HELO, but 
don't want to apply stronger protections to their PRA/MAIL FROM with the 
same name, there will be a macro-based way to give a stronger HELO policy 
than PRA/MAIL FROM policy, but I think HELO forgery is not enough of a 
problem for this to be interesting to most users.

>Is it clear to you that CSV has definite security advantages
> over SPF/Sender-ID?

My understanding of the issue:

The recent discussion over possible DDOS involving SPF doesn't really have 
to do with HELO checking specifically.  CSV supporters assert that because 
it doesn't support redirection, includes, macros or exists mechanisms, that 
this is a good thing.  There is general agreement that the smaller problem 
of HELO checking can probably be solved with a smaller set of tools, but 
folks differ on whether they want two sets of tools, or one tool capable 
for both jobs.

Therefore we should separate the issue of "whether SPF has problems" 
completely from the question of "whether CSV has something SPF doesn't". 
If SPF has problems, they should be addressed.  If CSV has features SPF 
doesn't have, the group needs to decide whether to advance CSV as well, or 
try to adapt SPF to include those features.  There is some advantage to 
end-users if we are able to produce a single RFC that can address both 
problems.

My opinion:

CSV may have a better security story, but I believe this is a direct result 
of deciding to include fewer features and less flexibility.

Regarding DDOS concerns, I think they can be solved by placing some limits 
on the amount of recursion possible and the total number of queries needed 
per mail message, and that should satisfy most concerns.  We should 
probably also do a bit more testing to see what such an attack might do, at 
the very least because then we can recognize it when it happens and we know 
what to do :)

I think there is enough consensus in the group that we need to protect PRA 
and/or MAIL FROM, and that HELO is of secondary importance.  Not everyone 
agrees with this, but I think a majority of folks think that HELO checking 
by itself is not enough.  So, if we are going forward with PRA/SenderID or 
something like it, it should be easy enough to adapt it to HELO checking as 
well.  I don't believe there is enough support in the group to adopt CSV as 
a second proposal, if SenderID can be easily modified to do both jobs.

If we separate the questions of "SPF is bad because X" and "CSV is great 
for Y", we will probably decide that SenderID needs some minor changes to 
be fixed, and that CSV provides not enough incremental value to proceed as 
an independent proposal.

Now, this is NOT to say that CSV is anything bad.  Not at all.  CSV authors 
and proponents should be proud of the current proposal, even if it doesn't 
make it to RFC. It has done an excellent job of promoting discussion and 
alternative ideas, and the latest Unified SPF is a direct result of that. 
The idea of using the HELO name as a whitelist/reputation identity is a 
good one.  CSV does what it sets out to do rather well, and if that were 
the complete problem set we were tasked with, CSV might just win.  Because 
people want to protect their PRA/MAIL FROM, and Sender ID is getting some 
support, it is to the group's advantage to try and merge them into a 
consistent, cohesive product, but I don't think this diminishes the value 
of CSV and the ideas it has brought to the table.

Sorry this has got so long.  Let me just say again, I think we are close to 
agreement and that the areas where we agree are wide compared to the 
remaining disagreements.

Thanks
gregc
--
Greg Connor <[email-masked]>

---

From: Douglas Otis <[email-masked]>
Date: Wed, 30 Jun 2004 12:48:24 -0700

> MARID, but it's certainly something that some domain owners want.

There is no means of stopping the problem without first identifying the
problem.  Any entity can be held accountable provided there are accurate
means available.

> mail comes from bad MTAs, which some have suggested is not well-supported.

Through use of dynamic lists identifying abusive MTA servers (this does
not include major ISP servers), more than 80% of the abuse is blocked. 
>From this, I must suspect this suggestion is in error or misunderstood. 
If this list could be fully vetted, it could be made available.  Hence
the efforts directed at CSV. : <

> spam coming from big ISPs, where good and bad mail may flow from the same 

This underestimates the potential enabled by CSV.  If CSV is acted upon,
the next battle will be to get ISPs to utilize SAP. MAILFROM/PRA/
SUBMITTER offers NO help whatsoever in this area however!  An accurate
accreditation system would help convince ISPs it would be in their
interest to implement SAP systems.  With SAP, the abuser can be
identified and disabled to protect their accreditations. 

> apparently-from and bouncing-to our own domains, a MAILFROM/PRA check is 

First, you need something that can scale.  MAILFROM/PRA/SUBMITTER does
not scale.  Fenton's "Identified Mail" does.  If ensuring mail identity
were critical, then an end-to-end solution that can reasonably ensure
the identity of the sender is needed.  MAILFROM/PRA/SUBMITTER will be
defeated owing to the general lack of SMTP security and lack of policy
enforcement as to make such assurance irresponsible.  CSV would allow a
measure to judge the level of SMTP security that can not be deduced
without accurately identifying the sending SMTP.

> Mechanically, CSV and SPF are both capable of checking HELO.

As data to comprise such checks are based upon a wholly different
paradigms, these checks do not provide comparable results.  CSV answers:
"This domain is administratively accountable for this SMTP outbound
server."  SPF answers: "This matrix of domains employ this server." The
domain administratively accountable remains unknown with SPF.  

In the current core Sender-ID draft the only mention of EHLO is as
follows:
: Is an SMTP client authorized to use a particular domain name in its
: SMTP EHLO command?  [CSV] attempts to answer this question.  It
: suffers from the fact that the EHLO name has a tenuous relationship,
: at best, with the contents of any mail message.
 
> Speaking only about the mechanisms themselves, here is a quick overview:

Why add this back into the "core" proposal?  CSV and Sender-ID are
totally independent functions that need totally different information.

> it probably includes that MTA.

The fact that the MTA may exist as a subset of the SPF information
overlooks that this information is obtained using different labels and
that being within a subset of the SPF list offers nothing useful for
accreditation. Keep CSV "as is" and keep this check excluded from
SPF/Sender-ID.  Treat these functions as orthogonal.

> (If not, for example if the machine calling itself "example.com" is a
> web server and not one of the MX mailers for the domain, then the
> bounces or notifications coming from this machine are already blocked
> by SPF).

If the SMTP server does not belong to the domain, then NO mail will be
accepted per CSV.  Accreditation could offer extended permissions to
enable a greater diversity of mail to emerge, but as SPF/Sender-ID does
not scale, a comprehensive list of allowable domains over any SMTP
server may require some other out-of-band system, and not DNS.  A
conventional method would be to publish a _service._tcp.domain SRV
record that would locate such an out-of-band service.

> record should be a union of the two policies.

This overlooks the purpose of the CSV mechanism entirely and overlooks
the protections offered by a much simpler record.  Again these to
schemes do not offer the same information.  Just because CSV may be seen
as a subset of SPF, such a record may be located with a different label
but must contain different information (less, far less information.)

> All machines able to use this name in HELO and all machines able to
> send outgoing mail from this domain should be listed.  It has been
> proposed that SPF add a macro that sites could use to post a different
> HELO policy than its PRA policy, but in practice most users will just
> go with the merged policy (if they are different at all)

Why preface the use of a simple mechanism with the adoption of an
incredibly complex scheme that is yet to be fully documented in any
Internet Draft.  Now you wish to have a macro to post this information? 
Why add such a significant amount of time to an issue that can and
should be addressed separately?   

> into the result when you do the check, and not really a factor of the 

It would seem you have completely overlooked the need for a valid
identity for accreditation.

> said this is a good thing.

CSV does allow "no statement" to be made regarding the sending SMTP
server.  How this gets interpreted is left to those that make policy.

> to people :) but I still believe that the same tool can be used for both 

Why mingle two records that are naturally accessed using different
labels anyway?  You have not justified this merger.  Saying the SPF
syntax can become more obtuse to _help_ differentiate the "group" that
_may_ be accountable does not improve the use of CSV.  CSV is designed
to be resilient and lightweight.  

> problem for this to be interesting to most users.

Add another layer to the cake, or is it stuff another nut into the
cheek? : )

> folks differ on whether they want two sets of tools, or one tool capable 

Leatherman or hammer? : )

> end-users if we are able to produce a single RFC that can address both 

Each of these two problems represent significantly different time
lines.  There is no technical merit combining these two functions where
even the current MARID "core" document has completely excluded
consideration of the EHLO domain.

> be fixed, and that CSV provides not enough incremental value to proceed as 
<snip>

Here we differ greatly in opinions.  I do not think that CSV by itself
is a complete solution.  CSV with accreditation is a potent solution
that has much greater promise.  I have great concerns regarding
expectations of using DNS to publish a correlation of 2822 identities
with all servers allowed to carry such messages.  DNS was never designed
to have a simple query offer such a comprehensive result.  This is
venturing into regions where there are many looking for a chink in the
armor.  Why expose yourself to harm if it is not needed?  A security
concern offers great justification for keeping these two proposals for
two separate problem sets, separate.

-Doug

---

From: "Hector Santos" <[email-masked]>
Date: Wed, 30 Jun 2004 16:24:58 -0400

Thanks for this Andrew.

> MAILFROM/PRA/SUBMITTER check?

A resounding yes.

Yes.

>  Is it clear to you that CSV has definite security advantages over

Yes, but more true if you said "offers some security advantages over.."  But
what is very important to understand the two are important.  They can work
together.  I hope to show this in my trust analysis (second point below).

I will outline my reasoning to help explain my answers to your questions:

o In general (CVS or otherwise), a validation at HELO offers an inherent
presumption of M2M (machine to machine) trust concept.  I agree with the
concept especially in internal networks where routers are involved. This may
include analysis of the first hop (assuming no tampering) to help in the
generation of the trust in non-internal routed situations.   It has to do
with the circle of trust idea.  Dave Crocker has provided excellent writes
up/presentations on the concept 1-2 years ago or more.  Its on his web site.
However, I believe this is all based that the enforcement or following of
the traditional SMTP operation of:

        - Non-Authentication/Trust is not required for Final Destination.
        - Authentication/Trust required for routes

Otherwise you have open relay situations and security violations in internal
networks.

o I have a LMAP trust analysis showing how HELO validation can help change
the results of a SPF MAIL FROM validation.  I hope to clean this up for
publication here.  CSV was not available at the time of this write-up, but
it
wasn't necessary per se.  It is an trust analysis showing how HELO LMAP
results can prevail, alter or change or not, the results of a  MAIL FROM
LMAP results.

o The problem with CVS specifically, its in augmentation of a 3rd party
concept (accreditation). This introduces a barrier for implementation
(atleast for me).  I prefer to work with a SMTP based functional protocol,
not a system-wide, product or application protocol.

o The problem of PRA/SUBMITTER or any related idea (including CVS first hop
analysis) is its dependency on post 2821 operations.  Basically, what it
means from a implementation standpoint, it is outside our design scope.  We
push all responsibility for post SMTP mail analysis to the system operator
(sysop) - by product design which has its beginnings related to product
liability.  In short, if the sysop wants to use "SpamAssassin, or SendedID",
he is on his own here.

o To help address the growth of 2822 analysis and also keeping with the
design of the system, we offer a DATA hook so that they can run a mail
content analysis (like SpamAssasin or McAfee AVS) by hooking it into at 2821
DATA state.  The key reason for this was to help solved the huge growth
problem of having "good but spoofed return path address" perpetuated with
the growth of SORBIG-based email viruses. This exploitation has taken on a
Code Red Principle mentality of attacking POST SMTP systems and all the
legacy systems that operate in this mode,  to help the virus with 2nd tier
bounce distribution.   Running a mail content analysis/rejection at DATA
helps this situation because it eliminates the second level (bounce mail)
assault.

Basically what the above says from my standpoint, is that I think SPF can
help provide additional HELO validation security with a modification of its
SPF lookup specifications.   CSV can help add more weight with first hop
analysis.

But all overall, the merging of other concepts such as accreditation is not
a
welcome idea for me as it makes the whole issue of providing a customer a
reliable anti/spam security product or feature more dependent on external,
more likely than not, profit driven service bureaus.   I wish to keep and
implement a solution purely on technical and SMTP/internet standard
compliancy grounds.  Not saying it can't be offered, but it will make CVS
less adoptable in my view.

The same is basically true with SenderID.

In this case, it requires a change in software and compliance by VALID
systems.  Therefore, unless we are ready to enforce the idea non-compliance
will add some level of acceptable mis-trust and thus rejection and/or
scrutiny, I don't see it working very well at all.  To support it, we then
go back in circles with encouragement of payload transmission.   Sure, it
can be supported at the DATA hook, but it definitely not something I can't
see it putting any weight on the trust concept:

- A spammer complies.  It doesn't address the validity of the
address?
- A spammer does not comply.  It could be a valid legacy system?

My prediction with SenderID or any promotion of a new world-wide standard
that requires 2822 analysis:

Spammers/Hackers will discover that there is new growth of Microsoft
Exchange servers in the market place will accept payload. They increase the
payloads to levels that is just under a system limit (ours is 5 megs per
email I believe).

The goal?

Frustrate systems to turn off the option due to new scalability and load
balancing problems.   Systems that don't support it won't have this problem.
Systems that rely solely Microsoft's SenderID are in big trouble.  Systems
that rely on other 2822 validation ideas suffer too because they will see
the growth of higher payloads on their systems too.

Of course, I reserve the right to be completely wrong.  Nonetheless, right
or wrong, it is a valid engineering design consideration that helps reduced
consumers risk.

Ideas like SendedID and any other 2822 validation logic "cries" for a change
in the SMTP transaction protocol, basically a new command to provide the
headers.  If this was to happen, I'll be among the first to jump on board.

In my view, MARID can probably be assisted with other ideas that may help
address the current items in discussion:

- How will the Spammer React?

- Where are we enforcing change?  Are the operator or spammer?  This is
related to Meng's reflection of the "Chaos Theory" - Chaos promotes
change/adaptation.  Well, we are changing, but is the spammers?  How?

- Can I lower of cost of operation (TCO) by using CVS, SPF, SPF2?  Or is
this still going to be a problem, and if so, this MARID help in the
tracking/audit process which is required by CANSPAM if you wish to use the
new law against a spammer.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com