Subject: FW: Drive Towards Consensus [was Re: On Extensibility in MARID Records]

From: "Jim Lyon" <[email-masked]>
Date: Fri, 18 Jun 2004 15:46:21 -0700

Sorry, I pushed the "submit" button before filling in the references.
This is a resend, with that fixed.

-- jimbo

As requested by the cochairs, here are two more examples of scenarios
for which the SPF syntax is insufficient.

My initial scenario (feedback tied to individual mechanisms,
http://www.imc.org/ietf-mxcomp/mail-archive/msg02021.html) kicked off
the thread.
The second scenario deals with reputation tied to individual mechanisms.
The third scenario deals with message types tied to individual
mechanisms.

Reputations Tied to Individual Mechanisms 
----------- ---- -- ---------- ----------

(This is mostly a repeat of
http://www.imc.org/ietf-mxcomp/mail-archive/msg02090.html)

Suppose I have a small domain with no reputation.  Suppose I'm a
customer of both MSN and Comcast, and I send some of my outbound mail
through MSN's mail servers, and some through Comcast's mail servers.  As
things currently stand, I'd publish a MARID record like:
    v=spf1 +indirect:msn.com +indirect:comcast.com -all

This authenticates me very well (assuming that MSN and Comcast each do a
sufficient job of policing their internal networks to keep other
customers from masquerading as me).

When we get into the question of reputation, the argument goes something
like:  If you get mail from me through MSN's mail servers, you should
believe it's not spam because MSN does a good job of keeping its
customers from sending spam.  Similarly, if you get mail from me through
Comcasts's mail servers.  The degree to which you as a receiver believe
my mail is not spam is exactly a function of one of my ISP's
reputations.

Using invented syntax, I could say something like:
    v=spf1 +indirect:msn.com/targetrep +indirect:comcast.com/targetrep
-all
which gets the point across nicely.

Contrast this with the case of hotmail's record. Hotmail has more
address ranges for outgoing servers than fits into a single DNS record,
even using SPF syntax. So they publish something like:
    v=spf1 +indirect:a.hotmail.com +indirect:b.hotmail.com
+indirect:c.hotmail.com -all

In this case, the indirections are a mere implementation detail, and
receivers shouldn't attempt to read anything into the fact that they
received a message from the "b" list of servers instead of the "a" list.

As a separate example, consider a domain that sends a lot of spam,
through its own servers. It also sends non-spam, indirected through
Comcast.  It may well publish a MARID record like:
    v=spf1 +mx +indirect:comcast.com/targetrep -all
So, if you get mail directly from this domain's servers, the domain's
lousy reputation applies. If you get mail from Comcast's servers,
Comcast's reputation applies. (This example sidesteps the question of
whether some vigilantes might want to reject a domain's identifiably
legitimate mail because it also sends spam -- a question I don't want to
go into right now).

Message Types Tied to Individual Mechanisms
------- ----- ---- -- ---------- ----------

Many domains send both non-bulk and bulk mail, generally through very
different parts of their organization.  It may be useful to have
annotations on an SPF mechanism that describe the kinds of mail they
send.  For example:
    v=spf1 +mx/bulk +indirect:comcast.com/nonbulk -all

Summary
-------

The common theme among these hard-to-represent-in-SPF extensions is that
the extension says something about an individual mechanism, instead of
saying something about the domain as a whole -- other proposed
extensions that say something about the domain as a whole are much
easier to represent using SPF modifiers (unless the somethings have
internal structure, in which case it gets hard again).

When you put it all together, we need a way to say something new about
an individual mechanism.  Furthermore, we need the ability to say more
than one something new about the same mechanism
(+indirect:comcast.com/targetrep,nonbulk).  Furthermore, some of the
somethings have some internal structure (?all/feedback=xxx,hdrs,p=.001).
[Note that I'm not proposing syntax here -- the commas in this last
example mean something different than the commas in the previous
example.]

XML gives you all of the above.  We don't have any other proposals on
the table that give you any of the above.

-- jimbo

---

From: Douglas Otis <[email-masked]>
Date: Fri, 18 Jun 2004 17:38:27 -0700

> Reputations Tied to Individual Mechanisms 

Do you expect to increase the workload at the MTA and require vetting of
accreditation services based upon the individual?  How can this
accreditation service be sure they are accounting for the right
individual?  Although there is some cost associated in creating a
domain, there is virtually no cost associated with creating a user. This
accreditation service would be left ferreting through forged mail,
spoofed complaints, and fictitious users as the IP checks just the
domain.  I see even more destruction of mail's flexibility in store to
support this innovation however.  Like a bull in a china shop, this
breaks everything around it.

Again, the Fenton proposal allows the needed assurance for such
individual vetting without destroying mail's best features.  Add another
SRV record where instead of using _krs (Key Registration Service) use
_urs (User Reporting Service) for such an individual service or add to
the krs dialog.  This URS could establish a dialog to allow complaints
to be registered, where the domain can then take the needed
administrative action.  Accreditation would be based upon the domain and
not something as unscalable as a user. Of course complaints would
require proper identification using the same key based checks. ; )  Here
the Fenton proposal is infinitely extensible without piling everything
together and would make such a service achievable.  Easily done at the
MUA and does not require breaking mail to accomplish this.

> Message Types Tied to Individual Mechanisms

Sure. I would believe this notation. : )

-Doug

---

From: wayne <[email-masked]>
Date: Mon, 21 Jun 2004 22:33:04 -0500

In <[email-masked]> "Jim Lyon" <[email-masked]> writes:

> Suppose I have a small domain with no reputation.  [snip]
>
> [snip]
> In this case, the indirections are a mere implementation detail, [snip]

I like this concept of using the mechanism that caused the IP address
to be validated as a way of determining the reputation.  This is not
unlike the checking of URLs within email to determine of they point to
spammers websites.

However, I see this as a case of being "not useful because no one will
believe unverifiable claims."

If you are going to do this kind of reputation checking, you should do
it whether there is a "/targetrep" flag or not.  A spammer who
publishes "v=spf1 include:optinrealbig.com -all" will almost certainly
not say to check the target's reputation, but a receiver may find that
very useful to check.  If the various hotmail include: mechanisms
aren't good indicators of reputation, the checking will make no
difference.  However, there is a chance that, due to the CIDR blocks
dedicated to dialups or different countries, the various hotmail
include:'s will make a difference.

The anti-spam systems that judge the urls within an email don't look
for some special tag and for good reason.  The same goes for SPF
records. 

> Message Types Tied to Individual Mechanisms

Again, I see this as a case of being "not useful because no one will
believe unverifiable claims."

> saying something about the domain as a whole 

This is true.  The current SPF spec says that modifiers are position
independent and therefore you can't use them to mark up individual
mechanisms.

I think we could change the SPF spec to allow modifiers to be
(optionally) position dependant and not cause problems for the
installed base of SPF records.  That is, we could say that things like
"redirect=" and "exp=" are position independent, but allow for new
modifiers, such as "targetrep=" and "mailvolume=" to be position
dependant.  So, it would be possible to say "v=spf1 mx mailvolume=bulk..."
and have the mx: mechanism marked up.

I have checked the SPF records that are listed in the Adoption Roll
and my survey of the 1.3 million email domain names.  I could find
none that wouldn't work just fine with this change in the SPF spec.

I'm not sure I really think it is a good idea to make this change, but
I think it is a valid option.

-wayne

---

From: "Hector Santos" <[email-masked]>
Date: Tue, 22 Jun 2004 07:34:30 -0400

----- Original Message ----- 
From: "wayne" <[email-masked]>
To: "IETF MARID WG" <[email-masked]>
Sent: Monday, June 21, 2004 11:33 PM
Subject: Re: FW: Drive Towards Consensus

> believe unverifiable claims."

Jim, I agree with Wayne.  I would not support MARID domain defined
reputation lookup.   In my opinion, reputation lookups should  be a
server-side sysop defined factor because sysops will trust thier setup.  Not
the remote domain. In our current design implementation, sysop defined
reputation lookups is done first before any LMAP method lookup.

I would suggest that Accreditation Service Bureaus (ASB) trying to "buy"
into the MARID concept by augmenting into the validation scheme, are
probably only going to work (feasible) as a "permit" concept.  For
illustration purposes only, the domain policy can look like this:

         (whatever format)  MARID=v1.0 ...  .... +accred:XYZ-######  -all

where XYZ-#### is some combo of site code+permit code assigned by the
Accreditation Service Bureau to the domain.  If the service is also part of
or supports this service, then the ##### number would be part of the lookup
table pointing to the site. example,  ABC-1002-232.  If the server is
supporting this permit, then ABC would be associated within the server-side
lookup table to get the site or lookup method and the entire permit + IP is
used to authorize the sender.

Ideally, the ASB can use a permit format for its members that also includes
other attributes or factors such as:

- Type of Member (Honest Direct Marketer?  Company?  ISP?)
- Expiration Date

but then again, this info could also be provided as part of the ASB query
response packet as it would offer a dynamic or more current ASB member
information.

The sysop will be the final judge as to whether it trust the ASB to validate
its mail sending members.  It (or something similar) is the only way I would
implement this concept into our package.  Not via a sender defined MARID
record with no linkage to the server-side/sysop approved usage.

> I think it is a valid option.

Wayne,  adding unknown extensions before the final ALL directive will break
(create a SPF_UNKNOWN directive error) our current installed based of
Wildcat! SMTP servers with SPF support.   Can't force people to upgrade and
it wouldn't be fair in general to all current SPF installations that may not
agree with an ever evolving "version of the month" SPF format.

Wayne, it is too bad in the "product world", 2nd comers always have the
advantage of learning and improving upon the originators of ideas.    That
is just par for the course. Microsoft has no installed base so they have
tons of redesign leverage to get it right.  SPF does not.  You have an
installed base that can not be ignored.  The worse thing to do here is to
continue to kludge SPF with an ongoing effort to match the XML format.  So
my recommendation for the MARID version of SPF is either a closer format to
XML or its basically time to go to V=SPF2 where you  can afford new well
defined extended SPF format ideas.

This will allow us to maintain compatibility for V=SPF1 systems and also
support the new V=SPF2 MARID systems.  SPF2 will naturally be backward
compatible with the extra considerations for extensions, including the rule
order/position issues.

Anyway, please kind in mind that this on-going MARID extension  "challenge"
between SPF vs. XML may be all for nothing (or less value).  There is no
guarantee implementers will support the extensions.  Maybe a few simple
ones, but not all.  Extensions need to make sense and it can't be something
that will increase the cost of implementation or adoption.

Lets use Reporting for example.

Do you honestly believe that sysops will add overhead to the outgoing queues
to send reports?

If anything, I see this as a possible "commodity" where maybe the IBM's and
the Microsoft's will pay a small fee (penny per report) to ISPs.    But I
sincerely doubt servers will start to send reports all over the place
especially when it becomes a popular extension and sender request.    If we
were to implement it (and probably will) it will only be offered as part of
current Report Generator where the sysop defines the report filters, report
formats, etc to create scheduled reports for admins and management.

We could offer a dynamic reporting system too if that is what the sysop
wants. But in general,  do you see what's going here?

The extensions, especially something like reporting/notification, increases
the design complexity and cost for MARID implementation.  All this "extra
gravy and candy" extension considerations should be secondary to the key
goal of MARID - authentication.   It should be explored, no doubt, but from
the standpoint that its part growth plans for future versions of MARID. But
it should not hinder MARID initial adoption.

Don't get me wrong. We love to write feature rich software. But if we use
this reporting extension idea, we are talking about 1 versus 3 possibly 4
project development efforts:

Core MARID implementation:  Total projects  = 1

    o  WCSMTP
        - New Core MARID options

Core MARID Plus Report Extension: Total Projects = 3/4

    o WCSMTP with more design issues
        - New Core MARID options
        - New Report Control Options
        - New Report storage design
    o WCREPORT (Report Generator)
        - more/new report filtering rules
        - more/new report output rules
    o WCEVENT
        - new MARID Report objects for scheduling
    o WCBILLING
        - just in case the SYSOP want to bill IBM for the overhead.

Never mind the increase beta testing and longer field testing involved.  For
what, a report extension?  Neat idea, but later. Not now.  Lets just make
sure that it will be possible to add the extension and others. I believe
that is essentially the key idea we want.  For now, MARID authentication is
the key to get it into the market place first.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com

---

From: Meng Weng Wong <[email-masked]>
Date: Tue, 22 Jun 2004 10:50:13 -0400

On Tue, Jun 22, 2004 at 07:34:30AM -0400, Hector Santos wrote:
| > > Message Types Tied to Individual Mechanisms
| > > ------- ----- ---- -- ---------- ----------
| > >
| > > Many domains send both non-bulk and bulk mail, generally through very
| > > different parts of their organization.  It may be useful to have
| > > annotations on an SPF mechanism that describe the kinds of mail they
| > > send.  For example:
| > >     v=spf1 +mx/bulk +indirect:comcast.com/nonbulk -all
| >
| > Again, I see this as a case of being "not useful because no one will
| > believe unverifiable claims."
| 
| Jim, I agree with Wayne.  I would not support MARID domain defined
| reputation lookup.   In my opinion, reputation lookups should  be a
| server-side sysop defined factor because sysops will trust thier setup.  Not
| the remote domain. In our current design implementation, sysop defined
| reputation lookups is done first before any LMAP method lookup.
| 
| I would suggest that Accreditation Service Bureaus (ASB) trying to "buy"
| into the MARID concept by augmenting into the validation scheme, are
| probably only going to work (feasible) as a "permit" concept.  For
| illustration purposes only, the domain policy can look like this:
| 
|          (whatever format)  MARID=v1.0 ...  .... +accred:XYZ-######  -all

I agree that assertions of rate limiting, etc. should be
made by accreditation agencies, not by the sender domains.

I say this based on two principles: agency and matching.

Agency means that if the assertions are made by the
accreditor, a self-policing dynamic keeps both parties
honest.  The domain wants to do the right thing or lose its
accreditation; the accreditor wants to make sure the domain
is living up to its standards or it loses its reputation.

Matching means that if the assertions are made by the sender
domain, without preexisting trust, there is no way for a
reputation services to evaluate those assertions.

At http://www.isipp.com/codelist.php we see that accreditors
have already thought through these issues and have created
exactly the kind of data structure that we allude to above.

---

From: "Hector Santos" <[email-masked]>
Date: Wed, 23 Jun 2004 09:15:24 -0400

----- Original Message ----- 
From: "wayne" <[email-masked]>
To: "Hector Santos" <[email-masked]>
Sent: Tuesday, June 22, 2004 1:40 PM
Subject: Re: FW: Drive Towards Consensus

> Caller-ID is dead.  It looks like XML will be dead by the end of the
> week.

I think it is important for the MARID group see what you think.

Also,  I haven't been paying attention to our statistics over the last few
months, but what I see now, June 2004 is the first month that the number of
MCEP sites looked up is nearly equal to SPF sites.

See http://www.winserver.com/antispam

It should be noted the LMAP remains to be a extremely low percentage of the
total anti-spam schema.

Which brings up another important point related to what you like to bring up
much: Total SPF domains currently published vs. MCEP.

It is not the total domains in DNS but the "Personal Mail Community and
Association" of a particular domain that defines the effectiveness of LMAP.

If your domain has a lot of mail from a particular group, and this has a
majority support for SPF, your rate will be high.   But in general, this
will not be the case.

On the other hand, if the top ISPs domains exploited by spammers published
LMAP records, then overall, everyone will benefit.  In addition, if a
community of related servers and her customers published records, they will
all benefit among themselves as been the case with our network of customers.

This all boils down to the #1 benefit of LMAP - protecting your own domain
and as we have learned with our product development while implementing the
LMAP methods,  new SMTP designs should inherently include support for what I
called Local DIPs:  Local Domain/IP associations.

Our statistics shows that atleast 10-15% of spoofers will use your own
domains. That is an instant, highly low overhead optimal rejection at SMTP.
DNS is not required.  And it makes sense. The SMTP receiver should be
looking at the 2821 HELO, MAIL FROM domains to perform a straight forward
local DIP check.  This should be a SMTP BCP or written into the RFC.  Just
imagine in terms of network topology, if every node did a local DIP check,
it would result in  major benefit both locally and as a network with
absolutely NO DNS overhead, locally or network wide.

Anyway,  I hope this isn't a cut and dry issue as you stated "Caller ID is
Dead", etc.  I certainly hope not.   I have voiced my opposition only to ONE
CONCEPT that Caller ID promotes - 2822 validation  I think this is the wrong
direction for both local and network wide operations. This will be
especially the case if a company like Microsoft does go ahead with Caller ID
in her products.  This will force many competitors and the industry to
support it too thus instantly creating a new undesirable higher than normal
PAYLOAD mode of operation.  Yet, I am also on record that if we altered the
SMTP transaction model to better interact with 2822 validation concepts,
then I am for it.

---

From: wayne <[email-masked]>
Date: Wed, 23 Jun 2004 11:28:29 -0500

> I think it is important for the MARID group see what you think.

Well, I don't know if it is normal business practices of Santronics
Software, Inc to take private email conversations public, but I will
stand by what I was quoted saying.

Whether MicroSoft adopts $MARID or not, I really doubt that they will
stick with the Caller-ID XML schema (the context of the private
emails).  Their newer Sender-ID XML schema is both more compact and
more powerful.  So, yes, I think that Caller-ID is dead.

And, yes, it looks to me like XML will be dead by the end of the
week.  I'm sure everyone here is perfectly capable of deciding how
this looks to them.  I really doubt that everyone sees eye-to-eye on
this issue.

> much: Total SPF domains currently published vs. MCEP.

Yes, as I mentioned in my private email to you, I plan on publishing
an update to my 1.3 million email domain name search today, now that
I'm below the 3-post/day suggested limit.

-wayne