Subject: RE: Against Extensibility in MARID Records

From: "Jim Lyon" <[email-masked]>
Date: Thu, 17 Jun 2004 17:36:54 -0700

In arguing against extensibility, John Levine argues that it's a bad
thing (he used the word "chaotic") to have information in a MARID record
that is not understood by everyone.

To rebut this, I note that substantially every successful data format
and protocol contains buckets for information that isn't globally
understood.  For example, consider headers in RFC 2822 mail messages.
The general ethos is that if you see a message that contains a header
you don't understand, you just ignore it.  Without this ethos, many
useful extensions (MIME, for example) would have been impossible.

A similar story applies to the headers in HTTP. Without the ethos of
ignoring what you don't understand, many commonly used features would
have been impossible.

Similarly, in RFC 2821, responses to EHLO contain a list of extensions,
many of which the client doesn't understand.  Again, the ethos is that
you completely ignore anything you don't understand.  Doing so is
exactly what enables SMTP extensions.

Or look at DNS.  In a DNS packet, there's only a single unassigned bit,
and some deployed DNS software gets pissed off if it's not zero.  This
fact, more than any other, has hampered attempts to expand the DNS
protocol.

Or look at the difference between ASN-1 and XML. ASN-1 can describe
anything for which the entire schema is known in advance, but it's hard
to extend a schema after the fact.  XML, on the other hand, has the
clear ability to carry goo whose semantics are not known by older code.
Guess which has been more successful.

Given that we *know* that we'll need more information in the future
(most of us are here to reduce spam, not just authenticate MTAs), it
behooves us to plan the extensibility now.  SPF's current modifiers are
a step in the right direction (they have the ignore what you don't
understand ethos), but as I've argued elsewhere, they aren't sufficient.

The reason they're not sufficient has to do with cases where you need to
attach new pieces of information to particular SPF mechanisms; SPF
merely gives you a way to attach new information to the whole record.
For this same reason, John's suggestion of just having a pointer to a
separate XML document isn't sufficient, either.

-- Jim Lyon

PS: John also continues the misconception that anything that uses XML
will be big enough to require DNS TCP. This just isn't true.  It looks
like most XML-encoded stuff runs about 20% bigger than SPF-encoded
stuff.  The size concern might be persuasive if typical SPF records were
pushing 400 bytes, but they're not.  They're usually around 50 bytes.
The number of characters in the XML framing is just not that big an
issue.  When you're spending 18 bytes to represent an address range, it
just doesn't matter that much whether you frame it with "+a:" and space,
or with "<r>" and "</r>".

---

From: "John R Levine" <[email-masked]>
Date: 18 Jun 2004 02:43:23 -0400

> that is not understood by everyone.

That's not what I said or what I meant, although I can see how you might
have misread it to say that.  What I meant is that it's a bad thing to
have info in the record that doesn't have a consistent and well understood
meaning.

The goal of MARID, the last time I checked, is to do authentication: "mail
with these characteristics really is from us."  SPF and its descendants
are well specified, and every MTA that looks up Sender ID data will use it
the same way to verify whether a message matches a domain's rules.

The proposed extensions I've seen are about reputation "here's why you
should accept our mail", or inbound requirements "here's how you can
persuade us to accept your mail", with a few items private to a domain
"here's how MUAs within our network can decode headers added by our MTA."

It strikes me as poor design to smoosh all that together.  We are just
beginning to understand reputation systems, but one thing that's clear to
me that reputation will mostly be vouched for by third parties, and
there's little a domain can say on its own behalf beyond "look at what
these guys say about us."

Inbound requirements are certainly an interesting area to think about, but
I see no reason to put them in the same database as MARID.  They're looked
up by senders, not recipients, and keyed on RCPT TO or header To: and Cc:
addresses, not the addresses that MARID uses.  Is there an operational
benefit to putting them in the same record?  Not that I can see.

Private domain info is yet a third separate area, again looked up
differently. My MTA delivers mail for a lot of different domains into
overlapping sets of mailboxes, and if I were looking up internal stuff,
I'd want to key it by POP mailbox name, not incoming domain.  Does it
belong in the same record as MARID?  Again, I don't see any advantage to
doing so.

> will be big enough to require DNS TCP.

Once again, I didn't say that, and I have to say I don't understand this
point.  On the one hand I see an argument that we need to be able to load
arbitrarily much stuff into a MARID record, and now it seems like you're
saying that in fact nobody will do that.  If you plan to load all of your
spam related info into the MARID record, the record will get big, and you
have to be prepared to face the consequences of needing vastly more TCP
DNS transactions than before, unless you're saying that there'll be so few
of them that recipients can just ignore any MARID records that don't fit.

If the record is indeed too big for UDP, I hope it's obvious to everyone
here why following a pointer to a URL is the same speed as falling back to
DNS TCP.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Information Superhighwayman wanna-be, http://iecc.com/johnl, Mayor
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 18 Jun 2004 04:23:14 -0400

Jim,  you raise a few key issues I have with the extension concept.

o Possible 3rd party or Microsoft proprietary or undocumented extensions:
o Transport vs. Application/Admin Level MARID feature/extension set:

I would like to state that I support XML for MARID.  But I hope you can
consider my professional opinion on all this. I will express both concerns
in detail.

o Possible 3rd party or Microsoft proprietary or undocumented extensions:

This one presents what I believe is an ethical and moral business dilemma.
I believe in a concept I call "CoComp" - for Cooperative Competition.  It
relates to the usage of common and standard technology that is standard at a
certain level but also allows for proprietary extensions that will not break
the standard usage yet offers a vendor a way to distinguish themselves.   I
presented this CoComp concept in a 1992/93 talk at the first (or 2nd) ONE
ISP CON trade show introducing a XML-like format (although not called XML
back then) for a common mail storage and transport format between the
hundreds of so mail products and the many different formats a product had to
support between heterogeneous mail networks.  This was before the Internet
Email finally trumped all existing mail systems and formats at the time and
rapidly became the industry standard.

As a Windows shop,  I've been involved with the Windows platform since its
inception.

On the positive side, it should go without saying the history of Microsoft
has been to assist the ISV market to move forward in supporting the Windows
platform by providing "tools," in particular helper APIs.

On the negative side, there is a history of new feature sets that were
either initially proprietary or undocumented.  This is one of the main
reasons why we do most of our tools requirement in-house is so that we are
not strategically dependent on specific items under Microsoft control.

Since you are involved in the development of Exchange, I don't think I will
be completely off based for you to have natural tendency and competitive
instinct to explore and implement new ideas that may make the Exchange
product work better than the competition. In fact, it would expected of you
and based on the way the Microsoft patent disclosures has grown,  this could
be an concern.

But others can do the same too.  We can come up with an extension only our
servers understand.

So I am believer in the CoComp concept, yet I can't help thinking XML will
promote this concept for MARID.  Proprietary Extensions from Microsoft or
other vendors could be used as the differential factor between products.
And of course, since Microsoft is the MCEP inventor and major promoter,   I
have a concern that this can be used against other smaller vendors.  I
sincerely hope you can understand this concern.

Please realized that I believe Microsoft and all others have a natural right
to do this and if everyone thinks this is "ok" for MARID where proprietary
extensions promotes usage, then I am all for it.

But it should be well noted that it may will happen with an extensions
concept.  MARID XML will promote extensions that may only work within
certain product lines which brings me to my final
point or concern.

o Transport vs. Application/Admin Level MARID feature/extension set:

Since the day we started the Anti-Spam research early 2003 to finally
address the abusive spoofing of mail, I began with my first time
participation in the IETF mail forum./discussion areas.  It was my hope to
be part of the new direction and help, assist and provide input where I can.
It didn't take long to see what I believe was a fundamental difference in
the philosophical positions many took on how Anti-Spam technology should be
designed and even implemented in current products.

I wish to strongly state that I wish no one take the following the wrong
way. I mean no ill intent or categorization. I just wish to highlight what I
believe is very important because this key difference in product operation
philosophy is what's driven much of the design thinking in many
mail/anti-spam forums with participants from a wide range of disciplines,
including here in my opinion.

The Dynamic vs. Post SMTP operations people run is one of the key
differences driving or molding the thinking, the debates, the discussions
and designs ideas to address the problem.

If you are running a mail operation where mail analysis and/or extensions
are only possible in a POST SMTP manner, you will have a philosophical
different mindset than the person who is running a mail operation where mail
analysis and/or extensions can be dynamically performed at the SMTP
transport level.

How is this related to MARID and Extensions?

Well, MCEP has a POST SMTP dependency in order to work right because of its
high RFC 2822 requirements.   While a mail system can also support MCEP
dynamically by performing the mail analysis at the DATA stage, this has a
major design change implication at SMTP and it also can impact performance
and scalaribility due to increase payloads.

I view POST SMTP analysis as a Application or Admin level methodology where
non-transport or 2822 entities are just one part of the total MARID package.

Similarly, when it comes to extensions, such as this Report Generator
extension concept, I also believe this is an Application or Admin level
methodology.  Not a SMTP level concept.

So my concern is that MARID extensions can be used to force a design
requirement that may not be appropriate at one level or another.

Lets use the report extension that is used commonly for illustration.  I
believe IBM was used as an example for the desirable MARID behavior using
the report extension.

At SMTP,  a MARID client processes a transaction where the RFC 2821 envelope
information points to an obvious spoof and also MARID policy that will
reject the transaction because the IP is not correct.  A report is
requested.

Example:

IP 1.2.3.4
HELO winserver.com
MAIL FROM: [email-masked]

In this example, the HELO domain is spoofed.  winserver.com is our domain.
The IP is not ours.  This is an immediate transport level rejection.

However, to follow MARID, the IBM.COM has a MARID domain policy that request
a report to be sent to IBM help IBM track the abused email addresses.

This level of reporting is outside the whelm of SMTP requirements.  The
application level report generation is now imposed on the SMTP client.
Besides the fact there is more overhead to obtain the report type or format
or extension that provides a report template, in this case, SMTP has a
obvious transaction rejection that should not require any more processing.

My main point here is whatever extensions are invented we need to make sure
that the functionality of the extension is well placed within or separated
from SMTP design requirements.

How you agree with this or not depends largely on your philosophy or current
mode of operation that I was describing above.

If you are running a POST SMTP system, then this isn't so much of an issue
and in this case, I will 100% agree because the report functionality is now
appropriately located in the right system component location - outside the
transport.  A very nice report can be generated with all the time in world
to produce it.

But if you are running a system that is performing dynamic SMTP validations,
this particular report extension is conflictive with SMTP product design and
operation.

I can understand Microsoft design is to use POST operations for total
validation.  So extensions are more affordable - outside the SMTP transport.
Your point of view is well understand - from a Application Design
standpoint.

But for product like ours,  it is a system design dilemma.

Don't take me wrong.  I am looking for a solution to all this.  But
extensions need to be clearly defined and separated per system vs
application component.   You use EHLO for example  All extended features are
for SMTP.  Not outside of SMTP.   You use HTTP request commands.  Its all
within the HTTP protocol.  It is all clearly defined on where the extensions
are used.  That is not to say an extension obtained at the transport level
can not be used to trigger post protocol operations.

But for this need, where in my view, it is highly desirable to block the
high rate of spammers at the transport level,  extensions such as
"reporting" may not be possible or less supported.

-- Hector

---

From: Dave Crocker <[email-masked]>
Date: Mon, 21 Jun 2004 16:49:28 +0800

Jim,

I think your note is particularly useful for prompting us to consider
what the history of Internet does and does not contribute to the
current consideration.

To state my own biases at the start:

1. For all intents and purposes, the Internet has literally no
large-scale experience with interesting policy mechanism operated
among participants lacking prior arrangement.

2. BGP probably counts as an exception, but we should note just how
constrained it is, if we evaluate it as communicating "policy".

3. The DNS is a simple lookup mechanism and our experience with it is
in producing simple records.  both in terms of searching
and in terms of complex records, efforts to make it act more like a
general purpose data based have all failed, or at least been highly
problematic.

4. Reliance on the DNS for core Internet infrastructure operation
dictates that its reliability and performance of those tasks be
guarded vigorously, even at the expense of interesting new
applications.

So with that in mind:

JL> In arguing against extensibility, John Levine argues that it's a bad
JL> thing (he used the word "chaotic") to have information in a MARID record
JL> that is not understood by everyone.

I took John's foundation statement as being:

> One of the main points of MARID, as I understand it, is to develop
> something that can be implemented relatively quickly and will
> interoperate among senders and recipients all over the net.

These match my own understanding and are extremely important as input
to a design process. History with Internet standards that succeed in
matching these requirements dictate considerable simplicity and very,
very limited choice in the core functionality.  Extensibility is
restricted to be outside that core.

JL> To rebut this, I note that substantially every successful data format
JL> and protocol contains buckets for information that isn't globally
JL> understood.

Such information is never part of the core that is needed to get basic
functionality out of the service. Things work quite well without any
of the extensibility.

JL>  For example, consider headers in RFC 2822 mail messages.
...
JL> A similar story applies to the headers in HTTP.

Internet protocols used between participants without prior arrangement
must specify a small, tight core that everyone supports, and that
small tight core must do something very useful. Any extensibility must
be optional value-add.

This is certainly true for email headers, for smtp, and for http headers.

There is a very big and very basic difference between extensibility
support between consenting participants, versus extensibility that
impacts non-consenting participants.

It is also worth noting that extensibility is typically a good thing
only AFTER the core is operational.  For all of the freedom with
RFC733/RFC822 headers, folks didn't take all that much advantage of
the freedom for many years.  And SMTP options did not appear for 10
years.  If you want masses of sites to implement something new
quickly, make sure that development and adoption take a minimum of
effort.

JL> Given that we *know* that we'll need more information in the future

The problem is that we have no serious idea what that information will
be.  We all think we do, but there is no experiential basis for
knowing what is true.  All of the deployed, successful, large-scale
systems use remarkably simplistic schemes.  Any expectations that
there will be deployment of clever, large-scale "policy" analysis
engines is not supported by experience, no matter how appealing such
engines might seem.

I think John Levine's follow-up point, about separation of reputation
versus authentication are also fundamental.

JL> (most of us are here to reduce spam, not just authenticate MTAs), it
JL> behooves us to plan the extensibility now.

Please take a look at the history of the SNMP security field.  It was
an example of a place-holder provided with similar thinking.  Security
obviously would be needed and we would figure out what that meant
later, so let's leave a field that will be define later.

It turned out that the security that was need required a rewrite of
SNMP.

Extensibility is best provided when there is a pretty good idea of the
details that will determine how it will be used.  The problem for the
current discussion is that the "pretty good idea" does not have all
that much empirical basis, nor even community consensus.

JL>  SPF's current modifiers are
JL> a step in the right direction (they have the ignore what you don't
JL> understand ethos), but as I've argued elsewhere, they aren't sufficient.

The problem is that there is actually very little operational
experience USING those values to do Internet-scale mail filtering.

JL> PS: John also continues the misconception that anything that uses XML
JL> will be big enough to require DNS TCP. This just isn't true.  It looks
JL> like most XML-encoded stuff runs about 20% bigger than SPF-encoded
JL> stuff.

I think the real issue is the generic and potentially unbounded nature
of the extensibility, rather than the syntax used for encoding it. Of
course, verbosity is not without its issues, for DNS records.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: "Hector Santos" <[email-masked]>
Date: Mon, 21 Jun 2004 11:23:22 -0400

I would like to state my own bias because of the historistical tradition and
design of all mail hosting and transport systems, not just the internet.
This is important, because no new considerations or design is going to alter
this implementation model.

Foremost, the traditional design for SMTP has been for all systems to follow
two fundamental rules, with the first being:

 Rule #1: Authentication is required for routing. No authentication required
for final destination.

Of course. the principle problem with the "anonymous-like" transaction
behavior with final destination mail has been the growth in the abuse,  a
problem I don't mind saying has been known and documented since the early
1980s.  However, the problem was not significant enough until the internet
was introduced into the public.   Which brings me to fundamental rule #2
that has its origins in the then new email provisions introduced in 1986
ECPA (Electronic Communications Privacy Act) related to email tampering and
altering the "intent of the user:"

 Rule #2: Mail Acceptance must be delivered or bounced.  It can't simply
disappear.

What does this rule mean from a implementation and product liability
standpoint, not just for mail transport systems but for all Interactive
Online Mail hosting systems dominate during the 1980s early mail days?

It meant that a rejection policy was legally acceptable at the user input
(online hosting) or for automated systems, at the transport level.   If the
mail was accepted for posting, then the system had a responsibility to
maintain a) the integrity of the mail (privacy for example), b) that it was
delivered and c) if it was not delivered, it be returned or a reason
provided to the author as to why it was not.

This gave strength the new "system policies" that were now required by
online hosting systems during a LOGIN process telling users what are the
system policies for acceptable mail. But even with a sysop system policy,
the "intent of the user" was an important factor in any tort, defamation,
malpractice suit.

The exception to the 1986 ECPA rule are employer based mail systems.
Employees do not have the same right covered by the 1986 ECPA provisions.

In any case, to make a long story short, this is why I strongly object to
any design concept attempting to address the "anonymous sender" user problem
that imposes or requires the acceptance of the payload for POST SMTP
analysis.

In my view, it will alter the landscape and will open a major "Pandora Box"
in new legal issues. It already has begun (See Scott Ricther current lawsuit
directly related to "intent" and malpractice),  I strongly believe the
promotion of POST SMTP validation is the wrong direction.

Finally, in direct regards to MARID,  if MARID imposes a 2822 validation
requirement, this will seriously limit or incur a higher development cost
for its consideration into our package.   If it is implemented with a 2822
requirement, that will force our design to provide a DATA hook where this
can be done and still follow the traditional and legally responsible design.
In other words, our MARID implementation will not be for POST SMTP
validation operation.

Hope this remark was not off base.  I am strictly stating how we will be
viewing MARID come implementation.

-- 
Hector Santos, Santronics Software, Inc.
http://www.santronics.com