Subject: On Extensibility in MARID Records

From: "Jim Lyon" <[email-masked]>
Date: Wed, 16 Jun 2004 09:37:41 -0700

I've been asked to defend the proposition that MARID records need more
extensibility than can be easily afforded by SPF syntax.  This thread is
an attempt to do so, using two possible extensions.  To paraphrase Mark
Twain, I apologize for writing such a long email -- I didn't have time
to write a short one.

To set the stage, let's consider Rich Segal's comments at the interim
meeting.  He says IBM will publish a record something like:
    v=spf1 +a:xxx +a:yyy +a:zzz -exists:%{i}.rbl.com ?all
This says that if a message comes from one of the servers he knows
about, it's good.  If a message comes from a blacklisted address, it's
bad.  Otherwise, he's not sure.

Now, he would like to be able to change the final "?all" to "-all", but
he's not sure he's found all of the servers that send mail on IBM's
behalf.  Therefore, he needs feedback about mail that matches the "?all"
so that he can refine his list.  At first blush, he could say something
like:
    v=spf1 +a:xxx +a:yyy +a:zzz -exists:%{i}.rbl.com ?all
feedback=[email-masked]

But that begs the question of what feedback he wants.  In order to get
feedback on mail that matches the "?all", the "feedback=" needs to bind
to the "?all".  So, we could invent syntax like:
    v=spf1 +a:xxx +a:yyy +a:zzz -exists:%{i}.rbl.com
?all/feedback=[email-masked]
(note the slash).  This would be a step forward.  It's also possible
that their legal people would like feedback about mail that matches the
"-exists", so that they can sue the offenders.  So maybe their record
ends up looking like:
    v=spf1 +a:xxx +a:yyy +a:zzz
-exists:%{i}.rbl.com/feedback=[email-masked]
?all/feedback=[email-masked]

Now, for purposes of refining their "?all", they may only want headers
of messages that match, but for prosecuting abusers, they may need the
entire message.  So they might want to publish something like:
    v=spf1 +a:xxx +a:yyy +a:zzz
-exists:%{i}.rbl.com/feedback=[email-masked],ret=full
?all/feedback=[email-masked],ret=hdrs

Now, IBM may very well want to monitor the activity of the third party
they contract with that uses address zzz, so that mechanism might look
like
    +a:zzz/feedback=[email-masked],ret=hdrs

All of the above has had to do with a hypothetical "feedback" extension,
with various parameters.  It's also possible that there will be
extensions that classify the mail coming from various sources.  For
example, if the address range +a:zzz is used by IBM for sending bulk
mail, the mechanism may want to say
    +a:zzz/type=bulk

Putting the two extensions together, we get
    +a:zzz/feedback=[email-masked],ret=hdrs,type=bulk

Now we're really pushing the limits. In the above example, "ret=hdrs"
modifies "feedback", which modifies the mechanism.  But "type=bulk" just
modifies the mechanism.  So the above example has syntactic ambiguity
built in. We need a heavier structure to resolve the ambiguity: brackets
or parentheses or tags or some such.  We could get there with
parentheses by writing something like:
    +a:zzz(feedback:[email-masked](ret=hdrs))(type=bulk)

But now we've already left the realm of a simple regex parser -- the
possibility of nesting implies at least a push-down parser.  If we ever
expect to be able to extend SPF records, we need to define the
extensible syntax now.  (Not the details of "feedback" or "type", but
the syntax for telling where each extension starts and ends).  That
syntax needs to account for special characters, like email addresses
that contain an "=" or ":".  Processors that we deploy today need to be
able to parse (and discard) this stuff, even though things like the
"feedback" extension may not be defined for another year or two.  By the
time we've got enough extensibility, the complexity of a conforming
parser approaches that of XML.

XML is rich enough to handle all of the above.  The spec is well
debugged.  XML Schemas include a mechanism for defining what's legal in
a document, even if part of what's legal is tags that haven't been
defined yet.  There are dozens of extant parsers -- for Linux, for Unix,
for Windows, for S/390.  Written in C, C++, Java, Perl, Python, C# and
probably even Intercal.

So, XML fits the requirements that I've outlined above, and it's the
only thing that I know of that does.  Of course, it would be possible to
design an SPF-like language that meets these, but "possible" isn't
enough.  It hasn't been done, and I doubt that we would get it right on
the first try if we did attempt it.

---

From: Jonathan Gardner <[email-masked]>
Date: Wed, 16 Jun 2004 10:26:49 -0700

Not quoting as I couldn't find anythine appropriate to quote. Read the 
previous email in its entirety for context.

It makes me wonder whether we would really want to add complexity like the 
above. I believe the K.I.S.S. (Keep It Simple, Stupid) applies here. Yes, 
XML can handle it. So could any other way of serializing data structures, 
however. SPF syntax could do it as well, as you have shown.

We already have serious problems trying to get HTML and CSS to behave the 
same in both IE and Mozilla. I would hate for that to transfer into 
something that should be simple. In the email world, it isn't X vs. Y. It's 
A vs. B vs.  ... Z. We have to make sure that every implementation behaves 
in exactly the same way. There is no room for ambiguity or contradicting 
results here. We can't get two web browsers to agree on a simple XML 
document. How can we expect to get 30+ implementations to agree on a 
different kind of XML? Extensibility is actually a bad thing, because we 
can't extend the syntax without upgrading every implementation out there.

Let's return to the original intention. We want to establish authority for a 
server to send email for a domain. That's it. We aren't going to be sending 
emails for every email we receive for a domain; that doesn't scale, and you 
get into a situation where you can actually start recursive mails. I know 
that what you are describing is hypothetical, and I know that you would 
propose ways to work around these problems. But the point is that anything 
beyond establishing authority should be out-of-bounds and beyond the scope 
of this group.

The original SPF syntax allows domain owners to express which servers are 
allowed to send email beautifully. Sure, it can't express every situation 
out there, but it expresses the vast majority of cases remarkably well. In 
the cases where the SPF syntax doesn't do a good job of expressing which 
servers are good or bad, the domain owners should probably find some way of 
simplifying their setup. There are already mechanisms where you can have a 
specific server calculate whether to deny or allow a server to send email 
based on all the information that's available when that decision needs to 
be made. What more can you possible want, without leaving the area of 
authentication?

- -- 
Jonathan M. Gardner
Mass Mail Systems Developer, Amazon.com
[email-masked]

---

From: Andrew Newton <[email-masked]>
Date: Wed, 16 Jun 2004 14:25:35 -0400

> parentheses by writing something like:

My first stab is:

+a:zzz/hdr-feedback-bulk=[email-masked]

or

+a:zzz/feedback-policy="dns:feedback.ibm.com?class=TXT"

-andy

---

From: "Michael R. Brumm" <[email-masked]>
Date: Wed, 16 Jun 2004 12:34:44 -0600

Jim Lyon wrote:
>extensibility than can be easily afforded by SPF syntax.

Well, it has been argued (successfully, IMHO) many times that the architectural advantage of SPF's non-extensible syntax is one of its best attributes.

By freezing the feature-set, SPF is more stable, easier to understand and implement, and more uniformly practiced. Less ambiguity provides the authors of SPF records a larger degree of assurance that their records will be parsed correctly. Frozen syntax and semantics make MTA implementations easier to write and debug, and more likely to evaluate the records uniformly.

A hidden problem with extensibility is the fact that non-uniform practices can emerge with multiple competing mechanisms fighting for dominance (imagine browser on SMTP). With a frozen spec, things are a lot more predictable because new mechanisms are thoroughly debated and vetted before being released uniformly as a new version.

In general, extensibility is not a benefit to MARID.

Michael R. Brumm

---

From: Andrew Newton <[email-masked]>
Date: Wed, 16 Jun 2004 15:02:54 -0400

I would appreciate it if this thread were dedicated to the technical 
discussion of the variability of SPF vs. XML, as Jim is trying to show. 
  If you have comments about the need or requirement for extensibility, 
then please use another thread (preferably the one Greg started on XML 
unity).  This does not mean your opinion is invalid.  I would just like 
these threads separated so people can track them more easily.

-andy

---

From: Douglas Otis <[email-masked]>
Date: Wed, 16 Jun 2004 11:32:28 -0700

On Wed, 2004-06-16 at 09:37, Jim Lyon wrote:
> extensibility than can be easily afforded by SPF syntax.  This thread
> is an attempt to do so, using two possible extensions.  To paraphrase
> Mark Twain, I apologize for writing such a long email -- I didn't have
> time to write a short one.
> 
> Now, he would like to be able to change the final "?all" to "-all",
> but he's not sure he's found all of the servers that send mail on
> IBM's behalf.  Therefore, he needs feedback about mail that matches
> the "?all" so that he can refine his list.  At first blush, he could
> say something like:
> 
> feedback on mail that matches the "?all", the "feedback=" needs to
> bind to the "?all".  So, we could invent syntax like:
> 
> that their legal people would like feedback about mail that matches
> the "-exists", so that they can sue the offenders.  So maybe their
> record ends up looking like:
> 
> All of the above has had to do with a hypothetical "feedback"
> extension, with various parameters.  It's also possible that there
> will be extensions that classify the mail coming from various
> sources.  For example, if the address range +a:zzz is used by IBM for
> sending bulk mail, the mechanism may want to say
> 
>     +a:zzz/type=bulk
> 
> modifies "feedback", which modifies the mechanism.  But "type=bulk"
> just modifies the mechanism.  So the above example has syntactic
> ambiguity built in. We need a heavier structure to resolve the
> ambiguity: brackets or parentheses or tags or some such.  We could get
> there with parentheses by writing something like:
> 
> possibility of nesting implies at least a push-down parser.  If we
> ever expect to be able to extend SPF records, we need to define the
> that contain an "=" or ":".  Processors that we deploy today need to
> be able to parse (and discard) this stuff, even though things like the
> "feedback" extension may not be defined for another year or two.  By
> the time we've got enough extensibility, the complexity of a
> conforming parser approaches that of XML.
> 
> 
> debugged.  XML Schemas include a mechanism for defining what's legal
> in a document, even if part of what's legal is tags that haven't been
> defined yet.  There are dozens of extant parsers -- for Linux, for
> Unix, for Windows, for S/390.  Written in C, C++, Java, Perl, Python,
> C# and probably even Intercal.
> 
> only thing that I know of that does.  Of course, it would be possible
> to design an SPF-like language that meets these, but "possible" isn't
> enough.  It hasn't been done, and I doubt that we would get it right
> on the first try if we did attempt it.

Just as XML is extensible, it could be done in a manner that requires
cooperation of the standards process before such changes are possible. 
Lock down what is contained in a related RR by making headers virtual
and standalone and bar additional definitions.  Change the possible
content by changing the version declaration to reference a new and
expanded virtual header.  Adding features that "might" be useful is
emblematic of "everything including the kitchen sink" that seems to
prevail when text and XML is considered.  DNS is not an http server.

It seems to be a wild claim an organization or ISP will be able to
define in this single record all possible avenues employed by their
users.  The most an open list provides is a check mark added next to the
mail subject line. 

Can these extra functions be done using other methods?  

Claiming records may be chained or some may use TCP does not help.  If
such innovation can not be confined to a simple expression with
constrained results, then add a single _mail-vouch._tcp.my-domain SRV
record and point to a real http server where the SRV record also
declares the port to be used.  Now all things web-like become possible. 
Use dynamic techniques to allow users to add or subtract from their list
automatically.  With this done, it will not cause great concern for the
health and function of DNS or MAIL.  This can happen at the MUA where it
can scale and perhaps even cache much of the information as much of this
will likely be repetitive at the user level.  A mail marking scheme does
not improve the basic mail infrastructure to be worth considering at the
MTA level.

-Doug

---

From: Greg Connor <[email-masked]>
Date: Wed, 16 Jun 2004 16:15:25 -0700

--Jim Lyon <[email-masked]> wrote:
> an attempt to do so, using two possible extensions.
>
[example of wanting extra data to be attached to a mechanism]
> debugged.

OK, for expedience I will concede that XML is better suited to extending 
the data in various directions, like a tree.

I guess I would like feedback from the group as to whether we see this 
particular type of extensibility as a "requirement", or whether there is 
feeling that name=value pairs placed inline is enough.

Also, did you want to say a few words about other things that might go in 
_ep besides out nodes?  Now would be a good time to bring that up too if so.

I will leave it to the group at large to chime in as to whether the 
"multi-dimensional extensibility" requirement Jim stated is more important 
than size, overall simplicity, etc.
--
Greg Connor <[email-masked]>

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Thu, 17 Jun 2004 13:10:38 +0200

On Wed, Jun 16, 2004 at 09:37:41AM -0700,
 Jim Lyon <[email-masked]> wrote 
 a message of 260 lines which said:

> in a document, even if part of what's legal is tags that haven't
> been defined yet.  There are dozens of extant parsers

You certainly know everything I'm going to write but many people on
the group are probably not XML experts, so I would like to emphasize
one point: XML is modular and not all parts have the same status,
speaking from an engineering point of view.

XML, the core, is indeed well debugged and understood, has efficient
(time and space) parsers for every operating system, every licencing
preference and every programming language.

But you say more: you talk about the rest of XML's very rich
world. You mention W3C's XML Schemas, someone else mentioned
Infoset. And these specifications are different standards (the W3C
says "recommandations") which are clearly not so well debugged and not
so well implemented.

To talk specifically about W3C's XML Schemas, they are not consensual,
there are at least two serious competing standards for XML validation
(three, with the old DTDs): W3C's XML Schemas and Relax NG (See RFC
3470 for a discussion). I personally prefer Relax NG. But both are
quite recent, have very few implementations and are far from being
widely mastered.

You cannot have your cake and eat it. Either you stick with XML, the
core, and most of the pro-XML arguments hold or you want to benefit
from the whole XML bestiary and many arguments (well tested, easily
available) vanish.

---

From: "John T Levine" <[email-masked]>
Date: 17 Jun 2004 10:33:10 -0700

> the data in various directions, like a tree.

Margaret Olson gave some good examples of info that one might want to 
put in an XML document describing a mail sender.  I can see why it could 
be useful to publish them, but I think it'd be a disaster to allow MARID 
extensions like that.

One of the main points of MARID, as I understand it, is to develop 
something that can be implemented relatively quickly and will 
interoperate among senders and recipients all over the net.  I don't 
know what I'd do with descriptions of how much mail a domain sent and 
for what purposes (starting with whether I'd believe those assertions, 
which I probably wouldn't unless it was from a sender whose mail I was 
going to accept anyway) but I'm sure that if I asked six people the same 
question, I'd get six different answers.  That's not interoperability, 
that's chaos.

I don't see any problem if the MARID record includes a pointer to a 
place where all of the extended XML goop can be found, probably as a 
URL. It's going to be big enough to need a TCP session anyway, and http 
is the only TCP service I know that is well standardized and is known to 
scale reasonably well via caches and the like.  RSS shows us that it's 
practical to fetch XML info records via http.

The MARID record needs to contain info that's universally understood. 
I understand what "these IPs can send our mail" means.  I don't 
necessarily understand all the implications, but I understand that 
assertion.  Much beyond that, it's still all in the fog, and I don't 
think any of us are well served by a spec that requires mail recipients 
to download fog they can't use and don't understand.  If people want 
private extensions for, say, a domain to tell its own MUAs how to parse 
the received headers, that's fine, but it's not needed to interoperate 
among domains so I don't see that it belongs in MARID.

If we take all the extra semantics off the table, the simple SPF format 
is quite adequate, and we can reasonably defer possible standardization 
of the XML over http, probably starting with conventions for the URL to 
ask about both the domain and the mailbox, until later.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for 
Dummies",
Information Superhighwayman wanna-be, http://www.johnlevine.com, Mayor
"I dropped the toothpaste", said Tom, crestfallenly.