Subject: Re: [Asrg] 2a. Analysis - Spam filled with words

From: "Terry Sullivan" <[email-masked]>
Date: Fri, 12 Sep 2003 07:58:00 -0500

On Thu, 11 Sep 2003 23:15:58 -0600, John Fenley wrote:

>From: "Terry Sullivan" <[email-masked]>
>>1) There were four distinct "types" of spam.     Variation within 
each 
>>spam-type was much    smaller than the variation between    
spam-types.
>>
>>2) Only one of the four spam-types was even    remotely close to 
"ham."
>>
>
>This reminds me of something I heard about a few years ago while 
>attending a lecture on multidimensional math.
[snip]
>When plotted in N dimensions vertebrae from different 
>species of dinosaurs formed distinct clouds that could be 
>distinguished easily.
>
>Perhaps a multidimensional Bayesian classifier could find these 
>spam/ham groups on it's own. Each method for bypassing filters in a 
>strange way might be easily discernable as a different cloud.

You read my mind, John.  The "distinct clouds" effect was more or 
less exactly what I was hoping to find.  (Unfortunately, the 
distinctions I found were not even remotely "cloud-like"--each one 
was more like "amorphous blob, well-anchored by a couple of 
outrageous outliers."  Which was still an interesting analytical 
result, but not at all what I had hoped to find.)  

The bad news: while "free" multidimensional methods are great for 
analysis, they are utterly impractical for classification, because 
they do not scale well at all.  (CPU requirements for these methods 
grow *at least* quadratically with the number of items being 
analyzed, and higher-dimensional solutions are even more demanding.)

Ultimately, your *core* point--that multidimensional characterization 
methods ought to work better than unidimensional methods--is exactly 
right.  (Exactly the opposite is true when one is trying to make 
fine-grained distinctions among already highly similar documents.  In 
that case, unidimensional methods are generally superior.)  
Ultimately, multidimensional characterization "works better" because, 
as Andrew recently reminded us, spam differs from regular email in 
lots of different ways (i.e., across multiple dimensions), while 
regular email is much more homogeneous.

- Terry

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 12 Sep 2003 15:37:16 -0400

Hello everyone,

I'm going out on the limb here.  So everyone with a heart,  please don't
jump on me, I'm just providing my technical opinion here.   This is a little
long winded, but its done with a purpose, hopefully with goal of reaching
people (especially developers of mail software) who feel the same way I feel
to finally address the spammer problem once and for all.   So brew of pot of
your favorite coffee, a smoke or whatever you do that relaxes you.

I recently learned of this group via a cyber space article who were talking
about what efforts are being done to combat the growing spam problem.  The
article was talking mostly about the basic approach of various method filter
methods, but it concluded with a statement that the solution may also
include a fundamental change to the SMTP system being orchestrated by the
new AETG.

"Finally!  YEEPEE!"  I said!  Finally, we are getting to the heart of the
problem instead of all the kludgy filtering solutions that a) didn't work,
and b) to a large extent played into the hands of spammers.  I wanted to be
part of this process that will fundamentally finally change the outdated
SMTP system!

So I wrote Yakov and others with eagered baited breath telling them I want
to HELP!  We have a pretty big commercial system but I didn't really tout my
own horn.  I wanted to really be part of the process to help define the next
generation software.  So I joined this group with one and only one goal in
mind, to help change the outdated SMTP specification in such a way that
would immediately stop the spamming problem.

Unfortunately,  what I found here was a little disappointing.  I was little
lost with whats going on with this working group.   Instead what I saw here
is a slew of proposals which to me, basically beats a dead horse and will
not solve the problem.

It makes no sense to me to study or analysis the "patterns of spammers" or
what techniques they use, etc, or what "tricks" they have up their sleeves,
or hearing what spammers had to say in this group. All I saw here weree
efforts that attempt to "play games" spammers using filters or other items
that is basically playing into their hands.   Sure, we can learn about them,
but does it solve the problem?   I don't think so.

In the end, working solutions will be based on what the software developers
and vendors  see as clear, sound technical design solutions and more
importantly,  they will only allocate resources that addresses and solve the
main problem head on.   They are not going to implement dozens of these
proposals.  They will see the best and work with that.

To me, we know what the problem is - the SMTP specification!  We change it
and the problem is essentially solved!

Sure, absolutely!  I understand the "spirit" of the internet of keeping a
open standard and the resistance to deviate from the current standards.

However, it is this "spirit" that we need to let go and admit where the
problem of spams really lies and ultimately, the solution will need to be
finally be based on the fundamentals of the solid mail distribution system.

I say this because of past experiences with standard mail systems that
resisted change.   If things don't change, further chaos will evolved and
ultimately, may threathen the very existence of the SMTP protocol, including
any say this IETF working group may have.

So allow me to share with you the story of Fidonet! - its raise and fail and
what we can learn from it.

The Raise and Fall of Fidonet.

Lurking this working group reminds me of the old pre-internet Fidonet days
where Fidonet developers and Fidonet standards administrators were two
conflicting breeds with two different mind sets.

Although we didn't have the major problem of "spamming" like we have today
in the internet mail market,  Fidonet technological progress was hampered
with administrators (usually non-developers)  conflicting with developers
eager for progress and the implementing of new and better methods that both
addressed and solved the issues of its time.

Understandably,  administrators were more concern about standards and
compliance with legacy systems while developers were concern about changing
outdated standards compliance issues to begin offering and supporting new
methods that were beginning to threathen Fidonet existence. This conflict
help kill Fidonet as developers and system operators moved on to newer
things - such as the internet.

I am 100% confidence SMTP as we know it today can suffer some of the same
consequences. In fact, we are already to beginning to see this with the
larger ISPs beginning to ignore RFC SMTP guidelines and basically doing
their own thing.

So hear me out.  If I am off base here,  I will pack up and go away because
I don't want to waste time with kludging an outdated system and play games
with spammers.  I think we can probably learn a few things about what happen
to Fidonet.

What (was) is Fidonet?

Fidonet was an early automated mail/file distributon standard popular during
the 80s and early 90s telecomputing modem and BBS systems days.   Fidonet
allowed remote servers to 'network' themselves to help with the distribution
of mail and files around the world.  It was inherently a modem based system
as the internet was yet available to the masses in the growing telecomputing
market.    Your early BBS remote system were your early "Service Provider"
operation, in fact, many of the larger successful ISPs today started out as
BBS systems and as Fidonet Mail/File Hubs in the network.

As the internet became popular,  Fidonet popularity began to dwindle but it
does still exist today among the purist, especially among for those who had
successfully merged internet methods with Fidonet methods.  Our product is
one those early BBS mail/file systems that have evolved to successful
integrate Internet solutions.

In its pinnacle during the late 80,  Fidonet membership grow to over 100,000
mail servers around the world with the majority US based.   As the internet
grow during the 90s,  I recall seeing an average NET lost of 50 mail servers
per week.   I had calculated that by 2002, there would be less 5000 systems
representing the purist.  Today, Europe holds the majority of the fidonet
network systems.   This was because the cost of the internet access was
still an expensive endeavor in Europe vs the United States.

Fidonet offered a flexible network mail/file distribution topology with the
following addressing format per server in the network called a  Fidonet node
address;

            zone:net/node[.point]

Each member of the network was assigned a fidonet node address or number.

Where

A Zone was basically represented a  continent, nation or country,  i.e,  1
for US,  2 to 6 for parts of Europe and Asia, etc.   See
http://winramturbo.com/fnsp/maps/world.htm  for a global representation of
the zones in Fidonet.

A Net represented areas within a zone such as a state, province , city or
town or country, depending on many nodes were in the area.    A fidonet
address with net number of 1 (net =1)  was usually considered the main hub
or coordinator for the specific net.

A Node was basically 1 computer, 1 system using Fidonet compliant mail/file
server software.

For the most part,  a Node was a "site" that offered online mail/file
services to connecting users.    Today, this could be a web site.

For example, my old zone 1 fidonet address was 1:135/382.   Net 135 covered
the Miami, Florida to Florida Keys area.  I was node 382 in this net.

The optional point field  represented an end-user for the node who wanted to
collect and send mail using Fidonet  technology.  A point was a compliant
fidonet node with no users but the point owner.   This is akin to a user
with a mail client such as Outlook Express interfacing with his ISP only.
He was only allowed to send using his mail hub (ISP).   This is akin to
whats going on more and more with the larger ISP systems restricting "users"
to send mail only using the ISP mail server.  Using their own mail server
for direct SMTP operations is being blocked.

Having a point user was rare in Fidonet.   It usually represented the more
sophiscated user who wanted to have the automated technology but really did
want to be in the business of serving users itself.     It also represented
a group of corporate implementations who had mail/file hosting servers and
provided fidonet point software to their exclusive end-users for automated
Store and Forward system operations.

In my opinion, this "group of point users" in relationship to the  internet
is probably the fastest growing group of potential mail systems for the
internet.

In Fidonet,  an address with Net #1 or Node #1, usually meant it was a
central server or hub in a distribution chain such as a star topology.

                    node      node
                       |               |
        node-----hub------hub---node
                       |               |
                    node      node
                                      |
                                  point

Fidonet Mail Distribution

Fidonet standards defined the management and process of network distribution
for both files and mail.  File distribution is outside the whelm of my
discussion. I will concentrate on the mail, however,  keep in mind that mail
is essentially files so the discussion here about client/server operations
is common to both.

In fidonet,  there were two types of message or mail distributions:

- EchoMail
- NetMail

EchoMail is equivalent to today's newsgroups, and NetMail is equivalent to
today's email.

Echo Mail flowed from nodes to nets, net to net, and nets to zones and zones
to zones, etc.   If a node subscribed to a particular echo forum
(Newsgroup),  Fidonet offered a built-in automatic distribution logic so
that as mail flowed thru the backbone,  echo mail was dropped off for
subscribing nodes in each net.

Netmail was a direct node to node connection concept pretty much like
today's direct smtp client to smtp server mail delivery logic.

Mail going to a destination address is looked up in a directory "phone book"
called the Fidonet Nodelist to begin the connection process and controlled
state machine conversation process,  just like SMTP' MX concept.

Net routing was also part of the fidonet mail distribution attributes. It
was usually user defined or system controlled.   For example:

  To:   Fred       Address:  1:232/45
  From: Tom    Address:   1:178/5
  [X] Send Direct

In this case, the mail was sent directly from 1:178/5 to 1:123/45.  If net
routed, it was sent as follows:

 1:178/5  to  1:178/1     Node #1 for Net 178  was considered the net hub
for that net.
 1:178/1  to 1:232/1      Sent to destination Net hub
 1:232/1 to 1:232/45     final destination

The same logic applied when crossing zones.

Often routed netmail piggyed back on the echo mail backbone distribution so
as echo mail flowed from hub to hub, any routed netmail was dropped off for
net delivery.

So how did this mail distribution and addressing system work?

The basis for Fidonet was its "NodeList" akin to today's DNS system.

The nodelist defined all members in the network and also defined the
capabilities and attributes of each system.

To be a "member"of Fidonet,  your system software installation had to follow
two mininum FTSC (Fidonnet Technical Standards Committe) operational
requirements:

1)  Foremost, Installed software must be compliant with the FTSC1
handshaking protocol, and

2) Each system must be available for mail delivery during ZMH (Zone Mail
Hour) and alway ACCEPT all mail from any system during this world wide
common mail hour period.

and that pretty much meant that it used the nodelist to decide how to send
or distribute mail and files, including the possible times a system was
available.

The above two requirements was probably the most heated debated issues that,
in my opinion,  help with the demise of the Fidonet network.

The basic issue was that the FTSC1 protocols was outdated and kludgy.  It
was based on a modified complex 7 bit XMODEM packet exchange handshaking
protocol.   Developers were putting together new handshaking protocols and
state machine client/server conversations logics that addressed the growing
technical requirements, especially with the internet on the horizon.   So
when new developers and software were coming on board, they often opted to
implement only the new protocols ignoring the FTSC1 protocol.    And even if
they attempted to honor FTSC1,  the complexity was so high,  there were many
bugs and problems, thus violating the FTSC1 minimum requirement.

The second requirement of ZMH was in place so that there was always a
guarantee that a message can be delivered either directly or routed.   This
was basically so that you can contact the person if there were problems.  If
your system was not available during ZMH,  it often meant you were
eventually expelled.   Some of the hard nose administrators and network
coordinators who had a bug up their ass to enforce compliancy would run
automatic ZMH compliancy events to check each node ZMH availability in the
net.

While this requirement wasn't a big issue for developers,  it probably
represented the early form of "unsolicated mail" operations since the ZMH
rule meant that your system must allow any address send you mail during this
special mail hour.   The idea, of course, that you must allow "reports or
system oriented messages" be sent to you.  This was particularily important
for first time operators.   But it was a problem for many simply because
many didn't want just anyone to connect to them, sending unsolicited mail
and files.

These issues, to me, was probably the main reasons fidonet began to suffer
and lose membership.

Adminstrators bent on FTSC1 compliance would "excommunicate" members from
the network for non FTSC1 compliancy and if your software did not comply,
developers were shamed and not allowed to received a FTSC product code which
was used to identify the client/server software used a session.  For many
new commercial vendors, it was hard to sell highly advance new software if
it was "labeled" as non-compliant with FTSC1.

Now  yes,  I can hear you say, "Thee Internet is different.".

I agree.  The internet offered an open society which appealed to many of the
fidonet members.  Those tired with the everyday working groups "Fight-o-Net"
arguments simply moved on to newer and more exciting times.

However, today, we are seeing the same growing "cries" from operators who
are sick and tired with spam. They are looking for solutions in almost the
same way old fidonet people were looking for newer and better solutions.

But allow me to continue with some of the technical aspects of the fidonet
network:

To become a member, you installed FTSC1 compliant software then applied for
a node number with your local area net coordinator (NC) who basically
assigned you the next empty or available node number for that net..
Comformancy testing was performed by the NC to verify compliant FTSC1
operability.

A NC weekly duty was to distribute a net change list file called the
"NodeDiff" file every friday to the zone coordinators (ZC).  By the next
morning or day (saturday), a consolidated NodeDiff was created and
distributed to every member on the network world wide.   Finally, the
Fidonet compliant software would recognized the new incoming nodediff file,
spawn a process to apply the diff file to the previous week nodelist
database.

Using the above netmail message example,  if the destination address
1:232/45 did not exist in the nodelist, then the fidonet mail software would
not attempt to send the message.  Each fidonet software vendor version had
it's own unique way of interacting with the user, but in short, if the
address was not in the nodelist, the  mail was not deliverable.

Also, the nodelist offered many attributes that described a particular
system. In particular, attributes such as:

- No Direct Netmail
- ZMH (Zone Mail Hour) only

Another minimum membership requirement was that you were reachable one way
or another.  If not directly, then via net route.  If ZMH only,  you could
not send any time of the day but only during  the "Zone Mail Hour" or ZMH.
ZHM was usually 5 - 6 am GMT.

Using the above netmail message example, if the address 1:232/45 system was
identified as as no-direct mail system, it y meant you could only net route
the message.  If the destination was ZMH only,  you can only send mail
directly or indirectly during ZHM only.

There were other nodelist attributes that described the capabilities of a
system, such as:

- Maximum Modem Speeds,
- Possible file transfer protocols,
- Internet Ready System (connect address a IP address vs phone number)

etc.

When two fidonet servers connected, a followed a state machine concept that
is basically summarized as followed:

1)  Client starts process by connecting to server sending a fidonet "packet
identifer"

The packet describes who the client is and what level of compliance it
supports.

2) Server verifies the client against the nodelist.  Optionally, a server
can use caller id technology to compare the client IP or phone number in the
nodelist.   If the client address is not in nodelist, or if there a IP/phone
number mismatch the server issues a negative packet and hangs up.

3) Once authorized,  client sends all data files (mail plus files) assigned
to this server.   It is possible that the client data files includes special
request for information.

4) Once the clients all its files, the roles are switched.

5) The server checks for any special request, if any, it is processed to
created the response.

6) The server sends all files it has for this client, including any special
request responses.

7) Once the server is complete, the session is complete and disconnected.

In short, that was how fidonet worked.

So what can we learn from Fidonet that can assist us in addressing the
problem of spammers in today's internet world?

Well, foremost, which is my primary concern, is to first recognize that
developers of SMTP servers are not administrators.  Second, to recognize dev
elopers do have very good handle of the problem and the solution.

As with Fidonet, administrators who were enforcing the compliancy  issues
restricted the progress and implementation of new ideas to solved and/or
address issues the system.

Now, again I fully understand the Internet is a more open system.  RFC,
proposals and drafts are well structured and compliance among systems is
what makes the system work today.   In addition, I fully understand the IETG
is not enforcing "rules" for "membership."

However, the mentality and effort to KEEP with the same standard is still
the same.

If we wish to finally address the spammer problem it will not be solved
using the current open, free for all outdated  SMTP framework that we have
today.

The solution is based in LOCKING out the unauthorized system using enhanced
SMTP authorization solution.

Anything else, and we are just wasting time.

So whether its feasible or possible, I throw these ideas up for grabs:

1)  Only compliant SMTP software should be allowed to distribute mail.

2)  SMTP software vendors should registered to obtain SMTP product codes.

3)  SMTP server DNS records should include attributes that better describe
the system, such as the following concepts:

- Product codes
- Send/Receive System
- Send Only System
- Receive Only System
- Private Only (Local domain only)
- Restricted Relay Allowed
- HTML allowed
- Attachments supported
- Bandwidth, T1, DSL, dialup, etc
- Open Hours

In summary,  what I am proposing is that we concentrate on the
"authorization" aspects of smtp servers.

In my opinion, we really don't have a choice on this matter.  We already see
it  happening with more and more ISPs restricting residential systems.
This is the first shot of the bigger ISP systems organizing themselves in
such a way to create a "backbone" and expand the restriction list to include
commercial systems.  Once one big ISP does something "different" that not
quite kolser with SMTP guidelines, others will follow.

So what about rDNS solutions?

rDNS proposals are probably among the best to enforce authorization concepts
and yet offer minimal software design requirements.  We need to concentrate
on the best proposal, enhance it, agree and put it out there for developers
to begin using.

However, I believe it needs include concept I described above otherwise, we
are just beating a dead horse.

In my view, the problem is simple, it is well understood and there is only
one optimal solution - only authorized systems can be allowed to send mail.

Now, how about valid unsolicited mail  (not spammers)?

The solution is simple.

All systems that want to be able to send unsolicated mail must "register"
with an organized registry system.  This can be done during the installation
process of the software.

I will guarantee that if we had such a system in place,  responsible
software vendors will include the "registration" logic  or risk being locked
out.

How about Open Relay or Open Proxy problems?

Simple, again, part of the registration process may include the logic of
confirming the operability of your newly installed SMTP software.   It can
test for open relay or open proxy operations as part of the qualification
process.

How about the politics, the "spirit of the internet"  conflicts with such a
system in place?

Well, as far as the "spirit" of the internet,  in lieu of the greedy aspects
of ISP using the "spamming problem" as an excuse to exclude the residential
systems,  well, this will solve the problem.   If the small "mom and pop"
system wants to have a small smtp system on his machine,  it too must
register with the registry.  In this case, the ISP has lost its excuse to
enforce static IP or commercial contracts.

As for the politics, well, the only thing I see is legacy systems or systems
with free software or abandoned software or what have you, with admins who
will always have a problem with "change" anyway.

The good news,  if this system is put in place,  it is of my strong opinion,
that there will be open software people who will write "wrapper" solutions
for legacy systems.

So even if people choose to keep existing software that works or they
already paid good dollar for and don't wish to "upgrade," etc,   I'm sure
there will be developers who will write "proxies" or "wrappers" that helps
make their system work.

There you go.  Please take it easy on me.  I'm just a developer anyway where
simplicity of solutions are not always the most realistic approach to
problems.   I prefer to work on a  solution that will mandates compliancy of
smtp operations and lock out unauthorized and unsolicated mail distribution.

Thanks for taking the time to read this.

Hector Santos
WINSERVER "Wildcat! Interactive Net Server"
support: http://www.winserver.com
sales: http://www.santronics.com

---

From: "Alan DeKok" <[email-masked]>
Date: Fri, 12 Sep 2003 15:59:43 -0400

> and the problem is essentially solved!

  The list archives (such as they are) contain a large number of
messages on this topic.  I won't repeat those discussions here.

> address issues the system.

  The IETF does not enforce or test for compliance to protocol
specifications.  Doing so would get into a political quagmire, which
is probably best left alone.

> 1)  Only compliant SMTP software should be allowed to distribute mail.

  Who enforces this?  How do you enforce their enforcement?  Will the
abuse by the enforcers be worse than the abuse by the spammers?

> 2)  SMTP software vendors should registered to obtain SMTP product codes.

  There is no central IETF registration of product codes for any
protocol.  I'm not sure it's a good idea to start one now.

> software vendors will include the "registration" logic  or risk being locked

  You, personally, guarantee it?  Hmmm...

  Alan DeKok.

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 12 Sep 2003 21:25:54 +0100

First, let me point out that as a developer, you should be aware that 
simple lists of authorised nodes do not scale to Internet proportions.  
There's a document explaining the technical considerations all 
proposals should look at.

> Anything else, and we are just wasting time.

How about you look over my recent draft paper on "E-mail 
Authentication", the link to which can be found in this month's 
archive.  You might find that my scheme covers what you want, but in a 
way that is far more scalable than Fidonet.  Tell us what you think of 
it, good and/or bad.

BTW, yes, we are aware that until a final solution can be found, all we 
have is an escalating war of countermeasures.  We'd dearly love to be 
able to throw out SMTP right now and put in something new and shiny.

The sad fact of the matter is that ISPs and users cannot and will not 
migrate all at once, indeed such a migration would probably take a 
decade or more.  Administrators do have some fear of change, but users 
even more so, and there are a lot more of them.  As developers, we have 
a responsibility to craft solutions that make any such migration as 
painless as possible.

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     [email-masked]
website:  http://www.chromatix.uklinux.net/
tagline:  The key to knowledge is not to rely on people to teach you it.

---

From: Yakov Shafranovich <[email-masked]>
Date: Fri, 12 Sep 2003 17:07:10 -0400

Dear Hector,

I understand your frustration and I am sure that it is shared by many 
others here and on the rest of the Internet. However, the developer's 
perspective is not what is needed here, you also need the designer's 
perspective.

The designer looks at the overall Internet infrastructure and the spam 
problem as related to that infrastructure. While the developer jumps to 
writing code and implementing solutions, the designer takes his time 
looking at the overall structure and how everything else might affected 
by the code to be written. We cannot mandate sweeping changes, we cannot 
change the SMTP protocol overnight, we cannot force millions of people 
and companies switch over to a "perfect" system at the tune of billions 
of dollars, euros, rupees, etc. We cannot do anything unless we analyze 
how it will affect every aspect of the Internet and its users. We are 
responsible to all of the Internet users on this planet, we maintain the 
Internet and keep it operating. This requires very careful analysis of 
any changes to the underlying protocols and standards on it.

There is a reason why no specific proposals are being concentrated on - 
we do not have any way to evaluate them. The technical considerations 
document 
(http://www.ietf.org/internet-drafts/draft-crocker-spam-techconsider-02.txt) 
outlines many of the issues that will face us in this analysis. However, 
most developers want to jump straight to code instead of analysing the 
impact of it on others. That is why so many efforts must be concentrated 
on analysing the problem because that is necessary for the analysis of 
proposals.

Therefore, I would like to suggest that our concentration should be on 
getting some documents out the door on the problem of spam, and how to 
analyse and evaluate solutions. The technical considerations document, 
the requirements document, the inventory of problems, the analysis of 
spam, the consent framework, the proposal guidelines, the evaluation 
model, etc. all of these need volunteers. If you want to kill the 
problem, that this analysis must be performed. If you want an overnight 
solution - stop looking - it cannot be found.

If you would like to continue this conversation, please email me off-list.

Sincerely,
Yakov Shafranovich
Co-Chair, ASRG

Hector Santos wrote:
<snip>
> any say this IETF working group may have.

<snip>

---

From: "Eric Dean" <[email-masked]>
Date: Fri, 12 Sep 2003 17:13:44 -0400

What do you suggest?

---

From: Yakov Shafranovich <[email-masked]>
Date: Fri, 12 Sep 2003 17:32:28 -0400

Eric Dean wrote:

> What do you suggest?
> 
> 
>>>To me, we know what the problem is - the SMTP specification!  We
> 
> change it
> 

 From 
(http://www.ietf.org/internet-drafts/draft-crocker-spam-techconsider-02.txt):

<snip>

      The idea of replacing SMTP is appealing because it
      permits thinking in terms of creating an infrastructure
      that has accountability and restrictions built in.
      Unfortunately an installed base the size of the
      Internet is not likely to make such a change anytime
      soon.  It seems far more likely that successful spam
      control mechanisms will be introduced as increments to
      the existing Internet mail service.

      Moreover, the feature of SMTP that is most responsible
      for spam is the ability to receive mail from strangers.
      Without this feature, there would be no flood of spam,
      but many of the most valuable Internet commercial and
      individual activities would also be impossible.
      Replacing SMTP with a protocol that allows strangers to
      send each other mail would not stop spam any more than
      SMTP-AUTH stopped spam, contrary to insistent claims to
      the contrary, before SMTP-AUTH became widely available
      and used.
<snip>

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 12 Sep 2003 20:33:58 -0400

> perspective is not what is needed here, ....

Its not?

> you also need the designer's  perspective.

I really hate to get into the semantics game.  I'm lumping "developer" to
mean a vendor of a software product with both project and product
development experience  What's  a developer to you?   A programmer who takes
specifications from a software engineer?   I don't wish to give you the
impression of cutting myself short,  but FWIW,  I'm a college educated
professionally trained Chemical Engineer who can't get any more "systems
design oriented" then most other disciplines. I have over 25+ plus years of
computer system or software design and integration experience in both R&D
and product development for "systems" as complex as Nuclear Power Plant
control systems to as simple as MAIL, to name just a few of the hundreds of
projects I've been involved with.  I've project managed systems designed
from as low 5M to 50M projects.  So I don't speak as just a "programmer."
I'm a "systems" guy  that is proposing a system that does take into account
aspect from the consumer to the operator as well as the operability of the
network with compatibility being a very important aspect of it as well.

> by the code to be written.

And it shouldn't be any other way.  Yet, a true engineer has the vision of
the recognize problems and the steps needed to be taken.   The better
engineer can propose a solution that will help in the transition.  Our
intranet system is one of the more sophiscated mail/file "client/server
systems" around.   Migration issues from upgrade to upgrade is very
important.

> We cannot mandate sweeping changes, we cannot
> of dollars, euros, rupees, etc.

I am not proposing such a mandate.  It would be a transition as any "Good
Engineer" would recognize as being suggested.

> We are
> outlines many of the issues that will face us in this analysis.

I'm not advocating your elimination nor your reason for living,  nor your
responsibilities, nor the need to do this overall analysis.  However,  I am
advocating the recognization that the problem is really quite simple which
is 100% based on outdated SMTP specifications, in addition for the
recognization that any change regardless of what is proposed will required a
change at the software level.  If that is the case, then borrowing a quality
control terminology,  lets "Get it right, the first time!"     It doesn't
take a "developers" with 1 or 2 years in the market to see this.   If what
is being proposed is going to require a change at the software level, if you
want people to follow it, then it better be RIGHT, the first time.

> However,
> most developers want to jump straight to code instead of analyzing the
> impact of it on others.

This is exactly what I expected to hear from the administrator.   You really
don't give "developers" the benefit do you?   I have thousands of customers
world wide.  Every new change and feature goes to thru extensive design
consideration cycles.  Every change is done with the UTMOST consideration
possible for usefulness as well as compatibility.

> That is why so many efforts must be concentrated
> on analyzing the problem because that is necessary for the analysis of
> proposals.

The analyze of the problem has already been done - SPAMMERS have
unrestricted access to SMTP systems because the specs allow it so.   You can
come up with 1 million kludges.  Ultimately, it boils down to client access
to the system.

> analyze and evaluate solutions. The technical considerations document,

Can you prove that there isn't a solution? I don't need to look any further.
I already now what the solution is, and no,  it is not based on overnight
thought process but many years of operational and practical experience.
Didn't you read anything I posted?  It doesn't sound like it.    Anyway,
this is exactly what I was afraid of.   This is a big was of time.  Good
developers, no, excuse me,  "designers"  hanging around here worth their
salt knows exactly what I am talking about.   Lets stop pussying footing
around and get a new SMTP specification worked out that solves the problem
yet offers a transition concept.

> If you would like to continue this conversation, please email me off-list.

No.  Why?  Keep it open.  What are you afraid of?  Anarchy?   I don't want
to talk to one person, an administrator is going to blew me off anyway.  I
want to talk to the community.

Listen, I'm well experience enough to look at all these proposals, using
your +/- check off list and tell you that the ultimate solution is a
restriction/authorization concept.    Do you design, develop and market a
MAIL system?  I don't know about you, but I do.  I know what the problem is
and I know what the solution is.  I know what my customers are looking for
and yet at the same time, I know how important "standards and compatibility"
means.

Hector Santos

---

From: Jonathan Morton <[email-masked]>
Date: Sat, 13 Sep 2003 01:41:13 +0100

> I know what the problem is and I know what the solution is.  I know 
> what my customers are looking for and yet at the same time, I know how 
> important "standards and compatibility" means.

So, are you going to tell us how your system works?

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 12 Sep 2003 20:55:02 -0400

Yup, I'm putting it to getting now....

---

From: "Hector Santos" <[email-masked]>
Date: Fri, 12 Sep 2003 22:07:17 -0400

----- Original Message ----- 
From: "Eric Dean" <[email-masked]>
To:  "'Hector Santos'" <[email-masked]>
Cc: <[email-masked]>
Sent: Friday, September 12, 2003 5:13 PM
Subject: RE: [Asrg] 2a. Analysis - Developer Perspective

> What do you suggest?

Thanks for asking.   I'm not going to buck the system so I'm putting
together an "official" IETF draft proposal for the system I have in mind.
That should make Yakov et al happy.   I'm going to put my entire weekend
into it.

In the mean time, do you know where I can find a collected list of SMTP
server software vendors across the board, free, shareware, commercial, open
source or otherwise?

thanks

Hector Santos

---

From: Kee Hinckley <[email-masked]>
Date: Sat, 13 Sep 2003 22:23:24 -0400

>restriction/authorization concept.    Do you design, develop and market a

So, you've made the technical decision.  Now the next question of 
course is a market question.  Will it be adopted?  (And this 
discussion has occurred in the archives many times.)

Prior experience indicates that the adoption of similar systems takes 
years.  In the meantime such systems need to run in parallel.  Since 
unconstrained gateways would simply import the spam problem, the 
system needs to remain unconnected to SMTP.  So anyone who is an 
early adopter still has to deal with all the spam they get via SMTP. 
They then find themselves with two systems.  One of which has a cost 
of of entry that is non-trivial in terms of support and registration 
costs (and you need to figure out how to scale that cost across 
national borders, or else cut out large portions of the world), has 
no spam, and which communicates with virtually nobody.  The other 
system has spam, but communicates with everyone they want to talk to. 
Or in other words--early adopters pay more, but get no benefit.

Or, since we are putting this in terms of dead email systems.  What 
you are proposing is very similar to back in the late 80's when I was 
developing mail clients for X.400.  X.400 has a lot of the attributes 
you are looking for.  Strong registration and directory services, 
very centralized.  It wasn't adopted. There was this simpler system 
out there that allowed everyone to talk to everyone. I can see no 
reason why it would be adopted this time around (even if 
significantly simplified).  The barriers to entry for a new system 
are large, and early adopters receive no benefit until far more than 
50% of the population has switched.

What FidoNet didn't see coming, and what also killed Prodigy, is the 
simple fact that people like to talk to people.  The web has brought 
people to the internet.  But the ability to talk to anyone keeps them 
here.  Any move to a centralized and restricted system will be 
ignored.

Also, you neglected the other post-implementation detail. 
Enforcement.  Let's assume that your system is 100% implemented.  Now 
someone from my system sends mail to someone on your system.  Your 
someone claims it's spam.  Mine claims it isn't.  Who presides over 
the dispute?  Who gets penalized?  Who pays for the cost of the 
dispute?  What happens when government borders are involved?  What 
happens when *governments* are involved?  What you are describing is 
incredibly expensive to run.  At very best you might end up with a 
highly specialized and constrained system used by corporations and 
wealthy individuals for mission critical communications.  But the 
rest of the world would use regular email for their communications. 
This group is chartered with looking at the problem with email.  The 
new technology would not be email as we know it, and since email as 
we know it wouldn't go away, our focus needs to remain on the 
end-to-end, available-to-all system that will remain.
-- 
Kee Hinckley
http://www.messagefire.com/  Anti-Spam Service for Companies and Individuals
http://commons.somewhere.com/buzz/  Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.

---

From: Yakov Shafranovich <[email-masked]>
Date: Sat, 13 Sep 2003 23:40:13 -0400

> That should make Yakov et al happy.   I'm going to put my entire weekend

I have no problems with technical submissions as long as personal and
philosophical discussions are minimized. FYI, the ASRG does not require
proposals to be submitted as Internet Drafts in ASCII, so you are
welcome to submit the proposal in PDF, Word, HTML or any other format.
However, if you want to submit it as an Internet Draft to the IETF as
well, take a look at the following links for more information:

http://www.ietf.org/ietf/1id-guidelines.txt
http://www.ietf.org/ID-nits.html

Also, take into account the technical considerations document available at:

http://www.ietf.org/internet-drafts/draft-crocker-spam-techconsider-02.txt

and the requirements document at:

http://www.irtf.org/asrg/draft-irtf-asrg-requirements-xx-01.txt

Please also take a look at existing similar proposals:

IM2000 - http://cr.yp.to/im2000.html
Advanced Mail Transfer Protocol (AMTP) -
http://www.danisch.de/tmp/draft_mtp.txt
Advanced Mail Delivery Protocol (AMDP) -
http://www.amdpmail.com/mod/fileman/files/draft-amdp-00.txt
Authenticated Mail Transfer Protocol (AMTP) -
http://www.ietf.org/internet-drafts/draft-weinman-amtp-00.txt and
http://amtp.bw.org/

> server software vendors across the board, free, shareware, commercial, open

This might help:

http://directory.google.com/Top/Computers/Software/Internet/Servers/Mail