Subject: Re: Adding SpamAssassin Headers to IETF mail

From: Sandy Wills <[email-masked]>
Date: Tue, 16 Dec 2003 19:34:35 -0500

[email-masked] wrote:
> ...we are planning to turn on SpamAssassin on all IETF mail...

 > ...this implementation is to allow the IETF community to get used
 > to having these headers in the messages, and allow us to make any
 > changes to the filtering rules.....

 > This will also allow us to look into scripts that will automatically
 > go into mailman admin queues and remove messages that it believes are
 > spam.  This will reduce the work load....

I would like to propose a solution to the looming religious war:

   Some people are serious about wanting to see every message that 
crosses the ietf domain, and will offer violence to anyone who wants to 
keep their daily dose of spam away from them.  Others (well, me.  I 
follow ietf and ietf-announce for my own entertainment and education.  I 
don't mind if you filter on "bigger dick".) will be willing to accept 
the server-based filter, in order to spare their inbox.
    Would it be possible to somehow flag messages as "valid" or "bogus" 
(or maybe "Holy Writ" or "Message from Satan"), and then allow 
subscribers to set their systems to automatically filter on these 
labels, if they care?
    Of course, if I wanted the mail server to work harder, I would 
suggest that the system keep track of each subscriber's preference and 
either send or pass, thus saving the bandwidth.
    If I REALLY hated the programmers, I would ask that this last idea 
be expanded to the 4-level[1] "We know this is important", "We think 
it's meaningful", "It may be spam but we don't really know" and "We're 
almost positive that you don't need this message", thus letting everyone 
get exactly what they want.

    As if that's ever really happened on Usenet.....

[1]  Could we come up with a 512-bit flag system[2]?  Would that be 
enough?  I don't want dick ads, I do want breast ads, I won't read 
anything from California, I really, really want stuff from Dell.....

[2]  Yes, I spend too much time on ASR and AT-SR.

-- 
: Unable to locate coffee.  Operator halted.
: Reply to [email-masked]

---

From: Gordon Cook <[email-masked]>
Date: Tue, 16 Dec 2003 21:05:17 -0500

>>...we are planning to turn on SpamAssassin on all IETF mail...

good

would it be asking too much to add [ietf]  to the subject line of each message?
-- 
=============================================================
The COOK Report on Internet Protocol,  609 882-2572 (PSTN) 703 738-6031
(Vonage) Subscription info & prices at 
http://cookreport.com/subscriptions.shtml 
Googin on real time global corp. http://cookreport.com/12.11.shtml  Purchase 10
years of back issues at http://www.cafeshops.com/cookreportinter.6936314 
E-mail [email-masked] or use [email-masked] Free World Dial up 17318
=============================================================

---

From: Franck Martin <[email-masked]>
Date: 17 Dec 2003 14:45:17 +1200

> >>...we are planning to turn on SpamAssassin on all IETF mail...

good...

Do you forbid attachments too? look at anonizer it clears e-mails from a
lot of harmfull code...

> would it be asking too much to add [ietf]  to the subject line of each message?

I second that motion... for having requested it eons ago...

It was replied to me a lot of technical reasons on how I could filter
otherwise. But I'm a human and I like to see it is an [ietf] mail.

Cheers

----
Franck Martin
[email-masked]
SOPAC, Fiji
GPG Key fingerprint = 44A4 8AE4 392A 3B92 FDF9  D9C6 BE79 9E60 81D9 1320
"Toute connaissance est une reponse a une question" G.Bachelard

---

From: David Morris <[email-masked]>
Date: Tue, 16 Dec 2003 19:36:16 -0800 (PST)

> > would it be asking too much to add [ietf]  to the subject line of each message?

Me three! Please! Some mail list software allows me to set my own tag but
a recent attempt to find such a feature on the IETF software leaves me
with no easy choice.

Dave Morris

---

From: Pekka Savola <[email-masked]>
Date: Wed, 17 Dec 2003 08:00:38 +0200 (EET)

> a recent attempt to find such a feature on the IETF software leaves me

I don't-  IMHO it's stupid to waste the precious bits in the subject 
line to say "[ietf] " because there is no need for such.  The messages 
can be filtered better using other thods as well, and humans can look 
at the headers..

-- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 17 Dec 2003 09:38:52 +0100

> crosses the ietf domain, and will offer violence to anyone who wants 
> to keep their daily dose of spam away from them.  Others (well, me.  I 
> follow ietf and ietf-announce for my own entertainment and education.  
> I don't mind if you filter on "bigger dick".) will be willing to 
> accept the server-based filter, in order to spare their inbox.

I would very much like it if this could be implemented. I get enough 
spam as it is; I don't care to receive even more through the IETF 
lists. The problem with just tagging the spam is that it still wastes 
bandwidth, and to add insult to injury, the extra anti-spam headers 
waste even more bandwdith. Fortunately, it isn't as bad as I expected, 
for instance your message contains:

X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on 
ietf-mx.ietf.org
X-Spam-Status: No, hits=1.1 required=5.0 tests=PENIS_ENLARGE 
autolearn=no  version=2.60

And before anyone asks: yes, I care because I regularly connect over 
GPRS and my (spam) filtering happens client-side.

I don't believe keeping track of the recipient's spam reception 
preference is a lot of work for the mailing list server: simply set up 
two lists, one that checks for spam, adds the headers and forwards, and 
one that checks for spam, doesn't add headers and forwards on 
non-spaminess.

Altenatively, wouldn't it be possible to create a system where list 
members sign their message, and when the signature is recognized, the 
message is automatically flagged as "not spam"?

---

From: Tim Chown <[email-masked]>
Date: Wed, 17 Dec 2003 08:57:23 +0000

> can be filtered better using other thods as well, and humans can look 

I agree, for filtering everything's in the header already.

Tim

---

From: Gordon Cook <[email-masked]>
Date: Wed, 17 Dec 2003 09:13:13 -0500

>I agree, for filtering everything's in the header already.

I  do not use eudora to forward list mail to separate mail boxes. 
therefore if i don't start list filtering into seperate mail boxes, i 
am forced to guess where a piece of mail might be ietf since adding 6 
characters to ever subject headers is judged unhelpful and 
unacceptable?

There is no sure way at all to tell from a subject line whether its 
IETF and to complain about adding 6 ascii characters to a subject 
line wasting bits if it gives several thousand humans a hint as to 
whether to open and read, or delete unopened, or delete mail to spam 
mail box seems to be strange.  But this point of view i guess is why 
the list keepers have not yet and probably never will do what most of 
the other lists that I receive do.

---

From: Dave Aronson <[email-masked]>
Date: Wed, 17 Dec 2003 09:24:03 -0500

On Tue December 16 2003 21:45, Franck Martin wrote:

 > It was replied to me a lot of technical reasons on how I could
 > filter otherwise. But I'm a human and I like to see it is an [ietf]
 > mail.

I see that it is an IETF mail by seeing that it got routed into my IETF 
folder.  (It gets there by my filters checking whether Sender is 
[email-masked].)

On the other claw, sometimes I have to get my mail over the web, in 
which case the folder routing rules haven't been applied.  For this 
reason, I'd support a *short* tag, such as "[ietf]".  To those who 
bitch about the waste of bits: clamp down on topic drift, large 
signatures, unnecessary headers (including spam-detection hints!), etc. 
etc. etc. first!

-- 
Dave Aronson, Senior Software Engineer, Secure Software Inc.
(Opinions above NOT those of securesw.com unless so stated!)
Email me at: work (D0T) 2004 (@T) dja (D0T) mailme (D0T) org
Web: http://destined.to/program http://listen.to/davearonson

---

From: Mark Smith <[email-masked]>
Date: Thu, 18 Dec 2003 01:10:53 +1030

I just match on either the 

"Sender: [email-masked]"

header, or the ML specific email address I've created.

I'm using Sylpheed though, it seems to be more flexible on matching header fields than most other email clients I've used in the past.

On Wed, 17 Dec 2003 09:13:13 -0500

---

From: [email-masked]
Date: Wed, 17 Dec 2003 10:39:45 -0500

> anything from California, I really, really want stuff from Dell.....

http://www.ietf.org/internet-drafts/draft-eastlake-xxx-06.txt

---

From: Keith Moore <[email-masked]>
Date: Wed, 17 Dec 2003 11:26:40 -0500

> would it be asking too much to add [ietf]  to the subject line of each message?

yes.  it's completely redundant information, and it interferes with readability,
particularly on small displays.

why don't you get a better mail reader that lets you classify mail as it arrives?
that is a much better way to distinguish one list's traffic from another.

---

From: Keith Moore <[email-masked]>
Date: Wed, 17 Dec 2003 11:24:58 -0500

> > ...we are planning to turn on SpamAssassin on all IETF mail...

I have serious concerns about the use of spamassassin to filter IETF
mail, but it depends hugely on the details.   

If the secretariat is just tagging mail, I don't have a big problem with
that.  

If the secretariat is proposing to use spamassassin as a way to
implement a  few very carefully chosen filters, that's fine, provided
those filters are  actually reliable indicators that a message is not
appropriate for the IETF list.  For instance if IESG were to set a
policy that all messages are plain English text (I'm speaking
hypothetically here), then a filter that excluded other content-types
and other languages would be defensible.  If IESG were to set a policy
that says messages should have valid return addresses, then a filter
that checked return addresses for validity would be defensible.  

In all cases the filters should be well-documented and subject to public
review prior to implementation, not something imposed at the whim of the
secretariat and/or IESG. 

If the secretariat is proposing to use SpamAssassin's default filters,
or even very many of the filters included in SpamAssassin, I would have
serious objections to that.  Many of SpamAssassin's criteria are
completely without technical justification, and it is not unusual for
SpamAssassin to block legitimate mail (as it did in that example I cited
a day or two ago).

Even if SpamAssassin only blocks one out of one hundred legitimate
messages, it is still unreasonable to impose a significant barrier on
that one poster.  SpamAssassin is not a good judge of what is correct or
reasonable, and we have had too much arbitrary censorship already.  It's
hard enough to contribute usefully to IETF without imposing the
additional burden that contributors second-guess SpamAssassin's filters.

---

From: David Morris <[email-masked]>
Date: Wed, 17 Dec 2003 11:04:21 -0800 (PST)

The point of [ietf] has little to do with programatic filters and much to
do with human visual filtering. Seeing the list tag in the list of
subjects provided in the index list provided by my mail client makes
human prioritization much easier. Headers are for programs, subject
content is for humans. There is a need though you may not feel the need.

Dave

---

From: [email-masked]
Date: Wed, 17 Dec 2003 15:07:02 -0500

> content is for humans. There is a need though you may not feel the need.

Procmail is your friend

:0
*^Return-path:[email-masked]
{

  :0
  * ^Subject:\/.+
  OLDSUB="$MATCH"

  :0 hwf
  | formail -I "Subject: [ietf] $OLDSUB"
}

It can probably be done more simply - I'm just winging it here.

(Yes, I know procmail isn't available on a certain popular system.  I don't
have much sympathy for the position that the IETF list should do substandard
things in order to support people who insist on using substandard tools.)

Proper chasing of Message-Id: and References: will even allow visualisation of
threading for off-list replies with changed Subject: lines, which is obviously
not anything the IETF mail server can help with.

---

From: Dean Anderson <[email-masked]>
Date: Wed, 17 Dec 2003 16:01:43 -0500 (EST)

This is ridiculous.  The IETF is not getting a lot of spam, so adding
SpamAssassin headers is a solution in need of a problem.

I think this kind of filtering violates the IETF charter on public
participation.  SpamAssassin in particular uses many dubious and
revenge-oriented blacklists to make it determination, and labels much
non-spam mail as spam.  Using SpamAssassin this way just enables these
revenge blacklists to suppress IETF mail, which is inappropriate and a
violation of IETF rules on public participation.  

Furthermore, such revenge blacklisting is a group boycott in violation of
anti-trust law (see Exactis V. MAPS for a good example).  Many businesses
participate in the IETF, and would be harmed by a group boycott.  It would
be unlawful for the IETF to participate in an illegal group boycott which
harms legitimate business.

If individuals want to scan their IETF mail with SpamAssassin, then they
are free to choose to do that themselves.

Dean Anderson
CEO
Av8 Internet, Inc

---

From: Dean Anderson <[email-masked]>
Date: Wed, 17 Dec 2003 16:20:53 -0500 (EST)

>  > changes to the filtering rules.....

The above seems like a thinly veiled attempt to make SpamAssassin headers 
a defacto standard supported by the IETF, without going through the 
standards process.  There is no reason the IETF should condone bypassing 
the standards process to favor one competing implementation.

Obviously, if the goal is to standardize these headers, then a standard
can be produced and put through the standards process.  Certainly,
experimenters can test SpamAssassin headers without putting them in all
the IETF production mail systems.  This is yet another reason why adding
SpamAssassin to IETF email headers is inappropriate.

Dean Anderson
CEO
Av8 Internet, Inc

---

From: grenville armitage <[email-masked]>
Date: Thu, 18 Dec 2003 09:09:28 +1100

i tend to agree with keith. this thread should have started life
with the subject line "I can't figure out how to use filters on my
client-side or web-side email system" and died right there. 

(Both hotmail and yahoo can at least filter on "To: or Cc:"
which'll catch emails sent to [email-masked] - the same rule's
worked for my old netscape 4.x client for years.)

gja

---

From: Franck Martin <[email-masked]>
Date: 18 Dec 2003 10:39:21 +1200

Because we, people on the road, use various mail systems and even web
based mail, where the filters are not applied yet...

Why such a war for just 6 characters, while all mailing lists do it?

Have you been out there?

Let's give it a try and see...

Cheers

----
Franck Martin
[email-masked]
SOPAC, Fiji

---

From: Keith Moore <[email-masked]>
Date: Wed, 17 Dec 2003 17:54:44 -0500

> based mail

so do the rest of us.  ever tried to read mail from a palm pilot?
those [foo] turds get *really* annoying...

> Why such a war for just 6 characters, while all mailing lists do it?

because I've tried it, and found it to be a royal pain in the wazoo.

> Have you been out there?

yup.

> Let's give it a try and see...

feel free to set up your own list mirror that prepends [ietf].

---

From: Bill Strahm <[email-masked]>
Date: Wed, 17 Dec 2003 17:30:48 -0500

Hmmm,

I am wondering if running this e-mail thread is adding a couple years worth of 6byte additions to the subject.

Seems silly to me - I prefer lists to do this - makes many peoples life easier - doesn't make anyones life harder (and frankly if 6 bytes is going to blow your bandwidth budget - you have worse troubles than this proposal)

Please consider this as someone who thinks it is a good idea because some people want it - regardless if they can jump through 10 more hoops and get the same functionality with filters (on what ? - I hate it when people bcc: mailing lists and you loose the from/cc field containing the mailing list you are filtering on) - procmail (oppps what about the people that don't use that, etc.

Bill

---

From: Keith Moore <[email-masked]>
Date: Wed, 17 Dec 2003 18:00:08 -0500

> doesn't make anyones life harder

it hinders readability, esp. on small screens
it hinders sorting of mail by subject
it gets messed up with conversations involving multiple lists
it's a pain to write filters to take the stuff out...

bandwidth is not the issue.

> Please consider this as someone who thinks it is a good idea because
> some people want it 

not everything that some people want is a good idea.  

> regardless if they can jump through 10 more
> hoops and get the same functionality with filters (on what ? 

return-path.  it's required to be there, you know...

> - I hate
> it when people bcc: mailing lists and you loose the from/cc field
> containing the mailing list you are filtering on)

that's because you're using the wrong fields.  nothing in the mail
standards has ever required every recipient to appear in a to or cc field.

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 18 Dec 2003 00:35:55 +0100

> subscribers to set their systems to automatically filter on these 

Spamassassin has the capability of (optionally) adding a header 
 "X-Spam-Level: " followed by a count of star which matches the spam
score calculated by spamassassin. This makes it easy to filter, at an
integer level, - if you want to filter at all. Set your mail program
to match and throw away whatever is at your tolerance level:
maybe X-Spam-Level: ***
   or X-Spam-Level: ******
   or X-Spam-Level: *********
(if your tolerance is really high).
Anything above that level will also match and be thrown away.

	Henrik

---

From: David Morris <[email-masked]>
Date: Wed, 17 Dec 2003 15:53:40 -0800 (PST)

> feel free to set up your own list mirror that prepends [ietf].

Or for all of those so free with advice that the rest of us endeavor to
program our clients, servers, etc. to add the tag, you could just program
yours to remove it. Clearly y'all have more expertise in this space and
interest in gaining expertise with mail software. Should be a snap.

Or just change the IETF list manager to which which makes addition of the
tag optional by subscriber. I've encountered that feature in the past so I
know it is available.

Dave Morris

---

From: [email-masked]
Date: Wed, 17 Dec 2003 22:36:49 -0500

> Why such a war for just 6 characters, while all mailing lists do it?

If "all mailing lists do it" (which in itself is a dubious assertion) is sufficient
justification, why are we bothering with an IETF?  Maybe we should just
disband and let Redmond write the standards, since "all computers do it".

---

From: "Spencer Dawkins" <[email-masked]>
Date: Wed, 17 Dec 2003 22:14:14 -0600

Is invoking Microsoft close enough to invoking Hitler to end this
thread? (Hint: "please!").

Keith is right. If you don't like the way the IETF discussion list
comes into your mailbox, start a reflector that fixes it the way you
like it. IETF+censored does more violence to its contents than we're
talking about now, and I wouldn't dream of subscribing without it...

Spencer

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Wed, 17 Dec 2003 23:28:58 -0800

Dean,

> SpamAssassin headers is a solution in need of a problem.

the reason you don't see a lot of spam on IETF lists is because it's sent 
to the list administrators, and they filter it by hand.

The chief beneficiaries of automatic spam detection and deletion in the 
current IETF setup is the list administrators.

---

From: Pekka Savola <[email-masked]>
Date: Thu, 18 Dec 2003 11:18:47 +0200 (EET)

> current IETF setup is the list administrators.

.. which do not/cannot use SpamAssassin to filter the bounces(?)

---

From: Mark Allman <[email-masked]>
Date: Wed, 17 Dec 2003 23:18:56 -0500

IETFers-
 
A tag in the subject line is clearly overdue.  But, if we're going to do
it, let's do it right.  Please use "[IETF]" not "[ietf]" because it's
more befitting of a proper acronym.  If people would really rather
defile the IETF's good name by calling it the "ietf" then maybe we could
extend the mailing list software to allow each user to define their own
"turd" to place in the subject line of mail they get from the IETF list.

Thank you,
allman

---

From: John Stracke <[email-masked]>
Date: Thu, 18 Dec 2003 09:32:04 -0500

Mark Allman wrote:

>more befitting of a proper acronym.
>
Just what we need, a mailing list that SHOUTS.

(Then again, for this list, maybe it constitutes fair warning...)

-- 
/=======================================================\
|John Stracke      [email-masked]                |
|Principal Engineer|http://www.centive.com              |
|Centive           |My opinions are my own.             |
|=======================================================|
|"Music is not a noun, it's a verb." --John Perry Barlow|
\=======================================================/

---

From: Keith Moore <[email-masked]>
Date: Thu, 18 Dec 2003 09:40:35 -0500

> the reason you don't see a lot of spam on IETF lists is because it's 
> sent to the list administrators, and they filter it by hand.
>
> The chief beneficiaries of automatic spam detection and deletion in 
> the current IETF setup is the list administrators.

I'm one of those list administrators and I can attest that having spam 
flood the review queues of the mailing lists is a huge problem.  It's 
not terribly unusual for the review queue of some lists to get so large 
that  you can't download and resubmit Mailman's review page without 
crashing the web browser (and I've tried several different browsers on 
different platforms).

but despite first-hand experience with the problem I'm still worried 
about using SpamAssassin - I've seen it block too many legitimate 
messages.

---

From: Keith Moore <[email-masked]>
Date: Thu, 18 Dec 2003 09:58:06 -0500

<sarchasm>
Maybe we should also rewrite the From header field so that people with 
dysfunctional MUAs won't have trouble replying to the list?

Maybe we should also rewrite the Reply-to field so that it doesn't 
matter when people get confused about the difference between reply to 
author and reply all?

And let's be sure to rewrite the To field so that everyone who uses 
that field to collate list traffic will get the messages put in the 
right bin.
</sarchasm>

It's taken us ~15 years to get rid of "features" like those mentioned 
above that well-meaning authors of list software put in to work around 
lack of user agent functionality.  We're still not rid of all of it.

I used to call it "header munging disease" - the idea that if a message 
passes through an intermediary, there's a strong temptation for the 
author of that intermediary to consider munging every header field (as 
well as the message itself) just in case it could somehow be useful to 
somebody -- never mind that this removes valuable information from the 
message, and reduces  everyone's ability to use the messages to that of 
the least capable MUA.

Putting [foo] in the subject header is just another example of this 
trend.  Sure, it might be useful to people with dysfunctional MUAs, and 
there are a lot of those people out there. There were once a lot of 
people whose MUAs couldn't do "reply all", too.

The short-term solution is to make [foo] an optional feature specified 
on a per-recipient basis.
The long-term solution is to fix the MUAs to recognize and do 
appropriate things with List-* fields.

---

From: Harald Tveit Alvestrand <[email-masked]>
Date: Thu, 18 Dec 2003 08:52:28 -0800

Keith,

the reason the secretariat is doing this in stages is exactly because we 
want to see how big the false-positive issue is.

I currently personally use Mailman 2.60 with Bayesian filtering and 
close-to-default rules; it seems to run at a very low rate of false 
positives.

--On 18. desember 2003 09:40 -0500 Keith Moore <[email-masked]> wrote:

> flood the review queues of the mailing lists is a huge problem.  It's not
> terribly unusual for the review queue of some lists to get so large that
> you can't download and resubmit Mailman's review page without crashing
> the web browser (and I've tried several different browsers on different
> platforms).
>

---

From: John C Klensin <[email-masked]>
Date: Thu, 18 Dec 2003 11:50:41 -0500

Keith and others,

While...

	(1) I agree that this (and any SpamAssassin or other
	header-insertion or filtering) would, ideally, better be
	done as a per-subscriber optional feature, and
	
	(2) I recognize that, if for some reason (unfathomable
	to me, but there is no accounting for taste), people
	encapsulate messages in message/rfc822 body parts and
	then sign them (or archive hashes of messages including
	the headers), any modification of the encapsulated
	message would wreak havoc, and
	
	(3) I've got an MUA (and an MTA) that are capable of
	filtering on Return-path and/or List-* and/or receipient
	(including subaddress)fields,

there are three things about this discussion that bother me...

(i) A number of efforts within the community have pointed to the 
advantages of having more routine work done in a routine and 
automated way by the secretariat.   Since the secretariat is 
operating with very tight resources (something else that has 
been in enough documents and presentations that I assume/hope 
everyone knows), it is in _our_ advantage to let them automate 
anything they can sensibly automate without causing _severe_ 
problems.  Conversely, asking for things that might take large 
amounts of time and energy (such as per-user setting of tag 
fields or application of spam filtering), is, IMO, pretty lousy 
prioritization.

(ii) Even with powerful filtering and organizing tools, some of 
us prefer (as a matter of taste) to not have, e.g., one folder 
or color per mailing list or other correspondent.  For us, a 
subject line indicator of source makes it easier to organize 
things cognitively.  Is it a big deal one way or the other?  Not 
for me at least; I can't speak for others.  But it is helpful to 
some of us, regardless of what the MTA or MUA may or be able to 
do.  And that makes me (at least) a little intolerant of people 
starting religious wars that, themselves, consume large amounts 
of (human as well as network) bandwidth, if only because...

(iii) I am, personally, getting concerned that the IETF is 
approaching the point where we are more concerned about process 
and administration than we are about doing high-quality design 
and engineering and getting high-quality results out.  I don't 
think we are there yet, and I think the trends in that direction 
are still reversible, but I take

	* the relative amount of energy the community seems
	willing to spend discussing two, essentially trivial,
	changes to mailing list management, or
	
	* the fine details (rather than broad issues) of a
	process WG charter, or
	
	* heated arguments about proposals for which most of the
	people actively participating in the discussions have
	clearly not read the relevant documents, or

	* IESG being willing to tie up Proposed Standards (or
	even lower-maturity documents) in order to make sure
	that all of the grammatical and procedural niceties are
	adhered to, or

	probably several other things that belong on that list...

as symptoms of serious and deep problems with our priorities and 
how we do business.

For the record, before I'm quoted out of context (as I probably 
will be anyway), our copying procedures from SDOs that have 
become much more procedure-bound, so much so that they often 
appear to no longer care about quality or adoption or 
interoperability of standards as long as the many procedural 
rules are followed to the letter and they can report getting 
more standards out one year than in the previous one would not, 
IMO, be a good idea ... indeed, it would be closer to the height 
of stupidity.

To make a distinction that may be useful before you (or someone 
else) replies, if you (or someone else) wants to get on a tear 
about NATs, I may or may not agree with you, and I may or may 
not believe that the flaming the topic tends to generate will 
result in any real progress or changes in behavior, but at least 
I'm sure the issue is important to the future of the Internet. 
Can you say the same for whether the Secretariat and its mailing 
list machinery adds (or does not add) a few headers to a message 
or a few characters to a subject line ... assuming they don't 
_break_ conforming software used in a rational way (e.g., with 
the robustness principle in mind)?   And, if the answer is "no", 
is there any hope of increasing the ratio of meaningful 
technical standards work to this sort of debate around here?

regards,
    john

--On Thursday, 18 December, 2003 09:58 -0500 Keith Moore 
<[email-masked]> wrote:

> <sarchasm>
> Maybe we should also rewrite the From header field so that
> people with dysfunctional MUAs won't have trouble replying to
> the list?
>
> Maybe we should also rewrite the Reply-to field so that it
> doesn't matter when people get confused about the difference
> between reply to author and reply all?
>
> And let's be sure to rewrite the To field so that everyone who
> uses that field to collate list traffic will get the messages
> put in the right bin.
> </sarchasm>
>
> It's taken us ~15 years to get rid of "features" like those
> mentioned above that well-meaning authors of list software put
> in to work around lack of user agent functionality.  We're
> still not rid of all of it.
>
> I used to call it "header munging disease" - the idea that if
> a message passes through an intermediary, there's a strong
> temptation for the author of that intermediary to consider
> munging every header field (as well as the message itself)
> just in case it could somehow be useful to somebody -- never
> mind that this removes valuable information from the message,
> and reduces  everyone's ability to use the messages to that of
> the least capable MUA.
>
> Putting [foo] in the subject header is just another example of
> this trend.  Sure, it might be useful to people with
> dysfunctional MUAs, and there are a lot of those people out
> there. There were once a lot of people whose MUAs couldn't do
> "reply all", too.
>
> The short-term solution is to make [foo] an optional feature
> specified on a per-recipient basis.

---

From: John Leslie <[email-masked]>
Date: Thu, 18 Dec 2003 12:15:40 -0500

> sent to the list administrators, and they filter it by hand.

   Clearly, this cannot continue (unless we come up with some way to
pay people to perform this service).

> current IETF setup is the list administrators.

   I am really in no position to criticize the use of SpamAssassin.
I started using it for my personal account just before I left for
IETF-58, and have little hope of turning it off. (It flags as spam
roughly 4,000 emails per week.)

   But I think we should stop short of endorsing it.

   It is, frankly, wrong to propagate to the list any email which we
consider to be likely spam. We should instead come up with a way to
verify/authenticate/intuit/whatever that it is an individually-written
message considered to be on-topic by some person we have no reason to
distrust.

   SpamAssassin is a technical marvel -- and I suspect it could be
useful as a sorting tool to distinguish messages which deserve to be
distributed immediately vs. messages which need further verification.

   But that further verification should be done _before_ anything is
distributed to the list. If the SpamAssasin filtering were applied
_during_ the SMTP session to ietf.org and a descriptive error (with
URL) was returned (rather than "250 - OK"), then we would have done
everything we reasonably could to notify an honest sender that we
needed further verification.

   (And, of course, any other content-processing tool could be used
instead of SpamAssassin -- indeed I'm not sure any useful purpose
is served by publishing which particular content-assessment tool we
use.)

   If we can't process during the SMTP session, then -- as a short-
term stopgap -- it is reasonable to flag messages for some automated
processing before distributing to the list.

   (None of this is to criticize anyone who runs SpamAssassin at
their own site to apply more rigorous rules -- I'm probably doing so
myself, even if unintentionally.)

   What I do wish to call into question is the wisdom of passing the
SpamAssasin headers to the list. I believe it creates the potential
for confusion as to what is or is not a legitimate message.

--
John Leslie <[email-masked]>

---

From: Dean Anderson <[email-masked]>
Date: Thu, 18 Dec 2003 12:15:47 -0500 (EST)

On Thu, 18 Dec 2003, Keith Moore wrote:

> flood the review queues of the mailing lists is a huge problem.  

Ahh. Mail from non-subscribers that has to be reviewed.  SpamBayes or
other content filters would be a far better approach to this problem, And
they don't have the feature of revenge.

Also, sorting out pre-existing subject lines, pre-existing message-id's,
that have been seen already on the list is probably useful.

> It's not terribly unusual for the review queue of some lists to get so
> large that you can't download and resubmit Mailman's review page without

A better, faster user interface could be useful.  I think you can have
mailmail messages sent to an imap store for approval, where you can sort
them into different folders based on certain criteria (like replies), and
use faster user interfaces to forward them to the list.

It is strange that it crashes your web browser. I've used web browsers
(Netscape and IE) with a Call Accounting system, which shows Call Detail
Record pages with 20,000 records per page, and IE and Netscape can load
it.

> about using SpamAssassin - I've seen it block too many legitimate 

Mostly, this is due to the revenge oriented blacklists that it uses.  

		--Dean

---

From: Leif Johansson <[email-masked]>
Date: Thu, 18 Dec 2003 18:33:40 +0100

Harald Tveit Alvestrand wrote:
| Keith,
|
| the reason the secretariat is doing this in stages is exactly because we
| want to see how big the false-positive issue is.
|
| I currently personally use Mailman 2.60 with Bayesian filtering and
| close-to-default rules; it seems to run at a very low rate of false
| positives.
|

In my experience lots of false positives from spamassassin+bayesian
is the result of the user making false assumptions about the linearity
of the spamassassin point-scale.

	MVH leifj

---

From: Keith Moore <[email-masked]>
Date: Thu, 18 Dec 2003 13:07:24 -0500

> "From" lines and "Reply-to" and whatever are headers that are meant to
> be processed by computers.  So, you can say all you want about how
> dumb MUAs do or do not process these (and how intermediate mail
> servers should keep their mits off).  Now, humans use these lines,
> too.  So, call them dual use.
> 
> The subject line, on the other hand, is just for people.  

Book titles are for people, too.  Does that mean that it's okay for a 
bookseller or library to change the titles on books, in order to help
the consumer indentify where they came from?

I'm a bit surprised at the frequency at which people who claim to be
networking protocol engineers fail to appreciate the benefits of clean
separation-of-function and layering.

---

From: "escom" <[email-masked]>
Date: Thu, 18 Dec 2003 19:10:15 +0100

I work on an approach to block spam with a database of hash (md5) string of
spam email:
1) Reporting a "verified" spam to the database server on the web
2) the mail client check incoming mail, generate a hash string send to and
verify the presence on the server, is yes block email.
3) download a hot list to block directly on the machine

i don't know if it's a good or bad idea.

--giuseppe

---

From: Joe Abley <[email-masked]>
Date: Thu, 18 Dec 2003 13:33:24 -0500

> i don't know if it's a good or bad idea.

http://www.rhyolite.com/anti-spam/dcc/

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 18 Dec 2003 12:02:51 -0700 (MST)

> From: "escom" <[email-masked]>

> i don't know if it's a good or bad idea.

The several existing implementations of something like that idea suggest
that some people think it is a reasonable idea.  I think it is useful
but has limitations.
See http://www.google.com/search?q=vipul+razor and http://cloudmark.com
for one (set of?) implementation(s).
See http://www.dcc-servers.net/ for another.  The DCC is often used
with SpanAssassin.  Refusing mail that has been seen anywhere else
(i.e. with non-zero DCC target counts) seems like a perfect fit for a
mailing list, but I'm probably biased and so won't suggest that.

I vote "no" if someone is taking a vote about trashing the Subject
headers.  Access to this mailing list should be more, not less difficult.
Anyone who cannot figure out how to sort mail from this list based on
its existing headers and rewrite Subject headers or anything else to
taste is not really interested in the nominal purpose of the list.

If it were practical, it would be good to require subscribers or at
least contributors to prove their interest by showing they can fetch,
compile, install, configure, and operate a simple TCP application such
as an SMTP server.  Anyone who lacks sufficient interest to do (or
already have done) something like that is unlikely to have anything
interesting to say to this list, except to other go-ers.  Such a test
might reduce the number of people who are interested only in non-technical
issues such as the administrative work of the IETF, the nasty evil
power grabbing U.N., ICANN, or legacy internet engineers widening the
digital divide, or whatever else concerns the people who are prompting
the continued statements of the painfully obvious.  (For some reason
perhaps related to procmail, I'm not seeing the questions that prompt
the obvious answers.  It would be swell if those offering the answers
would desist.)

Anyone who cannot find a usable POP3 or SMTP server with which to
subscribe to this list would certainly be better served and better
serve the rest of us by using the web pages of its archive.
See http://www.ietf.org/mail-archive/ietf/Current/maillist.html

Vernon Schryver    [email-masked]

---

From: Keith Moore <[email-masked]>
Date: Thu, 18 Dec 2003 14:12:52 -0500

John,

Trying to make this response a brief one, and hopefully the last message
I need to write on this topic for a while.

1) While I generally support reducing secretariat workload when
possible, I don't think it follows that it's to our advantage to "let
them automate anything they can sensibly automate without causing severe
problems",  particularly without taking due care in how it is done. 
We've had quite a few problems already with lists being subject to
arbitrary censorship, and many of spamassassin's criteria have no sound
justification.

I should at this point re-iterate that so far nothing harmful has been 
done, and it does look like there's some attempt at "due care".  I hope
that publicizing this issue will encourage more "due care".

2) I have given several reasons for objecting to adding [xxx] to message
headers, ranging from theoretical/academic arguments about
separation-of-function and layering to statements of personal experience
that this very practice causes problems with reading mail on small
displays, with searching, etc.  These are not absolutes but merely
factors that people should consider rather than immediately assuming
that subject munging is a good idea.

3) It's gotten to the point that almost any argument about a technical
subtlety on the IETF list gets labelled a religious war.  I suspect this
is partly because we're straining to articulate the justification for
our positions (so they look somewhat like religious arguments even when 
there's an underlying technical basis for them), but that's inherent
in the fact that these subjects are subtle.  

I remember a time when we valued the exchange that helped to illuminate
these subtleties and give justification for our positions, and when we
did not think that this level of exchange was inappropriate or an
excessive consumption of bandwidth.  I'm not sure what has changed, but
I hope it's not the case that we can no longer try to understand subtle
effects of technical decisions - because I believe our inability to do
that has caused the quality of our output to suffer tremendously.

4) I see the [xxx] labelling as a design issue.  Even if we claim we're 
only designing for ourselves, it's still a concern because to me the
casual attitude toward adding [xxx] reflects a lack of understanding of
fundamental network protocol design principles.   I see the spamassassin
filtering as a process issue, but one that affects our ability to
produce good designs, because I've seen several occasions where
valuable input from outsiders was discarded for arbitrary reasons and
the design suffered for it.

John, I know you well enough to know that 

- You've seen more than a few problems with header munging yourself, 
and with munging of protocols by intermediaries in general;
- You are more aware than most that the Internet is a diverse community
with widely varying needs and capabilities and that it is becoming 
more diverse all the time;
- You know enough about protocol design to appreciate the value of
separation of layers in general, and of separation of function between 
user agent and transport in particular; and
- You know enough about information storage and retrieval systems to
appreciate the value in keeping data models clean.

So I don't think I need to convince you of these things.  If I'm talking
to you specifically, I try to frame my statements with knowledge of your
experience and depth in mind. When I make statements like the above on
the IETF mailing list, I'm doing so for the benefit of people who don't
seem to understand these things (regardless of who is in the To field),
and part of my reason for doing so is to try to remedy that situation in
a small way.

Any good design is necessarily a compromise.  It might be that there are
cases where, _after_ considering the various factors, that adding [xxx]
is a reasonable compromise, particularly for a list that operates only
for a year or two - one can argue that UA capabilities won't change much
while the list is in use.  However such compromises are _not_ justified
by statements of the form "it works for me, therefore it is good for
everyone" -- particularly when the Internet is so diverse and when
there's a tendency for these practices to become entrenched.

It does seem like we often get bogged down in arguments between people
of widely varying depths, or between people of very different kinds of
expertise.  In the first case there is no basis for compromise because
the person who is out of his depth doesn't understand the need for
compromise or the basis that makes the compromise reasonable.  In the
second case compromise is difficult because there is little or no common
ground.  I'm not sure how to resolve either kind of impasse in a
reasonable fashion other than by discussion, though this does sometimes
get tedious. Yes, I'd like to find a better way.

At any rate, it seems difficult to get a compromise before it is clear
that people understand the issues associated with a decision.  Only when
people respond to an argument in a way that shows they understand and
respect the issues, or at least, that they're willing to try to do so,
does compromise become feasible.  If there's a way to get closure on 
such issues more quickly I suspect it involves recognizing this and
reflecting that recognition in our culture and processes.

Keith

---

From: John Kristoff <[email-masked]>
Date: Thu, 18 Dec 2003 13:14:36 -0600

On Thu, 18 Dec 2003 13:07:24 -0500

> separation-of-function and layering.

Hopefully the drawbacks are appreciated also.  Quoting Rich Seifert,
"Layering makes a good servant, but a poor master.  Use layering to
organize the way you THINK about networks, but don't let it restrict how
you DESIGN networks."  If I recall correctly, David Clark used to say
something very similiar to this in a protocol workshop class at Interop
awhile back.

John

---

From: "Jake Nelson" <[email-masked]>
Date: Thu, 18 Dec 2003 13:32:21 -0600

> Mostly, this is due to the revenge oriented blacklists that it uses.

You are aware that's it's trivial to disable all the blacklist testing in
the config, aren't you? SpamAssassin is extremely configurable.

-- Jake Nelson

---

From: John Stracke <[email-masked]>
Date: Thu, 18 Dec 2003 14:35:46 -0500

>3) download a hot list to block directly on the machine
It's been done, and the spammers have already evolved to get around it: 
they randomize the messages so that the hashes don't match.

-- 
/=====================================================\
|John Stracke      [email-masked]              |
|Principal Engineer|http://www.centive.com            |
|Centive           |My opinions are my own.           |
|=====================================================|
|"No, no, that's *not* a boat, that's Queen Victoria."|
\=====================================================/

---

From: Keith Moore <[email-masked]>
Date: Thu, 18 Dec 2003 14:48:42 -0500

> > networking protocol engineers fail to appreciate the benefits of
> > clean separation-of-function and layering.
> 
> organize the way you THINK about networks, but don't let it restrict
> how you DESIGN networks."  If I recall correctly, David Clark used to
> say something very similiar to this in a protocol workshop class at
> Interop awhile back.

there's clearly a limit to how much layering is desirable, and it's
often desirable to have a way to bypass layering in corner cases.
I'm convinced that IPv4 worked better because it _didn't_ separate 
location and identity, than it would have otherwise, because the cost
of the extra mapping layer would have been prohibitive for most of
IPv4's history (and may still be prohibitive, but we're getting
closer).

but failure to have clean interfaces and separation of function 

- makes the whole system more complex and less reliable (because 
components can't rely on other components functioning as advertised -
e.g. NATs try to second-guess apps and apps try to second-guess NATs)
and 

- makes the system less adaptable to meet unanticipated needs (because
assumptions about how things work are no longer isolated in certain
parts of the system but they permeate the entire system - meaning that
the entire system has to evolve rather than evolving it one piece at a
time)

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 18 Dec 2003 13:01:18 -0700 (MST)

> From: John Stracke <[email-masked]>

> ...

Unless you are mean naive and simplistic hashes, that is an overstatement.
As long as you want to accept mail from strangers, no spam filter can
perfectly predict whether copies of the next message from a stranger
are being sent to 30,000,000 of your intimate friends, but the various
hashing filters do some good work.

An estimate of the effectiveness of a large scale filter can be obtained
from what it sees as the spam ratio.  If it claims that 60% of all
mail is spam but the real ratio is 70%, then it must be 85% effective.

 ....

Concerning false positives for this mailing list--it would be wise to
define what mail is legitimate.  In many places, you must accept at
least 99.9% of all even remotely legitimate mail.  However, this context
is different.  Here a boolean "good/spam" is simplistic and wrong.
Instead we have a spectrum:
  1. on-topic messages from subscribers
  2. on-topic messages from non-subscribers
  3. noise from subscribers
  4. noise from non-subscribers
  5. pure spam such as advertisements for loan sharks

In this list, only #1 is clearly "good." It is good to avoid rejecting
#2, but there is surely no harm in sometimes delaying #2.  If the
senders of any rejected or "false positive" #2 received an informative
non-delivery report so that they could retransmit, what would be the harm?

SpamAssassin is reported to be better than 60% accurate.  #2 is surely
rare compared to #1.  Thus, as long as SpamAssassin white-lists all
subscribers, there would be no harm in the occasional rejection of #2.

---

From: Mark Allman <[email-masked]>
Date: Thu, 18 Dec 2003 13:19:29 -0500

> the consumer indentify where they came from?

Um, my library slaps a helpful identification tag on the spine of every
book to help me find it.  Your analogy, man ...

allman

---

From: Mark Allman <[email-masked]>
Date: Thu, 18 Dec 2003 12:00:37 -0500

Keith-

> trend.  Sure, it might be useful to people with dysfunctional MUAs,
> and there are a lot of those people out there. There were once a lot
> of people whose MUAs couldn't do "reply all", too.

This is just wrong.

"From" lines and "Reply-to" and whatever are headers that are meant to
be processed by computers.  So, you can say all you want about how dumb
MUAs do or do not process these (and how intermediate mail servers
should keep their mits off).  Now, humans use these lines, too.  So,
call them dual use.

The subject line, on the other hand, is just for people.  Sure we can
make programs and filters grok them to classify mail if there is some
standard format (e.g., i-d actions).  But, fundementally subject lines
are for humans, not computers.  So, comparing subject line munging to
reply-to munging seems to me to pretty much apples and oranges.

You might read the above as supporting your point that we should not add
"[ietf]" to subject lines because subject lines are not for computers
(or "dysfunctional MUAs") to process.  However, I think the correct
interpretation is that it is OK for the mail server to add these tags
**and** they may aid the entities that the subject line is actually for
in the first place (humans).  Hence, they are fine.

allman

(I cannot actually believe I am sending a non-snide comment in this
thread.  Someone should slap me.  I read through the whole thread last
night.  Every message was dumberer than the previous one (probably
including this one!).  I was literally laughing out loud.  I cannot
believe we are even having such a dumbass debate.  But, it was like a
wreck on the highway and I could not stop rubber-necking.  If we have
this much trouble about 6 characters in the subject line then we might
as well forget that problem statement thingy.  Really.)

---

From: Keith Moore <[email-masked]>
Date: Thu, 18 Dec 2003 15:39:58 -0500

The problem with this analysis is that it assigns greater value to 
contributions from subscribers than to contributions from 
non-subscribers.  But often the failure to accept clues from 
"outsiders" causes working groups to do harm - and filtering messages 
in the #2 category increases this tendency.  The occasional rejection 
of #2 messages can be very harmful.

---

From: John Leslie <[email-masked]>
Date: Thu, 18 Dec 2003 17:49:58 -0500

> 5. pure spam such as advertisements for loan sharks

   Agreed that these categories exist. Alas, we cannot necessarily tell
them apart. :^(

> In this list, only #1 is clearly "good."

   I'd greatly prefer to avoid flame-wars about how much difference
there is between #1 and #2...

   Personally, I consider the question pointless because we don't have
any dependable way to tell them apart. Please realize how trivially
easy it is to harvest poster addresses from archives and forge those
as From addresses.

> It is good to avoid rejecting #2, but there is surely no harm in
> sometimes delaying #2.

   I do not agree that there is "surely no harm". (But I'd _really_
rather not argue that question.)

> If the senders of any rejected or "false positive" #2 received an
> informative non-delivery report so that they could retransmit, what
> would be the harm?

   I _won't_ discuss the possible harm...

   But Vernon's point that a prompt non-delivery report minimizes the
possible harm is an excellent one.

> subscribers, there would be no harm in the occasional rejection of #2.

   This is where I must disagree. Whitelisting something as easily
forged as the From address is simply wrong -- and if it is published
rule, we're sure to see spammers forging whitelisted From addresses
as their standard operating practice.

   If, OTOH, Vernon would like to whitelist the combination of From
address and IP address of the sending SMTP server, that could be a
very worthwhile practice, virtually immune to spammer forging.

---

From: [email-masked]
Date: Thu, 18 Dec 2003 15:19:36 -0800

On Thu, Dec 18, 2003 at 03:39:58PM -0500, Keith Moore wrote:
> "outsiders" causes working groups to do harm

I don't believe this is true, for any normal definition of "often".  
"Occasionally" might be believable.

>  - and filtering messages 
> in the #2 category increases this tendency.

One could just as easily argue that such filtering would decrease the
tendency, because people would modify their behavior to subscribe to
groups they cared about.  Also, one could just as easily argue that
working groups are just as likely to be harmed by distracting comments
from outsiders... 

> The occasional rejection 

Seems more likely to me that the amount of harm would be lost in the
normal noise of ietf processes.

Regards
Kent

-- 
Kent Crispin                               "Be good, and you will be
[email-masked],[email-masked]            lonesome."
p: +1 310 823 9358  f: +1 310 823 8649               -- Mark Twain
SIP: [email-masked]

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 18 Dec 2003 16:36:43 -0700 (MST)

> From: John Leslie <[email-masked]>

> as their standard operating practice.

As is true of many theories about what spammers do or will do, practice
differs from (simplistic) theory.  In the real world, whitelisting by
sender works fine and is not abused often enough to matter.  Whether
it works today because it is rarely used is a secondary issue good for
no more than trying to predict the future.

Yes, I know that spammers often forge source addresses.  I get more
than my fair share of demands from lusers that I unsubscribe them from
this or that stream of porn or other offensive spam.  Nevertheless,
such problems are trivial in this context.

That reasoning involves a second error common to IETF talk about spam
and mailing list noise.  It is the academic pretense that all failures
are of equal gravity and completely unacceptable.  In this case, the
failure mode that supposedly makes whitelisting by sender unacceptable
is merely leaking a little spam.

> very worthwhile practice, virtually immune to spammer forging.

If you mean manual whitelisting, that sounds good in theory, but fails
in practice.  I've experience with various sorts of whitelisting,
because the DCC depends on whitelists to distinguish solicited from
unsolicited bulk mail.  Whitelisting by IP address fails in practice
because so much bulk mail comes from so many different and changing
SMTP clients.  For an example at the small end of the spectrum of
 bulk mail sources, I've had to regularly change the whitelisting
for IETF mailings.  Bigger legitimate bulk mailer often have too
many SMTP clients for outsiders to count, not to mention manually
whitelist.  You must find other ways to whitelist them.

However, whitelisting bulk mail by IP address is trivial compared to
whitelisting private mail by IP address.  I use greylisting (see
http://www.dcc-servers.net/dcc/greylist.html ) which can be described
as automated whitelisting by the triple (sender,sender-IP-address,target).
It works well, but only because it is automated and it uses 4yz soft
failures.  Many ISPs start sending a single message from one IP address
and switch to another after a few minutes--lather and repeat for up
to half a dozen different IP addresses for a single message.  It would
be hopeless to try to manually whitelist the IP addresses used by
customers of such ISPs.  The ISPs that do this sort of thing are among
the largest.

---

From: Keith Moore <[email-masked]>
Date: Thu, 18 Dec 2003 20:23:42 -0500

>>  But often the failure to accept clues from

if I look at why working groups do harm, the failure to accept clues 
from outsiders does seem to crop up "often".  Of course, this is my 
assessment (others might read the situation differently) and I can only 
make this statement about the groups I've actually looked at, which is 
a small and nonrandom sample.

> groups they cared about.

You're incorrectly assuming that people with clues have the time to 
subscribe to and follow every single group that might do something 
harmful.

>  Also, one could just as easily argue that

You could argue that.  I haven't found it to be the case.

> normal noise of ietf processes.

Some noise is more harmful than others.  Some WGs have more potential 
to do harm than others, and those are the very WGs that need outside 
input.

---

From: Keith Moore <[email-masked]>
Date: Thu, 18 Dec 2003 22:19:49 -0500

> It just strikes me as highly unlikely that a WG would ever change 
> course
> because of what would look like random comments from outsiders -- it's
> not consistent with the dynamics of a WG, or with human nature.

and that just might be one of our biggest problems, in a nutshell.

---

From: John Leslie <[email-masked]>
Date: Thu, 18 Dec 2003 23:18:00 -0500

Vernon Schryver <[email-masked]> wrote:
> differs from (simplistic) theory. 

   You're welcome to build your organization on the assumption that
spammers will continue doing the same thing they do today -- I choose
to design for what they're likely to do next month...

> In the real world, whitelisting by sender works fine and is not abused
> often enough to matter. 

   Seems to be true today...

> Whether it works today because it is rarely used is a secondary issue
> good for no more than trying to predict the future.

   I draw a distinction between "predicting" the future and "planning for"
the future. "Planning for" the future requires being ready for things
which may not be the "most likely" outcome.

   Thus, I'm not betting on what spammers will do next month. I'm hoping 
to be prepared for a number of different scenarios.

> this or that stream of porn or other offensive spam.

   Good to know you're aware of this.

> Nevertheless, such problems are trivial in this context.

   Today, maybe...

> are of equal gravity and completely unacceptable. 

   I really don't know who you're talking about here. Certainly I have
said nothing remotely like that.

> In this case, the failure mode that supposedly makes whitelisting by
> sender unacceptable is merely leaking a little spam.

   I work on a simple-minded principle: if you want more of something,
arrange to reward it. I choose not to reward spammers for forging
_obvious_ From addresses.

> If you mean manual whitelisting,

   No, I don't.

> that sounds good in theory, but fails in practice.  I've experience
> with various sorts of whitelisting, because the DCC depends on
> whitelists to distinguish solicited from unsolicited bulk mail. 
> Whitelisting by IP address fails in practice because so much bulk
> mail comes from so many different and changing SMTP clients. 

   I'd actually enjoy discussing this issue.

   However what I was discussing was whitelisting the _combination_
of From address and the IP address that sender normally sends from.
Until senders figure out how to forge the IP addresses of sending
SMTP servers, this should make the whitelist pretty safe.

> For an example at the small end of the spectrum of bulk mail sources,
> I've had to regularly change the whitelisting for IETF mailings. 
> Bigger legitimate bulk mailer often have too many SMTP clients for
> outsiders to count, not to mention manually whitelist. 

   I seriously doubt we need to worry about IETF contributors who
send through more than a few dozen IP addresses. Even if we do, it's
still easily automated. I see no scaling problems even with 1,000
different IP addresses a contributor sends through.

> You must find other ways to whitelist them.

   Perhaps, if you're doing it manually...

> However, whitelisting bulk mail by IP address is trivial compared
> to whitelisting private mail by IP address.  I use greylisting (see

   An interesting concept. I haven't tried it -- though I have used
something a bit close -- imposing 50% packet loss on IP ranges which
seem to contain many open relays. I find that legitimate email gets
through, while spam is significantly slowed. (Obviously I don't
consider this a "solution", just a stop-gap.)

> It works well, but only because it is automated and it uses 4yz
> soft failures. Many ISPs start sending a single message from one
> IP address and switch to another after a few minutes--lather and
> repeat for up to half a dozen different IP addresses for a single
> message.

   Certainly an ill-suited tactic. (I can't help thinking we'd do
well to write up an info RFC about stupid SMTP tricks...)

> It would be hopeless to try to manually whitelist the IP addresses
> used by customers of such ISPs. 

   Not really, the way I'm thinking about it. Any combination not
already evaluated goes to a spam-evaluator (whether automatic or
manual): if evaluated as spam, an error is returned, if evaluated
as valid, the combination is whitelisted. I see no reason for any
manual process, with the possible exception of spam-evaluation;
and that I'd give the benefit of pre-processing by the likes of
SpamAssasin.

> The ISPs that do this sort of thing are among the largest.

   The largest ISPs seem to have the most broken email services.
I find _many_ people are learning to avoid those broken services.
Fortunately, there are easily-available alternatives -- which we
may need to strongly encourage because, alas, the largest ISPs
have the worst record for shutting down spammers.

---

From: [email-masked]
Date: Fri, 19 Dec 2003 01:01:36 -0500

> book to help me find it.  Your analogy, man ...

A quick sampling of 15 books from our local public library shows that:

a) All 15 have spine tags for on the shelves and barcodes for check in/out.

b) The exact location of neither tag is standardized - the height of the spine
tag is variable and attempts to not obstruct the author/title originally
printed.  The barcode is *usually* placed on the back in such a way as to avoid
obstructing text, but on 2 books is on the *front* because less information got
overlaid that way.

Obviously, the library is telling us to try to not munge existing information by
sticking stuff in the Subject: line. :)

---

From: jfcm <[email-masked]>
Date: Fri, 19 Dec 2003 13:16:47 +0100

On 04:19 19/12/03, Keith Moore said:
>Content-Transfer-Encoding: 7bit
>
>>It just strikes me as highly unlikely that a WG would ever change course

True. From experience, the only real way I discovered to make a WG to
change its mind is to give it a rewarding R&D spirit and to work together
on experimentation, with a reverse pyramidal spirit. I mean: you say the
WG is to _solve_ a user documented problem with a real solution as a
demo, from a starting point (user documented so the target is not disputed
within the WG). Every outsider is welcome to explore new ways from
that staring point. these ways and blocking points are kept documented
in a sucess/failures/mailestones tree. So people may either try other
avenues or to fix failures. Also to compare global results between solution.
Often moving a failure within the tree makes it a solution.

What I find frustrating with the RFC system is that nothing final, proven,
validated at a given time so you can build on top. Only projects are final.
I would prefer a lore recipes orieted system where people could also say
"this is the way I do it (or updated it): feel free to help and copy". It 
would
have probably the same technical initial results but it would be more
rewarding and less blocking. WG would be dynamic clubs where to
concert about development, experimentations and compatibilities.

jfc

---

From: Gene Gaines <[email-masked]>
Date: Fri, 19 Dec 2003 15:12:35 -0500

At last a meaningful remark, quoting from below (far below)...

> I cannot believe we are even having such a dumbass debate.

With apologies, I do not appreciate is the number of individuals
who have made observations based on their "personal experience"
concerning the SPAM subject.  Trading war stories does not
contribute to meaningful technical work, and in fact works
to the detriment of the IETF.

Gene Gaines
[email-masked]
Sterling, Virginia USA

--

---

From: Dave Crocker <[email-masked]>
Date: Sat, 20 Dec 2003 21:42:28 -0600

John,

JCK> Since the secretariat is

JCK> operating with very tight resources (something else that has

JCK> been in enough documents and presentations that I assume/hope

JCK> everyone knows), it is in _our_ advantage to let them automate

JCK> anything they can sensibly automate without causing _severe_

JCK> problems. Conversely, asking for things that might take large

JCK> amounts of time and energy (such as per-user setting of tag

JCK> fields or application of spam filtering), is, IMO, pretty lousy

JCK> prioritization.

Let's take this a bit further: For any suggestion involving computing

and/or communication functionality, proposals should come with the

resources to do the major work, where the Secretariat only has to

provide some interface information.

JCK> (iii) I am, personally, getting concerned that the IETF is

JCK> approaching the point where we are more concerned about process

JCK> and administration than we are about doing high-quality design

yup.

So, here is a simple suggestion for anyone proposing anything in the

IETF:

Explain what real and significant problem it responds to and what

it will take to develop and operate it. Who must do the work,

what are their incentives for doing it and why should we believe

they will be successful at doing it anytime soon?

Interestingly, this applies both to protocol design suggestions

and to IETF process revision.

We need to start focusing on small sets of essential, near-term

problems, with core, near-term solutions. As a group, we have zero

success with any other approach.

d/

--

Dave Crocker <dcrocker-at-brandenburg-dot-com>

Brandenburg InternetWorking <www.brandenburg.com>

Sunnyvale, CA USA <tel:+1.408.246.8253>