Subject: Re: [testlist] Alleged spam complaint

From: Glen <[email-masked]>
Date: Thu, 25 Sep 2008 08:13:42 -0700

Lots of catching up here this morning:

Fred said:
> Ack. I just wanted to be sure you did.

Fred - Understood, and THANK YOU for watching my back!  

And thank you ALL for watching my back, for that matter!

Olaf said:

> Doug knows his stuff so I assume that his assessment that spams originate
> from 64.170.98.32. is true.

Good.  Establishing the legitimacy is always helpful.

> Glen knows his stuff too, so I assume that his assessment that spam does
> not originate from 64.170.98.32. is true too.

Ha!  I'm glad someone thinks so!  Of course, I'm just as fallible as the 
next person, and have made my share of mistakes.  

> Are there any other explanations for which both Doug and Glen make the
> correct assessments?

Jim points out one possibility (and an example of my mistakes!) below...

Marshall said:

> I am getting spam from IETF servers, and it doesn't appear to be either
> through-list or backscatter traffic. It might be, but it doesn't appear to be.

Yup.  As Jim points out:

> Your message was destined to ops-chairs.  All the "chairs" aliases are
> expanded on the tools machine.  It used to be that the IETF servers just
> forwarded all those messages to the tools server.

And that, Olaf, is the answer to the question.  There actually is a third
kind of outbound email we process - the "chairs" aliases...  which are in
fact just aliases, and nothing more (meaning, they are not lists, and so
they contain no moderation queue or anything like that), and they do all
forward to Henrik's machine for expansion and delivery, I assume using
similar methods.

Jim continues:

> Previously I would have said that the problem is there is no spam filtering
> in that path.  I don't know if that's still the case.  If there isn't, then
> that's the problem.  Alternatively, perhaps the spam filtering needs some
> tweaking.

By way of refresher, here is how spam is handled on the IETF servers:

1. Connection is opened to SMTPD port.  HELO, MAIL FROM, and RCVD TO occurs.
   A number of immediate tests (access control lists, DNS resolution tests,
   address validity tests) are run, and failures cause an immediate reject.
   If these tests are passed, the message is accepted in to the system...
2. Virus scan.  Infected messages are discarded.
3. Spam scan.  All messages get reported.  Messages scoring 5.0 or higher
   get tagged.  ALL EMAIL GETS PASSED ALONG.

That last point is key here.  IETF policy requires that we not discard any
spam that gets this far.  We just tag it.

Here, the path diverges.  For RT tickets we have:

4. Postconfirm scan.  Henrik's excellent challenge/response system checks
   for a confirmed sender, passing through only mail that has been confirmed,
   and holding other mail for a few days, then discarding it.

For mailing lists we have:

4. Postconfirm scan.  Henrik's excellent challenge/response system checks
   for a confirmed sender, passing through only mail that has been confirmed,
   and holding other mail for a few days, then discarding it.
5. Mailman filtering.  The various rules set up for each mailing list become
   the final arbiter.  Generally, tagged spam (5.0 and higher) is sent to
   moderation.  Various permitted and rejected senders are evaluated.  Rules
   are followed.  All of the items in this step are configurable per-list.

For aliases - which I hadn't remembered until Jim pointed them out - we have:

...  nothing further.  Tagged mail is passed through the alias chain out to
Henrik's server or in a few other cases, to the direct destination of the
alias ...

... where, I assume, responsible end-users have their own further spam 
filtering rules and actions in place.

At the moment there are 161 aliases forwarding to tools.  There are also 54
direct aliases of one form or another (Olaf, Ray and Russ have aliases, for
example) in the system.  Because these are just aliases, the messages get
tagged, but not filtered.  In my mind, this is the desired behavior, because
it should be up to the receipient to do the actual filtering... and it's in
line with what the IETF has repeatedly hashed out as its policy, generally.

I assume that Doug must be on one of these aliases, although I haven't
heard back from him yet, and I assume he's getting spam through that alias
just as Marshall did (Marshall - I did get your ticket, I'm going to resolve
it in reference to this discussion).

So as far as I can see, the options are:

1. Educate the people on these aliases about the process above, and 
   encourage them to use their own spam filtering.
2. Somehow modify the 200 aliases to go through postconfirm, or a spam
   evaluation system to discard tagged spam.
3. Convert all of these aliases to mailing lists, and allow those lists
   to distribute and/or filter the email.
4. Eliminate the aliases altogether, and use some other mechanism.

Item 1 seems the most easily done, in light of the current structure.  Items
2 and 3 would be huge undertakings, taking a significant amount of time, as
each alias would have to be handled individually somehow.  And I suspect
item 4 isn't feasible, although it would eliminate the problem.

And, of course, in Marshall's case, items 2 and 3 wouldn't have helped, 
because the message he cited in his ticket got a spam score of 0.3.  Hardly
high enough to trigger ANY automated processes, including his own.

I understand that spam is a pain, and unacceptable.  But speaking from the
point of view of a system administrator, it is IMPOSSIBLE to FULLY ELIMINATE
all spam.  Marshall's spam is one example.  We've had others.  We've even
had spammers actually manually confirm themselves to postconfirm, and then
flood us with spam (fortunately only the RT queues were flooded, but still...)

I assume that anyone on one of these aliases is in a position of some
importance in the IETF.  Regardless of anything else we do, I'd personally
really like to see those people briefed on the current spam filtering 
system in place, how it relates to their aliases, and what to expect (and
I wouldn't mind a request added in that they contact us directly, and NOT
send their complaint to the entire IETF discussion list, but that's just me).

That being said, if anyone has commentary or ideas, or would like some
significant change made, your commentary is all most welcome!

BY THE WAY:

If any of you are not on the testlist ([email-masked]) and want to 
be, please let me know and I will add you.  That list contains all the
people who so kindly helped test during the transition, and represent, to
me, the best of the best at the IETF.  It is to that list primarily that I
direct questions and commentary about issues like this, it is to me, like
having the IETF collective intelligence in my back pocket.  :-)  If you 
would like to join the list, you would be welcome.  Please let me know.

Thanks,
Glen
_______________________________________________
Testlist mailing list
[email-masked]
https://mail.ietf.org/mailman/listinfo/testlist

---

From: Marshall Eubanks <[email-masked]>
Date: Thu, 25 Sep 2008 11:55:10 -0400

Hello all;

In line.

> 4. Eliminate the aliases altogether, and use some other mechanism.

Wouldn't the obvious solution be to have Henrik's servers implement  
postconfirm for this incoming traffic ?
It is his solution, after all.

> flood us with spam (fortunately only the RT queues were flooded, but  

I have had others... 24 in the last week. Here is a sample :

95% discount
Patients can access Our pharmacy via the Internet 24 / 7

C A zN A DzAN     P z rH A RM A mCY

Do European Union Drugs not refuse you in plesure

Forget failures in bedroom for 3 month

porno dvd Full for ops-chairs

If these all scored low, then there is something wrong with your  
scoring system ! (As is is, I wonder if these headers will cause this  
email to bounce.) And, they are indeed all to IETF aliases.

The trouble with the above is that it may be acceptable if there are  
24 bad messages, but it is definitely not if there are 24 million. And  
experience shows that spammers will eventually catch onto and utilize  
any
security hole in industrial quantities.

Regards
Marshall

---

From: Glen <[email-masked]>
Date: Thu, 25 Sep 2008 14:09:28 -0700

Catching up here...

Jim Galvin wrote:
> I would recommend putting the postconfirm system in front of these aliases.
> The one thing to be careful of is that I recall there are automated
> scripts that generate messages to these aliases so we'll need to make sure
> that the appropriate From addresses are all preconfirmed. There may be
> other access control tests that would be helpful, e.g., making sure that
> the From addresses used by automated scripts never arrive from an external
> source.
> Clearly this will be work, as Glen points out.  

I have no problem with this in theory; however, it will represent quite a
bit of labor.  First, Henrik will need to evaluate and/or modify postconfirm
to support this particular configuration (it may already, I don't know.)

Then, we'll have to rework and remap each of the 200 aliases, possibly
creating additional support infrastructure, etc., around them (maybe
turning them into accounts, or mailing lists, I don't know - Henrik will
know) to ensure that they go through postconfirm.

If I made a conservative guess of 4 mintues per alias for configuration 
and testing, you're still looking at a good 12-16 hours of labor here to
engineer that... assuming it's that easy.  And I'm just speaking off the
top of my head.

> But I don't think the IETF
> wants to risk be identified or labeled as a source of spam.  That would be
> bad for everyone concerned.

I think that this is what is totally annoying me about this situation.  I
know what you mean, Jim, and I know you understand this already, and I am
not in any way directing this at you, but your wording illustrates an 
incorrect perception that is getting everyone riled here, and give me,
happily, the opportunity to comment.

The IETF servers ARE NOT A SOURCE OF SPAM.  We are passing email to in-the-
organization individuals through email aliases requested (at least implicitly
BY those individuals)  We are not "passing spam through", because the only
people "receiving" on these aliases are IETF members or leaders of some
type.  Hence it is true that we are "receiving" spam, but I take strong 
exception to the statement may by the original complainant that we are 
"passing through" an "ever increasing stream of spam".  We're not technically
passing through anything, IMO.

If the recipients of these aliases had POP accounts on the IETF server 
instead of forwarding aliases, this conversation would be vastly different.
Then it would be a matter of putting spamassassin on their POP boxes...
which would already have them... and adjusting their rulesets and blacklists,
and calling it done.  The fact that they're choosing aliases instead of POP
boxes should not suddenly implicate our servers as being "a source of spam."
That is, IMO, absolutely ridiculous.  

Okay, I'm done ranting.  But I felt I needed to make that point.

> It is his solution, after all.

I haven't seen Henrik chime in on this, but I know he has been extremely
busy recently.  I am standing by for his word and ideas on this.  Doubtless
he will make some comment that cuts through to clarity and makes the 
solution seem simple, as he always does.  I look forward with hope to that.

> If these all scored low, then there is something wrong with your scoring 
> system ! (As is is, I wonder if these headers will cause this email to 
> bounce.) And, they are indeed all to IETF aliases.

Ha!  Yes.  Something wrong indeed.  We're using SpamAssassin, of course,
with every available ruleset from sa-update and openprotect; however, even
those rulesets are, IMO, extremely weak in their default scoring.  Even
the most aggresive rules only default around 1.8-2.4; but almost all the
"normal" rules add only 0.1 to 0.2 to the score.  Reaching 5.0 is difficult.

OTOH, I'm surprised at the sheer volume of spam we're already trapping.  
Spam scoring in the 20s - 40s - 80s and 90s is not uncommon (and quite
humorous to read!).

I myself run much more aggressive scoring on my personal accounts, and I
get NO spam.  But that's just for me.

The problem we'll quickly run into if we try to tamper with default scoring
is the same one we ran into when we talked about gateway rule checks - we
will erupt into a huge debate about which test is "worthy" of what scoring -
and that's just more than I can take on right now.  So in order to not "rock
the boat", I keep the rulesets updated, but at their default levels - even
though I personally would LOVE to "up the weight" a bit.  :-)

> The trouble with the above is that it may be acceptable if there are 24 bad 
> messages, but it is definitely not if there are 24 million. And experience 
> shows that spammers will eventually catch onto and utilize any

Agreed.

When we took over the management of the IETF, we were told repeatedly that
the IETF was interested, in principle, in "cleanup" of legacy systems that
were no longer relevant or useful.

So what *I'm* wondering is - maybe now is the time to look at these aliases,
and why they exist, and perhaps *gasp* do something more than just put
postconfirm in front of them?  Maybe it's time to look at eliminating some
or all of them?  Changing some?  Moving to web forms instead of publishing
email addresses on the website in plain text for every pathetic little
robot to pick up?  Maybe set up POP accounts for key players, rather than
using aliases, which will give each user more options for spam handling?

I don't know... but it makes me wonder.  Since no solution is perfect, and
since, as Jim points out, there will be work and effort involved in dealing
with this, I REALLY want to do "the right thing" at the highest level, and
find a solution, rather than just one more band-aid.

Glen

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 26 Sep 2008 09:38:30 +1200

> engineer that... assuming it's that easy.  And I'm just speaking off the

It's for Henrik to say, but I think it's a bit more delicate, since these
aliases are regenerated automagically at some frequency. You realise
there is one for every I-D?

For example, draft-otis-dkim-adsp <AT> tools.ietf.org and
draft-otis-dkim-adsp-sec-issues <AT> tools.ietf.org both lead to
Doug Otis, and that's only his 2008 drafts. He has at least 24
drafts in Henrik's database, although I don't know if the aliases
exist for the older ones.

Every I-D author is automatically the beneficiary
of one of these. If they're not spam-proofed, oh dear.

    Brian

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 25 Sep 2008 23:22:26 +0200

Reducing to what I believe is the essential points below (sorry for coming
late to this discussion -- I've been away from mail for some 6 hours now,
and you've sorted this all out in the meantime ,;-) :

On 2008-09-25 17:55 Marshall Eubanks said the following:
..
> It is his solution, after all.

Yes and no.  I'd prefer to put postconfirm in front of the aliases on the
@ietf.org addresses; that will make the confirmation requests (where needed)
come from the original addressee, and it will avoid a daily transfer of
updated subscriber whitelists from mail.ietf.org to tools.ietf.org.

This has also been something I've desired all along; I have anti-spam
measures in effect at tools.ietf.org too, but I already know that these
are much less effective against spam forwarded through mail.ietf.org,
and thus have looked forward to putting something like postconfirm in front
of the aliases.

Unless this turns out to be impossible for some reason, I'd like to move
forward with this solution.

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 25 Sep 2008 23:36:26 +0200

Hi Glen, all;

Commenting on one particular point below:

> using aliases, which will give each user more options for spam handling?

The thing with these aliases is that they provide a very useful service in
a manner which (if we can fix the spam issue) costs both the senders and
receivers very little.  The fact that for e.g. the sip WG the addresses
[email-masked] and [email-masked] are stable across changes of chairs
and ADs, and continue to work without the need of any manual intervention or
additional work by senders, chairs, ADs or secretariat make them hard to
beat...

Now, regarding the solution which I think would be worth-while to pursue
here, which several people already suggested, of putting postconfirm in
front of the aliases; I think (and hope) that this will prove fairly straight-
forward.  I think that if Glen and I talk over how postfix handles aliases,
we'll find the right place to slot postconfirm in.

Regards,

	Henrik

---

From: Glen <[email-masked]>
Date: Thu, 25 Sep 2008 14:52:01 -0700

> of one of these. If they're not spam-proofed, oh dear.

These are a separate issue - since those aliases exist ONLY on the tools
server, which Henrik manages, I will have to defer to him entirely on that.

For the stuff on the AMS IETF servers, Henrik and I will implement his 
solution!  :-)

Glen

---

From: Glen <[email-masked]>
Date: Thu, 25 Sep 2008 14:51:12 -0700

> we'll find the right place to slot postconfirm in.

I am at your service, as always, as you know.  Email me directly and we'll
get it moving!

To the rest of you: THANK YOU for participating.  As I predicted, Henrik
will work some magic and I will assist as the apprentice (no Mickey Mouse
jokes, plaese! ;-)   Thank you Jim for the original suggestion as well!

Glen

---

From: Marshall Eubanks <[email-masked]>
Date: Thu, 25 Sep 2008 18:00:32 -0400

I may be speaking out of turn here, but I think that all of the  
aliases should be
mailing lists, with the possible exception of ietf-chair, irtf-chair  
and iab-chair (and those could
be actual accounts).

Simplicity !

Marshall

On Sep 25, 2008, at 5:09 PM, Glen wrote:

>> scoring
> IAOC mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/iaoc

---

From: Marshall Eubanks <[email-masked]>
Date: Thu, 25 Sep 2008 18:10:46 -0400

Having caught up with the entire email thread, I see the point of  
Henrik's statements re alias stability under
personnel changes and withdraw this.

I hope that applying postfix is not too hard !

Marshall