Subject: RE: [Asrg] Requirements for gathering statistics

From: "Sauer, Damon" <[email-masked]>
Date: Tue, 25 Mar 2003 16:27:50 -0500

I am trying to turn this back into a productive thread. I think it is
important and I don't want to cause it to die because of a disagreement.

 Believe me, I am *really* trying to understand what it is that we are
measuring and for what purpose.
 Maybe if you break your thoughts down for me, I can understand what it is
that we intend to accomplish.

 As I see it, these measurements are in part, useful as a analysis of the
advantage or disadvantage of different types of anti-spam measures. Or are
we even planning on breaking up the data in such components? If not, why
not? The data is available.
 Since just *having* anti-spam measures is a way to deter spam, I think that
showing these numbers is important.
How many spammers give up spamming a host just because they have
anti-spam... and why?

 Any other suggestions on how we can uniquely analyze the data?

Regards, 
Damon

-----Original Message-----
From: Sauer, Damon 
Sent: Tuesday, March 25, 2003 3:36 PM
To: 'Alan DeKok'; [email-masked]
Subject: [unproductive arguement]RE: [Asrg] Requirements for gathering
statistics 

"Sauer, Damon" <[email-masked]> wrote:
> Regardless of how the information is "meant" to be used. Any spammer
> can point at these numbers and say- 50% of our junk is getting through
<best
> Homer voice> Whoo Hoo!. 

  Do we really care about the opinions of spammers?

--- "Keep your friends close but keep your enemies closer."

> I believe that you are not looking at control sets in the correct
> light. In ANY scientific study you MUST have control sets or the
information
> is invalid.

  Nonsense.  Data collection doesn't require control sets.

  Data *analysis* may use control sets, but even that depends on how
you analyze the data.  There is no control set required to analyze the
data gathered from asking "how much spam is received by your domain?"

I think I'll stop here.  I *completely* disagree with your opinions
on data collection and analysis methodology.  I'll appeal to
authority, and let my background as a published nuclear physicist
stand for itself.

---- I believe that here we can agree to disagree.. being a published
(anonymously) postmaster, (not sure what nuclear physics has to do with
spam, but I am sure that it has its applications) I know that just by being
"protected" by anti-spam systems that some spammers just give up. My
internal numbers clearly show that 1/3 of all my spam traffic dropped off as
soon as "announced" myself as a site protected by certain anti-spam
measures. I am talking on the order of hundreds of thousands a day. Gone.
 I choose to not argue the point further, maybe I am just misunderstanding
what you intend to measure with the numbers. I usually tend to start
measuring from the beginning of the ruler rather than somewhere in the
middle, but then of course, I don't build nuclear power plants and the such.

Regards,
Damon

*****
"The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential, proprietary, and/or
privileged material.  Any review, retransmission, dissemination or other use
of, or taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited.  If you received
this in error, please contact the sender and delete the material from all
computers."
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked]
Date: Tue, 25 Mar 2003 11:42:10 -0800 (PST)

Usefullness of statistics depends directly on how they were gathered. 
I think what we need is to see exactly what is being proposed as far as 
how the data will be gathered, I'm not too happy to have this point left 
to some specific organizations, I'm fine with them conducting surveys but 
only if we know exactly how they will be do it.

---

From: "Alan DeKok" <[email-masked]>
Date: Tue, 25 Mar 2003 11:55:04 -0500

> measuring and for what purpose.

  We wish to gain confidence in our understanding of the current
impact spam has on our networks.  Some statistics exist, but as Dave
Crocker pointed out in the meeting, they're small samples, self
reported, and are insufficient for achieving a high level of
confidence.  (That's a loose paraphrase.)

  So we need a survey, with a large number of respondents.  The survey
needs to determine how much spam the respondents are seeing, and what
impact it has on their network.

  The purpose of any survey should be to collect data which is
objective, and requires no subjective judgement on anyone's part.
This will make the survey data much more useful, and will give us more
confidence in the results of any subsequent data analysis.

  There is NO need for any "control" in such a survey.  We will be
asking for statistics about the current traffic on peoples networks.
We will NOT be asking people to compare their networks to others.  We
will NOT be asking about people's opinions about their network
traffic.

> advantage or disadvantage of different types of anti-spam measures.

  I disagree.

  We have no way of knowing the effectiveness of anti-spam measures
used by the respondents.  There is no question we can ask about that
effectiveness which will give us a meaningfull answer.  The
respondents will be using many different anti-spam methods, some home
brewed.  This makes it nearly impossible to determine what is meant by
"anti-spam measure".  We will therefore NOT know what the data means,
so the data will be useless, and any analysis will be a waste of time.

  In contrast, we can ask the simple question: "How many emails did
you decide were spam?"

  The answer to such a question is objectivem and it's an easy
measurement to make.

  In addition, a survey with the goal of measuring the effectiveness
of anti-spam solutions will be socially contentious.  (Or more so...)

> How many spammers give up spamming a host just because they have

  How do you expect to be able to tell?

>  Any other suggestions on how we can uniquely analyze the data?

  Let's not talk about data analysis until such time as we're sure we
can gather meaningful data.

  Alan DeKok.