Subject: [Asrg] My Opinion regarding ietf asrg session (it went badly!)

From: [email-masked]
Date: Thu, 20 Mar 2003 14:12:30 -0800 (PST)

Hello everybody,

As I have little more time now, so I want to express my opinion regarding 
asrg live session on IETF meeting. I think it went badly and choice for 
presentations were not appropriate for the IETF conference. 

First part, were presentations on how bad spam is, what users should do, 
legal, etc. These are information to people who do not know about the problem 
and people at ietf are quite aware of the problem. Now these kind of 
presentations might have been ok if there were scientific in its 
nature or presentation, i.e. result of research efforts, like ARTG 
is supposed to actually do, but these were just views of particular 
organizations. But even that would have been ok if not for that 
fact, that these presentations took majority of time and being controversial
to some, resulted in lots of discussions. And regarding these discussions,
these were to a degree what we on the list call "noise". Now this noise 
is exactly the reason why some people who are experts in email have left 
this email list and do not participate any more, some of them were 
present at the meeting and I think they saw the same on the life meeting 
and this only made matters worse if we want to get any kind of support 
from IETF in general and IESG for whatever solutions we can think are 
appropriate to stop/reduce/control spam.

Second part of the meeting was supposed to be little more technical (half 
of the room was emtpy by the time it started...) but unfortunetly there 
was really not enough time for it and it was completely dominated by 
vendors trying to push their own solutions and all solutions were very 
commercial in nature. None was big help at this point and none (except 
verisign, though presentation was still one-sided towards S/MIME) tried to 
see what else similar is available that may not be commercial and may do 
the same. Also it can be noted that AT&T presented technology on which 
they have some patents and IETF has very clear guidelines that this is 
dscoraged to say the least and group chairs should when two solutions 
exist (one patented, one not) always choose non-patented solution, same 
applies for presentations, this should have been screened and patent-related
presentation made available if nothing else similar exist.

Also have to be noted that most presentations on the live meeting were by 
persons who did not try to participate at the mailing list and are completely
unknown to us and this is also against general policies for IETF and IRTF 
as first what are supposed to be presented are work done as part of mailing
list efforts (i.e. general work area) usually by people who are most active
on the list and only afterwards at the very end are there allowed presentations
by other organizations with similar interest or work area (and those 
presentations should be carefully chose to minimize commercial views). 

In short, like I said in previous email - the presentations were all 
opinions of particular organizations and all commercial (even spamcon is 
really what I call "commercial" non-profit foundation, as apposed to say 
CAUSE) and were not individual submission as is supposed to be done at 
IETF and IRTF. As a result engineers who were attending IETF probably 
got negative view of our group and our efforts. The only thing I can think 
of why chair did something like that would have been to get media exposure,
I'm sure media not be research or engineering centric would not see 
serious downsides that I have noticed. Now I'n not against additional 
media exposure, but I do not believe we're at the point where this is 
really needed, what we need are people serious about working on technical 
solutions and media will provide information about our group to general 
user audience that would only bring more noise. 

In the future meetings done in connection with IETF (are we meeting in 
Vienna?) if we're going to have so many non-technical discussions, I 
would strongly recommend separting into two sessions, one for this 
general/media-centric and one for actual engineers to work on technical 
issues that we may have. 

I do hope chair takes my comments seriously!
He really did a bad job today chosing what is to be presented (i'm sure he 
had multiple options), keeping discussions short (so that there was time 
for additional presentations) or in general being prepared for this life 
meeting (remember agenda is supposed to be reviewed by mail list and be 
announced well before the actual meeting, that is just one example).

-- 
William Leibzon
Elan Communications Inc. 
[email-masked]

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Hadmut Danisch <[email-masked]>
Date: Thu, 20 Mar 2003 16:50:54 -0800

William, 

I do fully agree with you and can 100% confirm what you said.

As I stated in the session at the microphone, I was deeply
disappointed about the session. Most of what was presented was
simply pointless. I continuously had the feeling of waiting for 
the commercials to end to get back to the subject of the group
- unfortunately they didn't end.

Also, I was astonished to find presentations mostly disjunctive
from what was done on the mailing list. The presenation were mostly  
a playground for commercial interests, and thus the session 
ironically appeared to be jammed by spam. I wonder what the
mailing list is supposed to be good for if it has almost no influence.

Those statistics were somehow interesting, but not worth
wasting the short time we have. And extrapolating an 
exponential growth from just two measurements is at least
questionable. 

I was waiting more than 2.5 hours for the beef to come. Finally, 
after the official end time at 11.30, the group was moving a little
bit to the technical part. Unfortunately, time was so short that three
talks had to be pressed in less than 30 minutes, and technical items
could not even be touched. The presented taxonomy was as short as
poor, but due to the short time left to present it, this didn't matter 
anymore. 

I came from Germany on my own private expense mainly to
discuss technical solutions. If I had known the agenda before,
I simply could have saved the money I wasted. 

I was quite amazed to see so many journalists participating.
I've never seen so many journalists in a IETF session before. 
This could have been a great opportunity to step into public
awareness as the ones who start to solve problem by engineering.
Unfortunately, this chance was lost. 

As a balance, that session didn't bring us any further. On the 
contrary, this casted a very negative light on us. That's why
many attendees, even some journalists, left early and were highly
disappointed, or even upset. 

After this session I am even more convinced, that any effort 
to effectively stop spam will be blocked in order to keep
a billion dollar market of spam filters open.

Hadmut

---

From: Frank de Lange <[email-masked]>
Date: Fri, 21 Mar 2003 02:28:35 +0100

> a billion dollar market of spam filters open.

I agree that the meeting was.... unbalanced. It would have be en more 
appropriate to have a meeting like this at a symposium or conference 
attended by management and 'decision makers'.

Like Hadmut, I came here on my own expense from Sweden. I expected some 
'product presentations', but I also expected 'independent' research 
projects to be presented. Unfortunately, the latter did not really 
figure on the agenda, for whatever reason.

We have three months to Vienna. In those three months, we need to show 
to ourselves and the world - which is watching, as shown by the presence 
of journalists in the meeting and earlier media interest - that the 
IETF/IRTF is an independent organisation which works to create and 
promote a vendor-neutral, univeral internet. Whatever we end up with 
(that is, the working group(s) which might come after this research 
group) is going to be implemented by commercial vendors (and free 
software projects, hopefully). This is a good thing! It should, however, 
not be the other way around. As I have always understood it - and this 
was supported by everyone at the plenary meeting last night - people at 
the IETF work with the best interest of the internet in mind, not the 
best interest of their employers. I trust this is the case.

Frank

-- 
    WWWWW      ________________________
   ## o o\    /     Frank de Lange     \
   }#   \|   /      +46-734352015       \
    \ `--| _/     <Hacker for Hire>      \
     `---'  \       +31-640037120        /
             \    [email-masked]    /
              `------------------------'
   [ "Omnis enim res, quae dando non deficit, dum habetur
      et non datur, nondum habetur, quomodo habenda est."  ]

---

From: Marshall Rose <[email-masked]>
Date: Thu, 20 Mar 2003 17:47:04 -0800

> promote a vendor-neutral, univeral internet.
> ...
    
at the risk of adding the voluminous nature of this mailing list by
stating the very obvious:
    
    the IRTF is not the IETF
    
applying IETF-like expectations to an IRTF research group will result
only in heartbreak.

although it is unlikely that the instigators of this research group had
this intent, no one should be surprised by today's meltdown. let's face
it, an open mailing list that averages 90 messages/day does not mix with
a group trying to do research. neither does press participation.
    
some of my more wagish friends have suggested that "the B ark"
lives. for myself, i merely note the usual about paving things with good
intentions.
    
/mtr

---

From: Frank de Lange <[email-masked]>
Date: Fri, 21 Mar 2003 03:00:26 +0100

> lives. for myself, i merely note the usual about paving things with good

Nâââh, I spotted quite a few dirty telephones here and there...

Really...

//Frank

---

From: Dave Crocker <[email-masked]>
Date: Thu, 20 Mar 2003 21:43:42 -0800

Marshall,

Thursday, March 20, 2003, 5:47:04 PM, you wrote:
MR> at the risk of adding the voluminous nature of this mailing list by
MR> stating the very obvious:
    
MR>     the IRTF is not the IETF
    
MR> applying IETF-like expectations to an IRTF research group will result
MR> only in heartbreak.

On the other hand, we need to ask what the goal of having an open
meeting at an IETF event reasonably should be.

Somehow, I do not think that filling most of the schedule with
presentations primarily trying to tell us that spam is a problem is
appropriate. Nor do I think that presentations that present invalid
statistics or simplistic suggestions are all that helpfuyl.

Somehow, I think that targeting technical research work IS
appropriate.  Perhaps that happened in the half-hour overtime after
the scheduled portion, but of course, many of us had to be elsewhere.

So, yes, we need to make sure we do not review this as if it were a
working group meeting.  However the meeting failed in plenty of other
ways.

d/
--
 Dave Crocker <mailto:[email-masked]>
 Brandenburg InternetWorking <http://www.brandenburg.com>
 Sunnyvale, CA  USA <tel:+1.408.246.8253>, <fax:+1.866.358.5301>

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 21 Mar 2003 23:01:29 -0700 (MST)

> From: Marshall Rose <[email-masked]>

> a group trying to do research. neither does press participation.

Or marketing or management participation.

How can anyone honestly expect research to appear in a group that was
created a matter of weeks before?  What would anyone expect in the
circumstances but general talk about how bad spam is and a pep ralley
for the need to work on the problem?

At the risk of violating IETF canons, I must observe that some of
the strongest condemnations in the list of whatever happened at
the meeting came from people who have made the some of the less
well founded contributions to the mailing list.  From here I can't
tell what happened, but I wonder if sour grape sentiments from some
people about their favorite solutions not being adopted by acclamation
colored other people's perceptions and so reports of the meeting.

For that matter, unless I'n confused, I thnk some of the strongest
complaints about the meeting came from people associated with spam
solutions that have web pages more than 6 months old but (as far
as I can tell) nothing more than talk about how they're obviously
the one true fix.

Vernon Schryver    [email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 23 Mar 2003 10:28:57 -0500

> statistics or simplistic suggestions are all that helpfuyl.

  Since my background is Nuclear Physics, I understand your concerns
about the methodology use to gather the statistics which were
presented.

  Yet... the stats were about as good as anything else I've seen on
the spam problem.  Yes, it was hand-waving, anecdotal and ad-hoc, but
until ASRG started, there was no industry forum which could gather
global statistics.  Even now, it's not clear to me how one would
gather (or pay for the gathering) of potentially billions of messages
scattered across thousands of domains.

  We can't instrument the core routers to see what traffic is spam.
Without that instrumentation, any statistics will *necessarily* be
self-reported, and incomplete.

> working group meeting.  However the meeting failed in plenty of other

  I was hoping for presentations like "This is a picture of the
network.  These points are where we see spam coming from.  These are
the stats for spam at those points.  These are the methods by which we
can provably stop some percentage of the spam.  These are the costs of
doing so."

  Since I've been one of the more vocal people here about spam, I
guess I should volunteer to collect some more statistics.  I'll put
out a call for self-reporting of stats, or a poll, later in the week.

  Alan DeKok.

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 23 Mar 2003 10:33:31 -0500

> for the need to work on the problem?

  The problem has been known for years.  Methods of dealing with it
have been known for years.  It's not naive for people to expect that
the meeting would start off with 30 seconds of "everyone knows spam is
bad... now let's start talking about what we might do"

> colored other people's perceptions and so reports of the meeting.

  The people I talked with who thought the meeting was unproductive
were not proposing the solutions.  The people who were proposing
solutions were frustrated that they had little (or no) opportunity to
do so.

  The vast majority of the meeting was an "intro to spam".  It was
good for the reporters in the crowd, but less so for the technical
people.

  Alan DeKok.

---

From: "Gordon Fecyk - Home" <[email-masked]>
Date: Sun, 23 Mar 2003 15:21:08 -0600

> out a call for self-reporting of stats, or a poll, later in the week.

That would be a great start to solving the problem actually.  Where are the
spam metrics?

I'm seeing the anti-spam "industry" is looking much like the anti-virus
industry before it could've been called an industry.  We know it's a problem
but we don't know how much it really costs us.  Maybe we need to ask
ourselves the same question Rob Rosenberger asked of anti-virus experts:

"How do you justify your job?"

<http://www.vmyths.com/rant.cfm?id=262&page=4>

I'm not suggesting we're not worth what we're being paid (provided we're
even being paid).  I'm suggesting that our bosses might think that.  And the
media.

--
PGP key (0x0AFA039E): <http://www.pan-am.ca/[email-masked]>
What's a PGP Key?  See <http://www.pan-am.ca/free.html>
GOD BLESS AMER, er, THE INTERNET. <http://vmyths.com/rant.cfm?id=401&page=4>

---

From: Ted Gavin <[email-masked]>
Date: Sun, 23 Mar 2003 16:27:49 -0500

>out a call for self-reporting of stats, or a poll, later in the week.

Here's an area where I think SpamCon Foundation can be of help. SpamCon
Foundation, one person's observations notwithstanding, is not a
"commercial non-profit organization, as opposed to CAUCE". SpamCon and
CAUCE do different things by design. In fact, CAUCE and SCF share many
participants because of that. 

On behalf of SpamCon Foundation, we'd be happy to commission a project
to develop exactly the type of data ASRG needs to get things going. But
to do that, this group is going to have to start developing a list of
actual needs.  

First, can this group agree as to what type of report *would* be helpful
in the group fulfilling its mission? Alan, you seem to have started that
list. Some more detail behind the desire (such as exactly what stats for
spam emanating from certain points) would be nice. 

- Size?
- Content?
- Routes/Paths through the network?
- Sender/Receipient/Ultimate beneficiaries?

What of the other hundreds, if not thousands, of delimiting factors?

Ted

---

From: Vernon Schryver <[email-masked]>
Date: Sun, 23 Mar 2003 15:51:13 -0700 (MST)

> From: "Alan DeKok" <[email-masked]>

> good for the reporters in the crowd, but less so for the technical

Meetings are above all about non-technical, human issues.  People
with legitimate claims to being "technical" know that such meetings
are not where research gets done, or is even likely to be first
reported.  Particularly in the IRTF/IETF world, a lot gets reported
first the mailing lists.

I stand by my guess about sour grapes from avocates of silver bullets.
The common silver bullets have nothing but months (e.g. hashcash) and
sometimes years (e.g. authentication and sender-pays money) of
non-technical and often technically inaccurate talk supporting them.

I've previously proposed that one should be ashamed to offer any solution
that requires X% adoption to have X% effectiveness for X>=0.01%.  One
should also have a very good and obvious reason for adovcating any spam
solution that is more than a year old but that has not been implemented.

Talk seems to be almost as cheap in the anti-spam world as the spam world.

---

From: "Alan DeKok" <[email-masked]>
Date: Sun, 23 Mar 2003 13:11:02 -0500

[email-masked] wrote:
> to develop exactly the type of data ASRG needs to get things going.

  That sounds like an excellent start (and potentially less work for
me.)

> - Sender/Receipient/Ultimate beneficiaries?

  The stats I'm most interested in are listed below.  Some ISP's may
be concerned about releasing "private" information (total traffic,
etc), so the questions may have to be normalized as percentages,
rather than absolute numbers.

  Having historical numbers would also be good.  e.g. Not just the
current numbers, but numbers for a number of previous years.

  I believe that gathering statistics on spam content or origin will
not currently be useful.  I would like to focus on simple & easily
gathered numbers, which will make the analysis less subject to
personal interpretation.  Similarly, the work (and amiguity) in
analysing network routing of spam & sender/recipients is too difficult
to be helpful right now.

  I would also like to focus the stats on domains reporting more than
10,000 spam messages a day.  That will limit the initial size of the
survey, and will reduce the noise due to systems at the low-end.

  Alan DeKok.

--------  Questionaire on Spam for Mail System Administrators ----

I) Size of the problem
 1. Volume
  a) Total spam volume (Mb/day)
  b) total number of spam messages 
  c) total number of "do-nothing" SMTP connections
     (e.g. connect, EHLO, disconnect)

 2. Relative amount of spam
  a) spam as a percentage of total email (spam messages / total messages)
  b) spam as a percentage of *non-spam* email (spam messages / accepted
     messages)

 3. Count of IP's
  a) number of IP's sending spam
  b) number of IP's sending "valid" email (may overlap with 3a)

 4. Costs associated with dealing with spam
  a) administrative (network support solely due to spam)
  b) technical (end-user) support
  c) additional infrastructure/machine costs
  (costs to end users should be initially avoided, as they're more
   difficult to quantify)

II) Current Solutions which address the problem
 1) blacklists
   a) SMTP conversations blocked due to the blacklist, as a percentage
      of the total
      (blocked conversations / total conversations)
   b) SMTP conversations blocked, as a percentage of non-spam email
      (blocked conversations / allowed conversations)

 2) whitelists
   a) spam which is in the whitelists, as a percentage of the total
      number of SMTP conversations
   b) spam which is in the whiteliests, as a percentage of the
      non-spam SMTP conversations

 3) content filtering
   a) percentage of non-spam marked as spam (false positive)
   b) percentage of spam missed by the filter (false negative)
   c) percentage of spam caught by the filter
-----
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: [email-masked]
Date: Sun, 23 Mar 2003 13:54:12 -0800 (PST)

I'm curious what are your ideas on how to do it in more in terms of scientific
research. I mean statistics previously given were very "broad". What I see 
that has to be done is first statistics on how much email goes into what 
types of mailservers (i.e. statistics on amount of email at larger isps - 
aol, earthlink, msn) as compared to amount of email at smaller isps as 
compared to amount of email at free email providers (hotmail, yahoo) as 
compared to amount of email at enterprise as compared to amount of email 
at web-hosting companies as compared to email small individual machines
(hosted on dls, cable, etc). After we have this statistics (how?) we a 
certain sample can be chosen for each of the groups and then statistics 
gathered. Just doing sample at aol or at enterprise will not provide 
accurate statistics!

> --------  Questionaire on Spam for Mail System Administrators ----
Add here first - Total number of email messages. Total amount of email 
received (i.e. traffic statistics on SMTP protocol).

>   b) spam as a percentage of *non-spam* email (spam messages / accepted
Would like to all see:  size of spam messages / total size of email messages
			size of regular emails / total size of spam

>   a) number of IP's sending spam
Please by geographical area. When reverse dns is present - by country 
(.cn) and by larger isp net (i.e. dsl at verizon, etc)

>   b) number of IP's sending "valid" email (may overlap with 3a)
And its interesting to know how many overlap!!!

>   a) administrative (network support solely due to spam
Difficult to quantify ...
>       (blocked conversations / total conversations)
What kind of blacklists!!!

This would be usefull for additional statistics on particular list of 
respondents especially if we really do it on wide basis.

>    a) percentage of non-spam marked as spam (false positive)
This one is really really hard to tell. Most non-spam marked is spam is 
never seen by readers. So actual percentage that is reported would be very 
very small.

---

From: [email-masked]
Date: Sun, 23 Mar 2003 13:57:25 -0800 (PST)

FY: I meant statistics previous given were not very "broad" and particular 
to one or couple isps! Stupid of me to forget to add "not"...

---

From: Kee Hinckley <[email-masked]>
Date: Sun, 23 Mar 2003 20:01:10 -0500

>			size of regular emails / total size of spam

I assume that we understand that these numbers are going to be very 
inaccurate?  Because if they were accurate, then presumably we'd know 
how to identify spam and we wouldn't be having this conversation.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Jim Youll <[email-masked]>
Date: Sun, 23 Mar 2003 20:23:24 -0500

At 20:01 -0500 3/23/03, Kee Hinckley wrote:
>inaccurate?  Because if they were accurate, then presumably we'd 
>know how to identify spam and we wouldn't be having this 
>conversation.

discussion reminds me of this...

"Criminal laws in this area are constitutionally limited to hard-core
pornography. I shall not today attempt further to define the kinds of
material I understand to be embraced with that shorthand description. ...
But I know it when I see it."
--Justice Potter Stewart (concurring), Jacobellis v. Ohio, 378 U.S. 
184,  198 (1964).

w/r/t numbers, I ran a 30-day study of my own Hotmail box last year, 
with these results...

			Size(K)	Count
Legitimate messages	406	46
Spams in inbox		1319	269
Spams in junk mail box	3819	552
Total spam		5138	821

"Spam" included only messages from entities I had never had contact 
with, not including advertisements (wanted or unwanted) from vendors 
with whom i'd done business or contacted deliberately.

---

From: "Alan DeKok" <[email-masked]>
Date: Mon, 24 Mar 2003 05:52:14 -0500

> how to identify spam and we wouldn't be having this conversation.

  I'm not quite sure what you're getting at.

- The numbers will be precise
  byte or message counts are easy to calculate correctly

- The numbers may be accurate
  they will count exactly the messages which the ISP's believe to be
  spam, and not any other messages

- The criteria for determining *which* messages are spam is vague,
  imprecise, ill-defined, and subject to personal bias

  However, by measuring data from from biased systems, we can get a
good indication of what those biases may be, how extensive they are,
and even discover the shape of the non-biased systems.

  Pollsters use these methods all the time.  They call people, collect
precise, accurate data about personal biases.  The people may lie, but
with enough data, that bias may be accounted for.

  I'm not saying the numbers will be perfect, or perfectly collected.
I'm coming at it from a method used in physics: A small step forward
is better than sitting still.

  What amazes me is the number of people who would rather do nothing
than use an imperfect system, and the number of people who fight
against something they agree with, because they think it doesn't go
far enough.

  Can we please learn how to walk before we start running?

  Alan DeKok.

---

From: "Alan DeKok" <[email-masked]>
Date: Mon, 24 Mar 2003 06:00:34 -0500

[email-masked] wrote:
> I'm curious what are your ideas on how to do it in more in terms of
> scientific research.

  By that, I would mean "research conducted according to established
scientific methods."

> ... After we have this statistics (how?) we a 

  You're arguing against a position no one advocated.

> (.cn) and by larger isp net (i.e. dsl at verizon, etc)

  I disagree completely.  Asking for those statistics will likely mean
that many companies will not participate.

> Difficult to quantify ...

  So?  They can pick a ballpark number.  It's still better than
nothing.

...
> What kind of blacklists!!!

  That question should be initially avoided.  There is too much
emotion involved in the issue.

  Alan DeKok.

---

From: Ted Gavin <[email-masked]>
Date: Mon, 24 Mar 2003 12:32:10 -0500

>> how to identify spam and we wouldn't be having this conversation.

[snip]

>  Can we please learn how to walk before we start running?

To that point, if a comprehensive survey is going to be done, my
impression is it should be as thorough as possible. ASRG's mission
necessarily places a lot on our collective plate. However, I have a
responsibility to SpamCon Foundation's donors to not spend their donated
funds on wasted or frivolous efforts. So, before we start throwing away
dollars (in actual costs, time, resources, or getting a university to do
the research project if it becomes large enough), we need to make sure
that the effort is worthwhile.

This group (ASRG) is part of our constituent population, as are each of
the participants here. That being said, this group needs to not only
learn how to walk, but at least have both legs working on the same
effort. To translate - if it's actual data we're looking for, then we
should at least agree, as a group, what data we want.

So, I recommend that the Chair determine the sense of the group and
state whether a survey of the sort we've been discussing here is germane
to this Research Group fulfilling its mission. If so, then perhaps we
all might have some discussion as to what data is necessary for the
survey to be actually useful to the group is fulfilling its mission.

               Ted Gavin * [email-masked] *
 Trustee & Officer, SpamCon Foundation  <http://www.spamcon.org>
              A California Non-Profit Organization
   Protecting email as a medium of communications and commerce
          Donations: <http://www.spamcon.org/donations>
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Dave Crocker <[email-masked]>
Date: Mon, 24 Mar 2003 13:02:36 -0800

Alan,

I would appreciate your clarifying your comments.  My confusion is
between survying attitudes vs. surveying behaviors.

Survey research is excellent for assessing people's attitudes.  In this
context, "bias" is about preferences. (Even with this we have remarkable
sensitivity to survey question formulation, making valid and meaningful
survey construction something of a black art.)

Survey research is very nearly useless for assessing people's actual
behaviors., past, present or future. This is because people do not track
and record the behaviors objectively, so their "self-report" is one of
their subjective sense of things.

For very simple assessments of very simple behaviors (did you wake up
yesterday, or which of two candidates will you vote for today) survey
technology can do pretty well.  Not for anything more complicated.

d/

Monday, March 24, 2003, 2:52:14 AM, you wrote:
AD> Kee Hinckley <[email-masked]> wrote:
AD>   However, by measuring data from from biased systems, we can get a
AD> good indication of what those biases may be, how extensive they are,
AD> and even discover the shape of the non-biased systems.

AD>   Pollsters use these methods all the time.  They call people, collect
AD> precise, accurate data about personal biases.  The people may lie, but
AD> with enough data, that bias may be accounted for.

Ad/

---

From: "Alan DeKok" <[email-masked]>
Date: Mon, 24 Mar 2003 11:45:33 -0500

> between survying attitudes vs. surveying behaviors.

  I would say that spam is about personal attitudes and network
behaviours.  Personal attitudes are difficult to measure.  Network
behaviour is not.

> their subjective sense of things.

  I agree, so far as your comments apply to individuals.  Individuals
do not objectively record their personal behaviours.  e.g. How many
times did you take a shower last year?  That information is not
generally recorded, and is subject to personal recollection.

 Individuals do, however record some kinds of information about their
public interactions with others (networking).  e.g. What was your
credit card bill for last January?  If you don't know, odds are you
can call the credit card company, and they'll tell you.  e.g. How much
money did you spend last year on your credit card buying flowers for
your wife?  That data is readily available.

  For spam, I was trying to acheive self reporting of behaviours which
are currently recorded, or which were easy to record.  Measuring how
much email an currently MTA gets is trivial.  The data exists, and
it's easy to summarize.  It involves no personal recollection, or
personal bias.

  Deciding *which* emails are spam involves personal bias.  But once
you've decided upon a flavour you like, it's again easy to measure how
you are implementing that bias.  Sufficiently large populations will
result in a bias which can be modelled, and thus accounted for.

> technology can do pretty well.  Not for anything more complicated.

  Simple behaviours include:

- how many packets were forwarded by your network?
- how many packets were dropped?
- how many SMTP connections did you receive?

  These questions are easy to ask, and easy to answer.  Collecting
such data will allow us to make better statements about how bad the
spam problem really is.  It should also give us some insight as to why
certain behaviours are happening.

  e.g. 10,000 SYN's from spammers a second is a good explanation for
why the ISP has firewalling port 25, and is using a whitelist.

  The point of the gathering these statistics is to gain confidence in
our understanding of the scope of the problem, and of our
understanding of the current methods people use to deal with it.

  Alan DeKok.

---

From: Kee Hinckley <[email-masked]>
Date: Tue, 25 Mar 2003 08:37:35 -0500

>and even discover the shape of the non-biased systems.

If an ISP is not filtering for spam, they can't even answer the 
question, "how much spam do you get".  If they are filtering for spam 
and 20% gets through, they their numbers will be 20% off.  How are 
you--looking at the aggregate information--going to figure that out?

>with enough data, that bias may be accounted for.

This isn't bias.  This is missing data.

>against something they agree with, because they think it doesn't go

I did not say do nothing.  I simply pointed out that you should 
realize that there is going to be a lot of error in some of those 
numbers.  Trust me.  If I had meant "do nothing" I would have said 
"do nothing."
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

---

From: "Alan DeKok" <[email-masked]>
Date: Wed, 26 Mar 2003 09:51:25 -0500

Kee Hinckley <[email-masked]> wrote:
> question, "how much spam do you get".

  Which is why I proposed asking the question "how many emails did you
mark as spam."  My original post did not make say that, but I tried to
clarify my position in the discussion which followed.

> If they are filtering for spam 

  You don't.  You can't.  You have no way of knowing how accurate
their marking system is.  But you DO know that they've marked some
number of messages.

> numbers.

  Not exactly.  The questions which are asked should be such that any
answers will contain little measurement error.  They may contain
system errors, but since we have no way of quantifying them, there's
little we can do to deal with them.

  However, over large populations, we can assume that the systematic
errors follow some standard distribution (e.g. gaussian).  We can then
correct for their effects.  The important thing is that we can reduce
unknown systematic errors from 1000's of domains to a *model* of
systematic errors with a few well understood parameters (distribution,
mean, variance)

  Alan DeKok.

---

From: Kee Hinckley <[email-masked]>
Date: Wed, 26 Mar 2003 16:21:56 -0500

>little we can do to deal with them.

I guess that when asking how much spam they have, we should also be 
sure to ask how they are measuring that.  Assuming we have some false 
negative/false positive data on some tools, so that should enable us 
to adjust the results accordingly.
-- 
Kee Hinckley
http://www.puremessaging.com/        Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society