Subject: [Asrg] Challenge to Challenge response

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Sun, 20 Apr 2003 11:26:50 -0700

OK this is a research group, how about someone tries an experiment?

I define the problem here to be 'lost emails', not false positives since
some challenge response vendors are already making the dubious claim of zero
false positives on the grounds that the lack of a response means that the
message must have been spam.

In the experiment I propose a number of users install a challenge response
system. We measure the following parameters

1. Number of messages whitelisted
2. Number of messages challenged
3. Time taken to respond to challenge (as histogram)
4. Proportion of legitimate emails where no response was given.

The outgoing message would be the standard one.

After running the experiment for some time the correspondents who did not
reply to the challenge are contacted, are told about the experiment and
asked to fill in a survey on why they did not respond, one of the questions
being how many such responses they have been presented with in the past.

The experiment could then be rerun with a modification to the outgoing
message stating that the message being challenged had been identified as
potential spam and that the message was being chalenged in that particular
instance.

It seems to me that it should be possible to get this type of thing past
human subjects study board, the information being gathered does not relate
to the individuals, does not expose personal data and the study is for the
long term benefits of the subjects.

	Phill
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Art Pollard <[email-masked]>
Date: Tue, 22 Apr 2003 11:57:29 -0600

>         S/MIME gives a much higher degree of assurance that the message is
>from an authentic recipient than C/R with zero UI overhead. If you whitelist
>after the initial C/R you have a very weak authorization system.

Why would it be weak?

If during the challenge response system keys were exchanged (if the 
challenge were met) so that I could verify through a digital signature that 
it actually came from the whitelisted individual then it would be very 
secure.  Furthermore, if the whitelisting occurred based on a signed key in 
the header then the user could use multiple mail clients -- each with the 
same key/signature and pass the whitelisting test from all the people he 
has communicated with in the past.

So given that a challenge response system could be set up complete with 
cryptographic authentication, it should be very secure.  Where am I wrong?

You mention that Outlook and Lotus already have S/Mime already in place. 
Are you also saying that because if this, I get spammed since I am using 
Eudora and that Outlook and Lotus users do not?

-Art
-- 
Art Pollard
http://www.lextek.com/
Suppliers of High Performance Text Retrieval Engines.

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Tue, 22 Apr 2003 12:10:21 -0700

> >after the initial C/R you have a very weak authorization system.

Because the subsequent whitelisted messages are not authenticated, the
credential is not bound to the message.

> challenge were met) so that I could verify through a digital 

OK you are now describing the standard scheme for C/R authentication used in
certificate enrollment. You do that once and the credential can then be used
anywhere.

This problem does not require CA issued certs.

> cryptographic authentication, it should be very secure.  

Your system becomes a variant of the existing cryptographic authentication
schemes in use and in development.

> Eudora and that Outlook and Lotus users do not?

The Eudora people have a problem, Quallcomm has no interest in developing
the product. They have no interest in supporting long established
specifications like S/MIME. Therefore it is unfortunately only a matter of
time before that product becomes obsolete and dies. Sorry, there is nothing
I can do when a product vendor abandons their product like that.

I have discussed the issue with Qualcomm directly they are not interested in
any ongoing product development, whether to stop spam or anything else.

		Phill

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Tue, 22 Apr 2003 13:22:07 -0700

Exactly where is the unfariness?

Is it being asserted that this particular test is unfair? in which case
please suggest modifications. Or is the assertion that any test of C/R is
intrinsically unfair? If so on what grounds?

I think the reason the test is being objected to is that we all know what
the result would be.

Kee does have a good point, it is the reason that I cannot afford to use a
scheme like that. A single lost email could cost my company thousands. I
don't get enough traffic on my personal email address to be worth tracking.

I actually use the spam filters we just installed to quarantine all my mail
so it does not hit my RIM pager. I check every message in quarantine twice a
day.

I propose a second part to the test to see which proportion of the test
candidates offered the use of the scheme are willing to try it. We could ask
some university to install it, we must be comming up to final year project
time somewhere in the world.

I will write up the project proposal after we bat it arround a bit more. If
the C/R proponents want to suggest modifications to make the test fairer
feel free to do so.

	Phill

> At 11:33 AM -0600 4/22/03, Art Pollard wrote:
> >At 11:26 AM 4/20/2003 -0700, you wrote:
> >><SNIP>
> >I don't think that this would be a fair comparison and I don't 
> >believe the results would be useful.
> 
> Agreed, but for a different reason.  Anyone who can't afford to use a 
> challenge response system wouldn't volunteer.

---

From: "Hallam-Baker, Phillip" <[email-masked]>
Date: Fri, 25 Apr 2003 13:29:56 -0700

Absolutely, anything we can test, let us test. The more empirical
measurement the better.

Although one thing to beware of, the fact that 80% of spam has signature
Y does not mean that we can use that fact to provide a robust long
term spamfilter because of the existence of countermeasures.

So measure, yes, but the figures I find significant are 'less than 5%
has feature X', '5-15%', '85-95%', 'more than 95%'. So lets not get
into arguments as to whether the figures are 3.2% or 4.5%. Either 
something is very common, very rare or kinda fuzzy.

		Phill

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Friday, April 25, 2003 4:13 PM
> To: [email-masked]
> Subject: RE: [Asrg] Challenge to Challenge response
> 
> 
> On 2003-04-22 "Hallam-Baker, Phillip" <[email-masked]> wrote:
> There's no need to lose any email for a /count/ of false positives.
> Send a challenge message, but deliver the email as if no challenge
> system was in place.
> 
> If you are going to run a test, I would like to test a lot of systems,
> not just challenge response.  For example, recording whether the
> incomming email uses pipelining, whether the rDNS is "correct",
> whether the IP is listed on blacklist(s), whether the envelope
> from is valid, and pretty much any other data that you can think
> to collect.  Again, you don't actually /block/ the email if it
> fails one of these tests, just record the results of what would have
> happened had that system been in place.
> 
> 
> Scott Nelson <[email-masked]>

---

From: Scott Nelson <[email-masked]>
Date: Fri, 25 Apr 2003 20:58:07 -0700

At 10:19 PM 4/25/03 -0400, Kee Hinckley wrote:
>At 1:13 PM -0700 4/25/03, [email-masked] wrote:
>True.  But I still wouldn't use it in a commercial environment.  As 
>many people have said--c/r systems are annoying.  I'm not going to 
>inflict that on a potential customer.
>

Indeed, and that probably should be tested for as well.
However the question came up, "how many people don't respond
when faced with a challenge?"  I've heard the number zero mentioned,
and I've heard people dispute that number, but I've no evidence in 
either direction.  The obvious answer IMO is to find someone who /is/ 
willing to test it, and count.  Or better still, a lot of someones.

Although I don't like challenge response as a primary defense, 
I do like it as a second line.  That is, I send you a challenge 
if and only if you fail the other spaminess tests.   
It's still annoying of course, 
but not as bad as having your email discarded silently, 
or bounced with no good recourse.  ("Sorry, your mail server's 
IP address is unacceptable.  If you want to send me email, change it...")

BTW - there is a class of people who actually would tolerate
annoying the sender as long as they could insure a near 0 false 
negative rate - parents.  No, I'm not suggesting that we change
the internet to appease them, just pointing out that there's a 
diversity of opinions on what's acceptable and what isn't.

>Detecting pipelining is presumably going to require software changes. 
>For the envelope from and the rDNS; what do you mean by "correct"?
>

Some people check if the rDNS matches the field given in HELO.
(supposedly, a large amount of spam comes from IPs that don't have 
rDNS at all)  That's what I meant by "correct" rDNS.

I expect that challenge response would also require a number of
software changes.  Even if all you do is install TMDA, it's going
to take some real work.  I think the software changes are trivial
compared with finding willing test subjects, but then I've already
written a lot of this stuff, so maybe I'm a bad test point.