Subject: Re: [Asrg] 2.c.1 and 6 - The importance of being Earnest

From: "Kurt Magnusson" <[email-masked]>
Date: Fri, 11 Apr 2003 09:23:58 -0200

Kee Hinckley wrote:
At 8:22 AM -0200 4/10/03, Kurt Magnusson wrote:
> > these URL's and phone numbers and shrunk my indata to > ca 1500 entries, 
>but it got far more effective. From ca 20 spam
...........
>
>I agree that it's a good filtering technique. However there are an infinite 
>number of email addresses and web urls. Plus there are

Skip the email addresses, we know they are fake. But in the spam 
repositories mentioned in an earlier thread "Spam Corpuses", if they have 
the full body, we can today block over 90 % of the existing websites and 
call centers, without collect any new and possible tainted data. And it is 
probably a ratio 100000:1 for e-mailaddresses to URLs/numbers. I.e. the data 
will be much less than any mail address dbs.

Almost every present adverticers would need to register new domains and 
contract new call centers or change phone no. As I wrote, if this need to 
happen once or twice, the less economic stable adverticers 
(sometime=spammers) will start to wonder if spam do pay.

>many many ways to encode those, making the filtering process time consuming 
>unless it takes place on the end-user machine. It is also a

I looked at the data I have since one year back and there are 4 basic 
URL-types,  normal, undisguised; IP-numbers (easy to identify); IP-numbers 
with decimal coding (or what it called - also easy to compare) and then the 
"web disguised" URLs, with web coded chars, non readable.

The first 3 types is not a problem, it is just to get spams and extract them 
for a match. The last is never used by ordinary e-mailcommunication, so when 
URLs with several web/mime char combinations with %, ; , & and = is 
encountered (as the =20 combination) assume spam and dump on site. We are 
not talking web, but e-mail with web addresses in them.

>reactive solution. Spam gets through until someone someone adds the new 
>data to the database. And that has to be a manual process.

I agree, that is the biggest disadvantage of the "earnest" method. But how 
is the present spamlist do it today? Are they really fully automatic.

I havent done it today, but as I run it I could let my URL/phone pattern 
filter process extract the data and update my database automaticly. I do 
catch some unknown URLs because I still have email domains, used in earlier 
spams, that way. I have not had one false positive since my last bug 
correction some time ago, but ca 1 "new" spam a day is not caught.

Just now I have a process based on visual filtering, but it could be based 
on me forwarding to an extra account doing this filtering and updating my 
local data.

As you mentioned, if on a "global" basis, it could be tainted, by spammers 
sending in mails, with correct domains, as well as people annoyed by a net 
user or a company/organization.

A solution to this is the ISPs.

Their abuse-centers could have a spam mailbox, you could send spam related 
mail to. The mailbox would do a first sorting against existing data and 
baysian filters to remove the obvious ones, having maybe a minor number each 
day to examin visually. They should also doing the first sorting with online 
reporting to the block list maintainers, so the first report of a new 
address, stops any further manual reviewing at other ISPs.

This function will only be for the ISP's, rest of us have to wait to when 
the list maintainer releases a new list, once a day? It is also secure, to 
not allow spammers to taint new data.

>Spammers frequently include links to legit sites in their spam--you don't 
>want to accidentally blacklist those. Finally, there's a

Yes, I agree, this is not solved, but with the steps above, it should be a 
lesser issue (in a global sense) and there have to be a process where a 
legit site can contact the list maintainer, get a copy of the spam and 
explain or start legal proceedings against the forger (if identified). As of 
today, if you can explain the mail or prove its falsity, you get of the 
hook.

>question of how you share this information, and how you trust what gets 
>shared. Spammers could pollute a database with valid URLs, thus

Why do we trust the present blacklists, Verisign or Hotmail. The process is 
similar to the present blacklisting, but with other data and _hopefully_ 
some support and finansing from the large ISPs/mail suppliers as well as the 
larger national bodies, handling domain naming.

>making people less likely to use it. And even the best intentioned users 
>screw up--that's one reason why people don't universally use IP 
>blacklists--too many false positives.

As I wrote, my idea is not perfect, it need other functions, some outlined 
above. But, as I found out, it decreases the number of mail to process 
further.

As I have it today, I have 50% of my spam from Korea, Japan and Taiwan, 
these I loose to 100%, since they include ASCII-8 chars, not in english, 
danish, norwegian, swedish, finnish, "baltics", islandic, german, french, 
greenlandic or any other language I do not know, but could have a letter 
with, because I identified some of these asian chars, loosing them before 
they get to the spamfilter.

If you loose 90 %, the rest could be attacked more effectively and 
forceibly, without extraordinary resources.

>I think the technique is a great addition to a BCP for filtering. But I 
>can't see a way that it can be used universally. (Is anyone doing a BCP for 
>filtering?)

Hope some of the above answer this issue.

Regards Kurt

_________________________________________________________________
MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*. 
http://join.msn.com/?page=features/virus

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Kee Hinckley <[email-masked]>
Date: Fri, 11 Apr 2003 10:06:10 -0400

At 9:23 AM -0200 4/11/03, Kurt Magnusson wrote:
>>many many ways to encode those, making the filtering process time 
>>consuming unless it takes place on the end-user machine. It is also 
>>a
>
>I looked at the data I have since one year back and there are 4 
>basic URL-types,  normal, undisguised; IP-numbers (easy to 
>identify); IP-numbers with decimal coding (or what it called - also 
>easy to compare) and then the "web disguised" URLs, with web coded 
>chars, non readable.

I was thinking more of where the email itself is encoded (e.g. base64 
and the like).  But it's not an insurmountable problem--just a speed 
issue.

>This function will only be for the ISP's, rest of us have to wait to 
>when the list maintainer releases a new list, once a day? It is also 
>secure, to not allow spammers to taint new data.

It's secure only if the reporting process is made and kept secure. 
Never mind the issue of which ISPs you trust.  Again, not a 
show-stopper, but keep in mind that there are rogue ISPs, and 
certainly there are going to be employee access issues.

>>Spammers frequently include links to legit sites in their spam--you 
>>don't want to accidentally blacklist those. Finally, there's a
>
>Yes, I agree, this is not solved, but with the steps above, it 
>should be a lesser issue (in a global sense) and there have to be a 
>process where a legit site can contact the list maintainer, get a 
>copy of the spam and explain or start legal proceedings against the 
>forger (if identified). As of today, if you can explain the mail or 
>prove its falsity, you get of the hook.

Okay.  So basically what you are describing is virtually the same as 
what we have to day, but with a centralized filtering system instead 
of centralized RBLs.

I guess I'd argue that this already exists and works pretty much the 
way you are proposing.  This is what anti-spam companies do for a 
living.  Because of the manual nature, it takes people and resources 
and thus is a paid service--but I'm sure that most of them use URL 
blocking as you describe.

Of course I found it amusing that at the MIT Spam Conference there 
were almost no commercial talks--and those companies that did talk, 
didn't give any details on the techniques they used to block spam. 
But I shouldn't talk--that's why we didn't present either.

>Why do we trust the present blacklists, Verisign or Hotmail. The process is

That trust is far from universal.  You tend to get what you pay for 
(except with Verisign, which still falls flat on its face when it 
comes to customer support, even though I *am* paying for it).
-- 
Kee Hinckley
http://www.messagefire.com/          Junk-Free Email Filtering
http://commons.somewhere.com/buzz/   Writings on Technology and Society

I'm not sure which upsets me more: that people are so unwilling to accept
responsibility for their own actions, or that they are so eager to regulate
everyone else's.
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg