Subject: Re: 7. BCP - Mail Administrators: Checking HELO (was: [Asrg] 0. General - Administrative - for M. Wild)

From: [email-masked]
Date: Mon, 1 Sep 2003 00:55:49 -0400

On Sun, Aug 31, 2003 at 02:09:34AM +0200, Brad Knowles wrote

> 	If it's behind a NAT, how would it know the external DNS name? 
> If that IP address on the NAT device is dynamically assigned and the 
> machine is not an intelligent host running software capable of 
> updating a dynamic DNS record (as 99.999% of all NAT/router devices 
> are almost certainly going to be), then how would the internal host 
> know what this external DNS name is?

  That point is moot... because if you're sending direct to the remote
MX from a dynamic IP address, there'll be a helluva lot more machines
rejecting you for being on a dynamic address than those worrying about
a syntactically correct HELO.  IOW, your mail is a lot more likely to
be accepted with an incorrect HELO from a static IP address.

> > One could argue that if they don't understand DNS, they shouldn't
> > run a mail server.
> 
> 	I would like to be able to make such an argument.  I really 
> would.  Indeed, I could even see making such an argument part of a 
> BCP, but only if the recommended practice did *NOT* involve rejecting 
> the connection outright just because they did not have reverse DNS 
> properly configured.

  How about total lack of rDNS ?  I block on that, not on mismatching
rDNS.

-- 
Walter Dnes <[email-masked]>
Email users are divided into two classes;
1) Those who have effective spam-blocking
2) Those who wish they did

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Steven F Siirila <[email-masked]>
Date: Mon, 1 Sep 2003 15:12:54 -0500

On Sun, Aug 31, 2003 at 03:49:24AM +0200, Brad Knowles wrote:
> At 8:04 PM -0500 2003/08/30, Steven F Siirila wrote:
> 
> > We have seen about 1,200,000 different IP addresses connecting to us in 
> > the
> > past six months and we block about 1,000,000 SMTP connections on our MX
> > servers every week on average.  What other stats would be of interest?
> 
> 	What's your breakdown of why those connections are blocked?  How 
> many unique blocks are due to which blacklist?  How many unique 
> blocks are from lack of rDNS?  How many unique blocks are there by 
> domain-based blacklist?  How many unique blocks are there due to 
> other techniques?

I will try to post some data soon.

> 	I assume that once you have a positive response on one of those 
> checks, you reject the connection and you don't bother trying the 
> other checks as well.  Could you re-process your data to apply all 
> checks to all IP address/envelope sender/recipient combinations, to 
> see where certain blocks might overlap?

No, we reject on potentially multiple reasons.  Dynamic IP, no rDNS when
rDNS is required, known spammer, known open proxy, known open relay, etc.

---

From: "Tom Thomson" <[email-masked]>
Date: Mon, 1 Sep 2003 21:43:58 +0100

> 3) Require the parameter to be an FQDN and check whether its A record
>    matches the sender IP.
>
>    This will additionally block all senders which don't have an FQDN or
>    don't know it (which currently seems to be true for a lot of open
>    proxies, but may change quickly).
>
> I see very little legitimate mail in my log files which doesn't pass
> tests 1) to 3). For the few cases which would break on 3), I would
> consider it reasonable for the administrator on any legitimate server to
> get an FQDN and a matching A record. This is even possible for dynamic
> IP addresses via dyndns.net and similar services.

I think test 3 is two steps too far.  Suppose I normally send through the
(obiously private/closed) SMTP relay mailer.somecorp.co.uk so I pass
tests 1,2, 3, annd 4 without trouble.  The one day mailer.somecorp.co.uk
is horribly dead (teh thames burst its banks and its under 8 feet of water
and mud); what I will want to do is send the mail direct from my desktop
so it can either say "helo tom.somecorp.co.uk" and fail tests 2 & 3  since
that machine has no public IP address, it's behind a firewall which provides
address translation and never under any circumstances passes an incoming
connection request to it or I can say something like
"helo (10.1.1.67) Sorry about the crazy IP, but I'm nasty NAT and DHCP"
and pass test 1 (yes, that extra text after the closing bracket is a
legitimate part of an "address literal") and fail test 2 because 10.1.1.67
is not the translated address that gets to your server, and test 3 because
I've used an address literal instead of an FQDN.

So maybe it would be better to have just two tests:
1) the helo record contains either an FQDN or a an address literal
2) If it contains an FQDN then either the DNS lookup for that FQDN delivers
the sender IP adress, or the reverse lookup for the sender IP address
delivers something with a reasonable relationship to that FQDN; if it
contains
and address literal, either the address part of that literal is the same as
the sender IP or it is in a range which is not allocated for "normal" use.

The disadvantage is that it fails to discriminate against transmissions
which
have the characteristics which you don't like; the advantage is that it
doesn't
com[pound a disaster (in the suggested event of a flood, it might be a life
and
death mater to get the email out).

Of course RMX styule proposals fit in perfectly well with this view - we
just
make all the public addresses used by our NAT system appear in the RMX
records
for somecorp.co.uk.  So you could add that test too if some form of RMX-like
proposal were agreed (which would also force people to be a little more
secure
in their mail habits at the same time as improving spam attributability).

Tom

Tom Thomson

---

From: "Hector Santos" <[email-masked]>
Date: Sun, 31 Aug 2003 08:04:28 -0400

the
> This would also be relevant to the analysis area. We are definitely
> interested in all the data we can get.

Here is some statistics I can provide::

In regards to RBL,  I wanted to see how spammers are able to learned from a
growing database of open relays and proxies systems to exploit the idea that
a good percent of these sites do not fix their system immediately.

Our RBL implementationin Wildcat! SMTP mail server uses (or was until this
week) relays.osirusoft.com and make good use of the 127.0.0.X.  response
codes.

Using our April 2003 log, 8062 RBL filters were recorded.

Total blocks: 8062

127.0.0.2      872     Verified Open Relay
127.0.0.3      206     Dialup Spam Source
127.0.0.4     2253    Confirmed Spam Source
127.0.0.5        1        Smart Host (In progress)
127.0.0.6     3336    A Spamware software developer or spamvertized site.
127.0.0.7        0        A list server that automatically opts users in
without confirmation
127.0.0.8        0        An insecure formmail.cgi script. (Planned)
127.0.0.9     1388    Open proxy servers

Next.  I wanted to test what percentage of the Open Relays were still open
so I wrote an open relay tester that was feed a list of IPs.  Using just the
open relay sites, the result was:

Total checked: 807
Bad connects : 446   55.3%
Good Connects: 295   36.6%
Open relays  :  66    8.1%  or (18.3% of GOOD connects)

This was just for the month of April/2003.  The test was performed on  May
10, 2003.   The test took a few hours to complete as it had to connect to
the systems and perform a standard open relay test.  (PS: please don't ask
me where the missing 65 sites (872-807) are.  I probably hit escape because
it was talking so long).

I wanted to fine tune the analysis by adding a time element,  "how long did
it take for the relays to get fixed?"  however,  I didn't want to go any
futher as my main goal in the analsys was to see how useful is the database
to the spammers to quickly find new open relays and proxies as they are
being added.

So just using this one analysis, in one month time, atleast 80% of the
ligitimate systems do fix there systems, however, spammers are left with a
good amount of sites that they can use to spam the world.

I suggest that spammers use the databases with the knowledge that they have
X days where a majority of the sites are still open, declining every day,
however, enough to provide ample sites to do their dirty deeds.

I have the data and the C code if anyone whats to investigate this "time
element."

In regards to a breakdown of SMTP conversations,  for the month of August, I
have breakdown:

August/2003

1 day
2 connects
3 accepts
4 helo/ehlo rejects
5 rbl rejects
6 total rejects
7 unknowns ( (unknown users, etc)
8 pct accepts

1   842 142 0 371 371 255 16.86
2   768 111 0 392 392 217 14.45
3   773 196 0 366 366 199 25.36
4   926 356 0 328 328 218 38.44
5   853 218 0 395 395 226 25.56
6   883 242 0 423 423 211 27.41
7   1044 266 0 412 412 341 25.48
8   824 201 0 425 425 181 24.39
9   679 94 0 411 411 174 13.84
10   719 167 0 327 327 239 23.23
11   691 112 0 317 317 241 16.21
12   2032 1475 4 309 313 232 72.59
13   776 216 0 289 289 208 27.84
14   727 148 0 343 343 208 20.36
15   660 105 0 276 276 251 15.91
16   648 120 0 280 280 231 18.52
17   586 88 0 211 211 246 15.02
18   759 214 0 268 268 251 28.19
19   1341 241 0 269 269 233 17.97
20   944 178 0 298 298 281 18.86
21   1357 339 11 239 250 222 24.98
22   1516 221 275 147 422 275 14.58
23   963 201 141 199 340 259 20.87
24   1169 167 401 151 552 261 14.29
27   1820 246 213 401 614 277 13.52
28   1693 155 265 491 756 332 9.16
29   1273 236 300 386 686 251 18.54
30   412 61 153 86 239 71 14.81

Note: I didn't begin helo/ehlo rejects until the 21th.   Next month will
give me better data. But it looks on average, 15-20% is rejected just on
helo/ehlo.

I am currently working on a stats program for my trace logs.  What I found
out is repeatibility information.

---

Hector Santos
WINSERVER "Wildcat! Interactive Net Server"
support: http://www.winserver.com
sales: http://www.santronics.com

---

From: "Peter J. Holzer" <[email-masked]>
Date: Sun, 31 Aug 2003 09:25:44 +0200

On 2003-08-31 02:09:34 +0200, Brad Knowles wrote:
> At 11:19 PM +0200 2003/08/30, Peter J. Holzer wrote:
> 
> > RFC 2821 requires the parameter to be either an FQDN or an address
> > literal. A client which sends an unqualified hostname is in violation of
> > the RFC without any good reason. (broken software and lazy
> > administrators are not good reasons, IMHO)
> 
> 	I don't see this requirement in 2821:

3.6 Domains

   Only resolvable, fully-qualified, domain names (FQDNs) are permitted
   when domain names are used in SMTP.  In other words, names that can
   be resolved to MX RRs or A RRs (as discussed in section 5) are
   permitted, as are CNAME RRs whose targets can be resolved, in turn,
   to MX or A RRs.  Local nicknames or unqualified names MUST NOT be
   used.  There are two exceptions to the rule requiring FQDNs:

   -  The domain name given in the EHLO command MUST BE either a primary
      host name (a domain name that resolves to an A RR) or, if the host
      has no name, an address literal as described in section 4.1.1.1.

> 	Note that the RFC explicitly states that the IPv4 address should 
> be in square brackets.  Which is precisely the kind of behaviour that 
> I believe you said that you were refusing.

I didn't say I'm refusing anything. I listed four different tests one
could apply to the hello parameter and tried to identify who would be
affected by those tests.

In the first two tests, address literals are allowed, in the other two
they are forbidden.

> > Any reason why it must identify itself as [10.0.1.5]? Why not with the
> > external IP address or as bradknowles.dyndns.net?
> 

The external name stays fixed. You register a name at a dynamic dns
provider (I am using dyndns.org (sorry, not dyndns.net, that seems to be
something else) as an example here because a few people I know use it -
there are others). Then you configure your mail server to always use
that name in the helo command. Finally you run a program which detects
your real address (e.g., by connecting to http://checkip.dyndns.org/)
and updates the DNS record at regular intervals.

Yes, this works only with forward lookups, not with reverse lookups.

	hp

-- 
   _  | Peter J. Holzer    | Humor ohne Emoticons ist trockener Humor.
|_|_) | Sysadmin WSR       | 
| |   | [email-masked]         |	-- Toni Grass in aip
__/   | http://www.hjp.at/ |

---

From: Brad Knowles <[email-masked]>
Date: Tue, 2 Sep 2003 23:13:25 +0200

>  be accepted with an incorrect HELO from a static IP address.

	This is irrelevant.  The point was what you check in the 
{HE|EH}LO and on what basis you reject if there is some sort of 
failure, not where the IP address may be from.  After all, it could 
easily be coming from a dynamic IP address assignment that is not 
known to the appropriate RBLs.

>    How about total lack of rDNS ?  I block on that, not on mismatching

	Can you be sure?  If Dean Anderson were to send you a mail 
message from his mis-configured machines in av8.com (e.g., 
concorde.av8.net -> 130.105.11.50 -> relay1.av8.net, or 
concorde.av8.com -> 130.105.11.3 -> concorde.av8.net -> 130.105.11.50 
& 130.105.11.3), would you accept or reject that message on the basis 
of the way he has reverse DNS set up?  What method have you used to 
ensure that this is the case?

	I ask this because the postfix option of reject_unknown_client 
will reject a connection for either non-existent rDNS or incorrect 
rDNS.  Many IP addresses will have essentially useless rDNS defined 
for them by their ISP, even if the person using that IP address is 
totally unaware of this fact.  Are you sure that your code (or 
sendmail itself) doesn't do the same?

-- 
Brad Knowles, <[email-masked]>

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."
     -Benjamin Franklin, Historical Review of Pennsylvania.

GCS/IT d+(-) s:+(++)>: a C++(+++)$ UMBSHI++++$ P+>++ L+ !E-(---) W+++(--) N+
!w--- O- M++ V PS++(+++) PE- Y+(++) PGP>+++ t+(+++) 5++(+++) X++(+++) R+(+++)
tv+(+++) b+(++++) DI+(++++) D+(++) G+(++++) e++>++++ h--- r---(+++)* z(+++)

---

From: [email-masked]
Date: Tue, 2 Sep 2003 21:58:47 -0400

On Tue, Sep 02, 2003 at 11:13:25PM +0200, Brad Knowles wrote

> sendmail itself) doesn't do the same?

  It's not "my" code.  I'm a customer off clss.net.  They run a modified
Qmail that parses a config file in the customer's home directory after
the RCPT: stage. (I don't admin the MTA, I admin the filters for my
account).  Any email that the config file decides to reject gets the
big 550 before the DATA: stage.  There are 3 different rules that might
apply here.  Direct quotes from "man dnsblfilter"...

       PARANOID reply
              If the sending IP address has a reverse DNS pointer
              that is not matched by a forward (address)  record,
              reply is printed and the message is rejected.

       REJECTNOHOSTNAME reply
              If the sending IP address has no name in any avail-
              able address-to-name database, reply is printed and
              the message is rejected.

       SUPERPARANOID reply
              If the sending IP address has no  name,  or  has  a
              reverse  DNS  pointer that is not matched by a for-
              ward (address) record, reply  is  printed  and  the
              message is rejected. This is equivalent to PARANOID
              combined with REJECTNOHOSTNAME.

  I use REJECTNOHOSTNAME in my ruleset.  It catches really gross stuff
where there simply is no rDNS period.  For instance...

[waltdnes@m450 waltdnes]$ host 192.168.1.1
Host 1.1.168.192.in-addr.arpa not found: 3(NXDOMAIN)

  Let's look at Dean's setup...

[waltdnes@m450 waltdnes]$ host concorde.av8.net
concorde.av8.net has address 130.105.11.3
concorde.av8.net has address 130.105.11.50

  Looks like a load-balancing act.  How do each of the two addresses
work out ?

[waltdnes@m450 waltdnes]$ host 130.105.11.3
3.11.105.130.in-addr.arpa domain name pointer concorde.av8.net.

  That one looks OK.

[waltdnes@m450 waltdnes]$ host 130.105.11.50
50.11.105.130.in-addr.arpa domain name pointer relay1.av8.net.
[waltdnes@m450 waltdnes]$ host relay1.av8.net
relay1.av8.net has address 130.105.11.50

  What's wrong with this one ?  130.105.11.50 has a name (different from
the original name) that resolves back to 130.105.11.50.  I don't see
how this is "misconfigured", unless all load-balancing is considered to
be misconfiguration.

---

From: Steven F Siirila <[email-masked]>
Date: Mon, 8 Sep 2003 14:06:25 -0500

> domain-based blacklist?  How many unique blocks are there due to 

I won't break down on a per-DNSbl basis, but here's some data for the past week:

Insecure server	  342,122	(79,449 unique -- blocked ONLY for this reason)
Known spam src	  351,648	(288,234 unique)
rDNS blocks	  591,570	(378,792 unique)
Dynamic IP addr	1,095,792	(950,424 unique)

Total blocked	2,048,523

-- 

Steven F. Siirila			Office: Lind Hall, Room 130B
Internet Services			E-mail: [email-masked]
Office of Information Technology	Voice: (612) 626-0244
University of Minnesota

---

From: "Chris Lewis" <[email-masked]>
Date: Tue, 09 Sep 2003 10:04:46 -0400

> Dynamic IP addr	1,095,792	(950,424 unique)

Strange.  Is there some sequencing at play?  Which are done first?

Here's moderately comparable breakdowns from our spamtrap (percentage of 
full spamtrap load):

CLASS DUL                  345446   3.04
CLASS MANUAL               154099   1.36
CLASS MISC                4106790  36.12
CLASS PROXY               8804470  77.43
CLASS RELAY                 24925   0.22
CLASS SPAMSOURCE           163314   1.44

[These overlap, but there is no sequencing dependencies]

I would assume your "Insecure server" is analogous to a combination of 
our PROXY and RELAY classes.  One DNSBL, CBL, _alone_ manages to catch 
74% of the 77% that PROXY gets.

What BL are you using for Dynamic?  That 3% is PDL+SORBSdul.

MISC is a combination of several things, including a "No rDNS + 
complaints" and a "spamcop or ORDB block + complaints" BL.

Open relay is isn't worth bothering with anymore, tho, none of the ones 
I'm using are particularly good (SORBSsmtp + a local one - can't use 
ORDB...).

---

From: Jose Marcio Martins da Cruz <[email-masked]>
Date: Tue, 09 Sep 2003 16:16:08 +0200

Hello,

> On Sun, Aug 31, 2003 at 03:49:24AM +0200, Brad Knowles wrote:
...

>>blocks are from lack of rDNS?  How many unique blocks are there by 

This is some stats from our server. Some explanations before...

We accept only 8 connections a day coming from gateways with bad DNS resolution (resolve 
fail or forged).

This is usually nough to accept legitimate messages coming from gateways with bad dns 
resolution.

This kind of filtering is becoming less efficient than before. One year ago, it was 
better, and we could usually block more connections, mainly on weekends and during the night.

The trafic level of this server is around 50000 connections a day.

These numbers concerns only connections from gateways with bad dns resolution.

     Data displayed are in cumulative mode

  SUMMARY       |    06H    12H    18H    24H    48H |
  ----------------------------------------------------
  CONNECT...... |   1293   2281   2947   3957   7291 | connections
  ACCEPT....... |    680   1256   1762   2426   4816 | connections
  REJECT....... |    613   1025   1185   1531   2475 | connections
  REJECT RATIO. |  47.41  44.94  40.21  38.69  33.95 | % connections
  MEAN......... |    2.4    2.5    2.4    2.4    2.4 | connections/gateway
  STD DEV...... |    8.2    8.2    7.7    8.0    7.8 | connections/gateway
  GATEWAYS..... |    529    909   1252   1671   3044 | Total = 3044 addresses
  REJECT....... |     55     81     96    126    194 | gateways
  REJECT RATIO. |  10.40   8.91   7.67   7.54   6.37 | % gateways
  GATEWAYS..... |    220    396    547    729   1332 | 90 % connections

  RESOLVE FAIL  |    06H    12H    18H    24H    48H |
  ----------------------------------------------------
  CONNECT...... |   1176   2094   2667   3532   6379 | connections
  ACCEPT....... |    598   1116   1555   2123   4179 | connections
  REJECT....... |    578    978   1112   1409   2200 | connections
  REJECT RATIO. |  49.15  46.70  41.69  39.89  34.49 | % connections
  MEAN......... |    2.5    2.6    2.4    2.4    2.4 | connections/gateway
  STD DEV...... |    8.7    8.7    8.1    8.4    8.1 | connections/gateway
  GATEWAYS..... |    464    805   1096   1460   2653 | Total = 2653 addresses
  REJECT....... |     49     73     86    110    171 | gateways
  REJECT RATIO. |  10.56   9.07   7.85   7.53   6.45 | % gateways
  GATEWAYS..... |    220    396    547    729   1332 | 90 % connections

  RESOLVE FORGED|    06H    12H    18H    24H    48H |
  ----------------------------------------------------
  CONNECT...... |    117    187    280    425    912 | connections
  ACCEPT....... |     82    140    207    303    637 | connections
  REJECT....... |     35     47     73    122    275 | connections
  REJECT RATIO. |  29.91  25.13  26.07  28.71  30.15 | % connections
  MEAN......... |    1.8    1.8    1.8    2.0    2.3 | connections/gateway
  STD DEV...... |    2.3    2.4    2.7    3.3    5.5 | connections/gateway
  GATEWAYS..... |     65    104    156    211    391 | Total = 391 addresses
  REJECT....... |      6      8     10     16     23 | gateways
  REJECT RATIO. |   9.23   7.69   6.41   7.58   5.88 | % gateways
  GATEWAYS..... |     37     52     76    104    190 | 90 % connections

-- 
  ---------------------------------------------------------------
  Jose Marcio MARTINS DA CRUZ           Tel. :(33) 01.40.51.93.41
  Ecole des Mines de Paris              http://j-chkmail.ensmp.fr
  60, bd Saint Michel                http://www.ensmp.fr/~martins
  75272 - PARIS CEDEX 06      mailto:[email-masked]

---

From: Steven F Siirila <[email-masked]>
Date: Tue, 9 Sep 2003 09:21:27 -0500

> Strange.  Is there some sequencing at play?  Which are done first?

We query all DNSbls all the time.  We tell the remote user (via URLs),
all reasons for the block, not just the first one we run across.

> 74% of the 77% that PROXY gets.

Correct.  Open Proxy, Open Relay, Insecure Formmail.cgi, etc..

> What BL are you using for Dynamic?  That 3% is PDL+SORBSdul.

We use several; most notably MAPS DUL and SORBS.

> I'm using are particularly good (SORBSsmtp + a local one - can't use 

Agreed.

---

From: Steven F Siirila <[email-masked]>
Date: Sat, 30 Aug 2003 20:04:38 -0500

On Sun, Aug 31, 2003 at 02:09:34AM +0200, Brad Knowles wrote:
> 
> 	My subjective views have been based on my experience as the Sr. 
> Internet Mail Administrator for AOL from '95-97 (up to five million 
> users), and the Sr. Systems Architect for Belgacom Skynet from 
> '98-2001 (peak of ~one million users).
> 
> 	I'm not sure if we've got anyone on the list with current 
> experience and access to log data from such large ISPs, but I do know 
> that we have some people on the list with current experience and 
> access to log data from ISPs in the six-figure range.  I would be 
> very interested to see their data.

We have seen about 1,200,000 different IP addresses connecting to us in the
past six months and we block about 1,000,000 SMTP connections on our MX
servers every week on average.  What other stats would be of interest?

---

From: Brad Knowles <[email-masked]>
Date: Thu, 28 Aug 2003 23:59:29 +0200

At 5:04 PM +0200 2003/08/28, Peter J. Holzer wrote:

>  Still, I don't think there are many legitimate sites which don't have an
>  A record.  Requiring the sender to send a FQDN which resolves to the
>  sender's IP address doesn't seem unreasonable to me (even for dynamic
>  IP-Addresses, you can use dyndns.net or a similar service).

	Check the recent traffic on NANOG.  Because of stupidity on the 
part of AOL, they've been discussing this subject intensively.  I've 
been tagging 75-90% of the recent messages as input for the BCP 
review.

	In short, let me say that I used to think this was a good idea, 
but my views are in the process of changing.  Maybe we could tag and 
score on the basis of whether or not the sender's IP address has 
proper reverse DNS (or whether the sender's domain name exists in the 
DNS and/or in some sense "matches" the IP address of the incoming 
connection), but I think it would be foolish in the extreme to 
outright reject messages on this basis.

>                                                                In fact,
>  looking at my log files this seems to be a very good indicator of
>  legitimate mail servers (I checked several weeks of logs some time ago
>  and only found one legimitate server which identified itself with an
>  unresolvable name (I think the box is NATted).

	I'm NAT'ed.  Many people on NANOG appear to be in similar 
situations, or have run into them frequently.

	Just because we haven't personally experienced a particular 
situation doesn't mean that this would necessarily make itself a good 
target for filtering and outright rejection.