Subject: Re: forged bounces, was [Asrg] A CAPTCHA that automatically

From: "Michael Kaplan" <[email-masked]>
Date: Fri, 03 Jun 2005 22:31:07 -0500

> >> it doesn't require that senders remember all the mail they've sent
> >> to know which challenges are to real mail and which to forged,
> 
> > This 'remembering' in invisible and effortless as far as the sender
> > is concerned.
> 
> Um, smoke alert.  Maintaining a database of sent mail for even a
> moderately high volume mail server is a seriously hard problem.
> High-speed updates are the hardest part of a database system, and this
> is a worst case scenario because the info for a message needs to be
> available as soon as the message has been sent, since we can expect an
> automated C/R system to fire back a challenge immediately.  The usual
> way to deal with large volumes of updates is to batch them, but that
> adds delays.  I suppose you could batch the database updates and delay
> the outgoing mail until the batch runs, but now you have another
> locking problem to know when to release the outgoing mail.

Is maintaining this database of sent mail really such a serious technical issue, at least as far as maintaining a reasonably functional bounce/challenge filter?  Is this really more of a technical challenge than a system such as Domain Keys?

I will not argue any further if others state that this is a serious technical problem.  I appeal to any other knowledgeable individuals to either support or negate the above position.

Thank You,
Michael Kaplan

-- 
_______________________________________________
NEW! Lycos Dating Search. The only place to search multiple dating sites at once.
http://datingsearch.lycos.com

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: der Mouse <[email-masked]>
Date: Fri, 3 Jun 2005 23:53:39 -0400 (EDT)

> Is maintaining this database of sent mail really such a serious
> technical issue, at least as far as maintaining a reasonably
> functional bounce/challenge filter?  Is this really more of a
> technical challenge than a system such as Domain Keys?

I do not know the details of either.  But, speaking as an implementer,
I would much prefer to be tasked with implementing something basically
stateless like Domain Keys than anything requiring saving any
information about sent mail and feeding it forward to assist with
incoming bounce handling.

Of course, this could be based on a misunderstanding of one or both of
the things at issue; I trust my choice of words above gives enough of
an idea of my understanding for you to tell to what extent any such
misunderstandings invalidate my conclusion.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: John Levine <[email-masked]>
Date: 4 Jun 2005 04:55:36 -0000

>incoming bounce handling.

That's exactly the point.  DK, or any other signature system, puts
all of the message's state in the message itself, so the sending
host can then forget about it.  Anything that requires that the
sending host remember all the mail it's sent adds vast complication.
As Mark D. pointed out, for large domains like Yahoo not only are
there many sending hosts, but they can be spread around the world.

R's,
John

---

From: "william(at)elan.net" <[email-masked]>
Date: Fri, 3 Jun 2005 22:05:14 -0700 (PDT)

> sending host remember all the mail it's sent adds vast complication.

This is exactly the problem with SES and why I told them its not workable
large-scale, although otherwise its good idea.

BTW - the same is also true for BATV (in fact the technologies are really 
identical except SES allows for public dns verification server and BATV 
its all private).

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: John Levine <[email-masked]>
Date: 4 Jun 2005 06:06:28 -0000

>identical except SES allows for public dns verification server and BATV 

BATV doesn't require the sending host to remember anything about
messages it's sent.  It uses signatures.

R's,
John

---

From: "william(at)elan.net" <[email-masked]>
Date: Fri, 3 Jun 2005 23:18:30 -0700 (PDT)

> messages it's sent.  It uses signatures.

So does SES. But what happens is that if you just encrypt bounce address
this is visible on the net and the address can be reused (replay attack).
The partial solution to that is to include timestamp and change keys 
often, but that does not eliminate it - just makes it more difficult to
organize it.

Another approach is to include message body in signature hash. But that
means that to validate the signature you must have original message body
as well. The problem is that unfortunetly when bounce is received back
it does not always include original message and when it does it can be
mangled or it could have been changed in transmission before it is 
bounced or more likely the mail server doing bouncing is responsible
when it adds its own comments or when it removes some attachment, etc.

So in the end you're left with option of using message body to create 
unique bounce-address signature at the time email is sent but also have
to keep that signature in your database to be able to validate it. So
why bother with cryptography then? It might as well be that you create
long random number (alphanumeric random) and use that for each bounce
address and keep that in database.

---
William Leibzon
Elan Networks
[email-masked]

---

From: Dave Crocker <[email-masked]>
Date: Sat, 4 Jun 2005 06:53:05 -0700

>  identical except SES allows for public dns verification server and BATV

Even with John's posting to correct this mis-statement, it appears the 
point was missed:  BATV places the state information into the message.  It 
does NOT require the BATV creator or interpretor to retain state 
externally.

(BATV allows alternative algorithms.  If someone wants to go use the BATV 
syntax and design a scheme that DOES require external state retention, 
then, well, yes that scheme would require external state retention. 
However the documented scheme does not, nor does the one envisioned using 
public keys.)

Therefore, the statement: "the same is also true for BATV" is incorrect.

  d/
  ---
  Dave Crocker
  Brandenburg InternetWorking
  +1.408.246.8253
  dcrocker  a t ...
  WE'VE MOVED to:  www.bbiw.net

---

From: "william(at)elan.net" <[email-masked]>
Date: Sat, 4 Jun 2005 09:58:04 -0700 (PDT)

On Sat, 4 Jun 2005, Dave Crocker wrote:

> point was missed:  BATV places the state information into the message.
> It does NOT require the BATV creator or interpretor to retain state
> externally.

My statement was exactly correct - SES and BATV work almost the same
way and suffer from the same problems. Both allow for alternative 
algorithms too so one algorithm could require retention of state on
the origin server (never on intermediate sites as far as I know,
although SRS appears to do just that...) and one would not but suffer
from other problems.

Let me go through this slowly:

I. How this all works

   Both SES and BATV add an encryption signature into RFC2821 MAILFROM
   (bounce) address. Right now both are using symmetric key cryptography
   so I'll focus only on that and not on any proposed public key scheme.
   I think (though BATV did not specify) both SES and BATV are using HMAC
   symmetric key cryptographic algorithm, though really actual algorithm
   does not make any difference.

   The signature added is a hash of the data which would always include
   actual email address for RFC2821 MAILFROM and may also on other data.
   These other data can be:
    1. Time of when signature was added
    2. HASH of actual email body
   When time is added, the timestamp would also have to be added as separate
   data in RFC2821 so that signature could be verified (its possible though
   to use time from "Date" header field)

   The difference is that SES provides for public verification server. So
   for SES a system that wants to send a bounce, would when it sees the
   signature call this server and ask - IS THIS SIGNATURE CORRECT?
   If answer is no, it would not send a bounce.

   For BATV, the bounce is always sent and then when its received, the
   original sender system (that added signature) would try to check if
   signature is correct and if not would discard the bounce to /dev/null

II. Problems with MAILFROM signature approaches

   The data from RFC2821 is not 100% private for mail system interaction only.
   It is seen by users in Return-Path field ans some of the locations where
   email come to are archived in real-time. It is sometimes revealed in
   trace data, in particular some mail MTAs add it to Received as part of
   nowhere defined "envelope-from" clause or sometimes in (envelope-from ..)
   comment. It is required part of the Received-SPF trace header field, it
   maybe data that goes into Redirected trace field. It maybe data that is
   found in Authorization-Results field. Entire trace data is publicly
   available for mail lists and almost all mail lists keep public archives
   with majority of the archives providing access to raw message data as well.

   So any signature data used there should be considered to have become
   publicly available info once message has been sent and is subject for
   use in replay attack (replay means that somebody, probably a bad guy,
   would create a message and use somebody else's MAILFROM signature).

   It should also be noted that the bigger number of MAILFROM signatures
   sent that are substantially the same, the easier it is to find in public
   places and use that for replay attack.

   1. MAILFROM signature is signed hash of MAILFROM address

    Totally open to replay - just find somebody else's full MAILFROM address
    with signature and start sending messages with with the same address
    and then all bounces would come to the victim.

    However one form of protection that could be used is changing private
    key often, say every day and then if signature is received that is not
    signed by one of the last 5 keys used, it can be considered invalid.
    This is almost the same as MAILFROM with timestamp described below,
    just that timestamp is not used and is simulated by changing keys.
    It suffers from the same problems plus with timestamp each signature
    would be unique where as with changing keys they would be the same
    for entire day.

   2. MAILFROM signature includes timestamp

    In this case replay attack would only be possible within email delivery
    time period after signature was sent (usually considered 5 days).
    However I do not believe that it would be very hard to create a program
    that would look at various mail list and similar archives and find address
    for spammer use within 5 day period in automated manner. So all the same
    problems as classic replay are all there.

    Possible protection can be achieved by having sender keep track of
    bounces received and if more then one or two are received for same
    message, all other bounces would be rejected. This does mean keeping
    database on the sender side - but its only database of bounces and
    not all mails sent.

   3. MAILFROM signature includes hash of message data

    In this case each message signature is guaranteed to be unique, which
    is good (but same uniqueness can be done with timestamps above). The
    problem is that to be able to verify the signature entire message data
    or hash of it would have to be known. This gives several options:
     a. Include hash of message as part of addition data in the MAILFROM
        Unfortunately the maximum size for MAILFROM local part is 64 bytes
        and with signature already being cryptographic data, it takes most
        of it and including additional cryptographic hash (SHA1 is 20 8-bit
        data bytes and 32 for BASE32 representation which would be needed
        for MAILFROM) is not much of an option.
     b. Keep track of the hash on the server that added the signature and
        only include in the MAILFROM unique id of the database record or
        include in the database entire MAILFROM record with signature.
     c. Do MAILFROM signature verification only when message DATA is available

    For BATV option c is possible (preferred?) since bounces are dealt
    with after being received, but the problem is that bounce messages do
    not necessarily include original message data intact: sometimes bounce
    received would only say message has been rejected without specifying
    as to why, sometimes the data would include only original message header
    but not the body, sometimes it would be only one body part, etc. So
    I do not believe that this is viable and workable solution for BATV
    in current email environment.

    For SES, the option c is not possible since SES does verification at
    the time of MAILFROM as part of SPF check. They do appear to try a,
    but run out of space in too many cases... So only viable option is b
    which is to keep local database of all messages sent and hash or
    signature that ended up as a result. But as I've already noted if
    you keep local database, why bother with cryptography as one can just
    randomly generate long string, attack that to MAILFROM, i.e.
    "[email-masked]" and put "rrrrrrrrrr"  in local database
    and keep track of bounces that come back and check if its valid.
    This also is still open for replay attack as one can just take this
    bounce from public location and attempt to send messages like that and
    it would validate. To deal with replays, the verifying server would
    have to keep track of bounces, most likely in opposite way to what
    I described in 2 as instead of keeping track of bounce, when bounce
    is received, its unique rrrrrrrrrr is removed from the database and
    so it would no longer validate.

    However the problem with having to keep local database of all sent
    messages is that its not scalable for when it needs to be deployed in
    the environment with millions of messages sent per hour. So while
    it may work for small personal domain, for large one it would not.

III. Summary & Conclusion

    Of above the most viable option appears to be 2 which is to use timestamp
    to guarantee uniqueness of MAILFROM signatures in all messages sent and
    keep database locally for bounces that are received to deal with replays.
    Additionally changing of keys often would also help in insuring
    cryptography is not easily broken and allow for additional check to
    restrict when signature is still considered valid.

    SES people have lately been advocating #3 (include message signed data
    in signature) which I think is total mistake and would make their
    solution largely unworkable in anything but a small test environment.

    If comparison is between SES and BATV with both creating signature
    with only MAILFROM and timestamp (BATV is then basicly same as SES
    but without public access to verification server), then SES is better
    because it offers an option to remotely verify if signature is correct
    and that verification can be easily integrated with SPF and so allow
    all bounces to be discarded where as with BATV bounces would all be sent.

    Additionally I think in order to ensure replay attacks can not easily
    be done, it needs to be mentioned that mail servers adding trace data
    SHOULD only include in that original localpart and note entire MAILFROM
    signature, then finding the signature to use for replay would not be
    easy in automated manner.

    The above requires awareness of the signature system and format for all
    MTAs (and not only of the sender as BATV has been saying) and that is
    why I have strongly advocated some months ago that BATV, SES and SRS
    (which could also suffer from replay attack) all agree on common syntax
    but unfortunately BATV and SES people can't stand each other and their
    personal dislike is going to be downfall for all of their inventions.

    BTW - in my personal opinion both BATV and SES could/should be merged
    and become one technology with some people running public verification
    server for SPF and some (those who do not support SPF) not running
    it and doing verification locally for received bounce.

For further technical reference see:
  BATV -  http://www.mipassoc.org/batv/draft-levine-mass-batv-00.txt
  SES - http://ses.codeshare.ca/files/Working_SES_Format_Definition_16.html 

> Therefore, the statement: "the same is also true for BATV" is incorrect.

That statement was correct. And I dare you to tell me that I dont understand
the technologies or that I'm seriously wrong in above description and 
evaluation.

---
William Leibzon, Elan Networks:
  mailto: [email-masked]
Anti-Spam and Email Security Research Worksite:
  http://www.elan.net/~william/emailsecurity/

---

From: John Levine <[email-masked]>
Date: 4 Jun 2005 17:35:51 -0000

>organize it. ...

>address and keep that in database.

This completely misrepresents the purpose and function of BATV.  I
know that we've had this exact same exchange at least once before, and
I really wish you would cut it out.

BATV does one thing -- it defends against bounce blowback from mail
forgery.  It does not purport to be an all-purpose anti-spam or
anti-forgery technique.

The point of BATV is to ensure that whoever sent you a bounce had a
message from you in hand when it sent the bounce.  That's all it does.
It does that by adding a one-way hash of the bounce mailbox and domain
to the bounce address that the original sending domain can check.  As
we've noted, the MTA needs no per-message state to generate or check
the hash.

People say "it doesn't defend against replay."  Well, duh, it doesn't
even try.  There's two reasons for that.  One is that replay isn't a
problem and isn't likely to be.  It is exceedingly rare to see a
deliberately forged bounce, and it is hard to see a plausible scenario
in which bad guys would want to forge them, other than as DOS attacks
for which they already have lots of other techniques.  The other is
that even if it seemed desirable to do so, you can't, since there is
no way to tell how many bounces a message should legitimately get.
You can't tell whether a message is going to one person's mailbox or
to an exploder that will remail it to a thousand people.  My version
of BATV has a timestamp which I added mostly so that if someone
scrapes old bounce addresses off web mail archives, BATV will reject
them, but I have seen so few BATV failures due to expired timestamps
that I'm not sure I would put in the timestamp if I were doing it
again.

We have contemplated one minor extension. Once the DK signature scheme
settles down, we might do a version of BATV that uses the DK key to
create a remotely checkable version of the signature.  There's still
no per-message state, and it's only signing the bounce address, but
this allows as an optimization that a remote MTA could check and
decline to generate the bounce instead of the original MTA rejecting
it.  This is a fairly minor optimization, since the current scheme
rejects bad bounces very efficiently.  The PC where abuse.net lives
rejects 400,000 bounces on a bad day, without noticably increasing its
load.

SES attempts to extend a BATV-like system to an all purpose
anti-forgery and/or message validation scheme, which fails for all of
the reasons that William mentioned and more.  So we're not going to do
that.  If you want to prevent "too many" deliveries, don't bother.  If
you want message body signatures, use DK (or DKIIM or whatever it is
now.)  If you want to defend against forgery bounces, use BATV.  If
you want to do both, use both, but please don't waste your time and
ours discovering yet again that one combined scheme won't solve both
problems.

R's,
John

---

From: "william(at)elan.net" <[email-masked]>
Date: Sat, 4 Jun 2005 11:39:13 -0700 (PDT)

> I really wish you would cut it out.

In my view the main purpose is to stop forgery and spoofing of MAILFROM 
address and make sure that only legitimate bounces are sent on the 
internet.

But if you can achieve that and carry some additional data as part of 
MAILFROM, there is nothing substantially wrong with that either...

Its probably that last part that you disagree with me about, but
for now we can forget about this debate. I'm no longer developing 
techniques related to carrying anything extra in MAILFROM, there is
just not enough place for it there.

> the hash. People say "it doesn't defend against replay."  Well, duh, it 
> doesn't even try.

Ding, ding! IETF security personnel to the evaluation desk, fast!

>  There's two reasons for that.  One is that replay isn't a
> problem and isn't likely to be.

You don't know it.

> It is exceedingly rare to see a deliberately forged bounce, and it is 
> hard to see a plausible scenario in which bad guys would want to forge 
> them

After being a victim of half a dozen joe-jobs, I disagree.

I'm saying that they will continue to do as they do now, but in more
complex way - they will run program that will look for encrypted MAILFROM 
address in some public mail list (which people like you and me, who they 
don't particularly like, participate in quite good numbers and post often)
and will extract MAILFROM address recorded there and use in their spam.
Their spam messages would be rejected and all come to you and me, how do 
you like that? Is your system really working then?

> other than as DOS attacks for which they already have lots of other 
> techniques.

> The other is that even if it seemed desirable to do so, you can't, since 
> there is no way to tell how many bounces a message should legitimately 
> get. You can't tell whether a message is going to one person's mailbox 
> or to an exploder that will remail it to a thousand people.

First of all exploder with thousand of recipients is probably a mail list
and those reset MAILFROM to themselve.

I agree that you can't easily tell how many legitimate bounces could be 
received, but I believe you can set the number to something low like 2 or 
3 and be fine with it. Can also try to increase default number if message 
is going to known "exploder' or just has multiple recipients in RCPT TO 
but it would require database record for sent message even if it would 
not be for every one and those with multiple recipients, it would still 
be a problem for busy mail server, so it might not be scalable.

And BTW, I doubt that your use of BATV for abuse.net can be considered
good experimentation on if replay is a problem because I dont think
you're using abuse.net regularly as your MAILFROM address and you also
seem to be the only user on that domain.

> My version of BATV has a timestamp which I added mostly so that if 
> someone scrapes old bounce addresses off web mail archives, BATV will 
> reject them, but I have seen so few BATV failures due to expired 
> timestamps that I'm not sure I would put in the timestamp if I were 
> doing it again.

Good. Your BATV use with timestamp is exactly what I have in mind. The 
only thing else is that its good to change private key often. If I 
understand SES also worked the same way when originally done, but its
designers went into wrong direction...

> this allows as an optimization that a remote MTA could check and decline 
> to generate the bounce instead of the original MTA rejecting it.

There is not enough space in localpart of MAILFROM to use anything but
384bit public key crypto (and security reviewers for MASS were quite clear 
that you want 1024bit keys). So its not strong enough and given how its 
already subject to replay attacks, I doubt of the value of using public 
key unless local-part could be extended so that it has more then 64 bytes.

> ours discovering yet again that one combined scheme won't solve both

You might be amazed to discover that it actually can, though not quite
in the way you imagined. Stay tuned for next week's online paper 
presentation coming to asrg and other relevant lists.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: der Mouse <[email-masked]>
Date: Sat, 4 Jun 2005 16:16:19 -0400 (EDT)

>> There's two reasons for that.  One is that replay isn't a problem
>> and isn't likely to be.
> You don't know it.

"Isn't a problem" - John apparently has experience as a high-profile
forgery target that says it *isn't* a problem.  Do you have countering
experience to give?

"Isn't likely to be" - that's a judgement call, but, tentatively, I'd
certainly be inclined to believe him.  Do you have evidence to the
contrary?

>> It is exceedingly rare to see a deliberately forged bounce, and it
>> is hard to see a plausible scenario in which bad guys would want to
>> forge them

Forged bounce != bounce of forgery.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign