Subject: [IPsec] I-D Action:draft-ietf-ipsecme-ipsecha-protocol-00.txt

From: [email-masked]
Date: Sun, 5 Sep 2010 20:15:09 -0700 (PDT)

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the IP Security Maintenance and Extensions Working Group of the IETF.

	Title           : Protocol Support for High Availability IKEv2/IPsec
	Author(s)       : R. Jenwar, et al.
	Filename        : draft-ietf-ipsecme-ipsecha-protocol-00.txt
	Pages           : 16
	Date            : 2010-09-05

IKEv2 and IPsec protocols are widely used for deploying VPN.  In
order to make such VPN highly available and failure-prone, these VPNs
are implemented as IKEv2/IPsec Highly Available (HA) cluster.  But
there are many issues in IKEv2/IPsec HA cluster.  The draft "IPsec
Cluster Problem Statement" enumerates all the issues encountered in
IKEv2/IPsec HA cluster environment.

This draft proposes an extension to IKEv2 protocol to solve main
issues of "IPsec Cluster Problem Statement" in Hot Standby cluster
and gives implementation advice for other issues.  The main issues to
be solved are:
o  IKE Message Id synchronization : This is done by obtaining the

message Id values from the peer and updating the values at the

newly active cluster member after the failover.
o  IPsec SA Counter synchronization : This is done by sending

incremented values of replay counters by the newly active cluster

member to the peer as expected replay counter value.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-ipsecme-ipsecha-protocol-00.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

---

From: Pekka Riikonen <[email-masked]>
Date: Mon, 6 Sep 2010 07:43:13 +0200 (CEST)

>From the draft:

   There were some concerns about the current window sync process.  The
   concern was to make IKEv2 window sync optional but we beleive IKEv2
   window sync will be mandatory.

The IKEv2 message id sync is definitely mandatory, but the IPSEC SA seqno
sync IMHO isn't.  Although, none of this would be an issue if IKEv2 would 
allow initiator to move the window forward freely (that would be real 
"fix").

I'm not sure if any of the following should be mentioned in the draft:

- Multiple failovers.  In a cluster with three or more nodes, it's 
possible that failovers happen in rapid succession.  The protocol and the 
implementation must be able to handle new failover even if they are still 
processing the old failover (for example doing the IPSEC SA seqno sync, 
which, if we leave it in, can take time).  I see no reason why the 
protocol wouldn't allow for this, but implementations must make sure they 
can handle it.

- Simultaneous failover at both ends.  If failover happens at the same 
time in both ends, implementations must be able to handle situation where 
they receive SYNC_SA_COUNTER_INFO request before they receive response to 
their own request (they may receive the response only after 
retransmission).

- Should there be way to negotiate support for IKE SA window sync but not 
for IPSEC SA sync?  Currently by sending SYNC_SA_COUNTER_INFO_SUPPORTED 
you agree to support both, which can mean you may receive the IPSEC SA 
syncs.

There's also still bunch of typos in the draft.

	Pekka

---

From: Raj Singh <[email-masked]>
Date: Mon, 6 Sep 2010 22:41:24 +0530

Hi Pekka,

Thanks for your comments. Please find my reply inline.

Best Regards,
Raj

On Mon, Sep 6, 2010 at 11:13 AM, Pekka Riikonen <[email-masked]> wrote:

> The IPsec SA replay counter sync is required for these reasons:
1.  In cluster environment, IPsec SA reply counter will get get updated from
active to standby in
     periodic manner. Suppose, we sync IPsec reply counter for every 1,000
IPsec packets and
     last sync happened at 30,000. So, the next sync will happen 31,000.
     Now, say failover happened at 30, 500. So, standby member becomes
active, and it start
    using IPsec replay counter from 30, 000. It will be considered as Replay
Attack and SA has to be destroyed.
    This applies to both outbound and inbound replay counters.

   So, IPsec replay counter sync is required as it mitigate the these
attacks.

2. Another option is doing IPsec Rekey after failover. Now consider a
scenario where gateways has thousands of SA.
    and after failover, we are doing IPsec rekey for them. The gateway will
be so much loaded. If we follow IPsec replay counter
    sync mechanism, we can sync many IPsec SA with single sync message.

So, if replay attack and load on server is OK, this option is not required.

I'm not sure if any of the following should be mentioned in the draft:
> Very interesting point! Opened a ticket for it.
http://trac.tools.ietf.org/wg/ipsecme/trac/ticket/196

> I don't think we have any solution for this and any solution is possible
(?). This basically means that both sides has lost the SA.
So we have to establish SA from scratch. This draft assumes, that both side
has the SA and their message windows has mis-matched.

> This is very much possible and we can have flag for IPsec SA sync in
SYNC_SA_COUNTER_INFO_SUPPORTED notify payload.
If that flag can be TRUE if we support IPsec SA sync, else we support only
IKEv2 SA sync.
More discussion on this will be interesting.

> _______________________________________________
> IPsec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ipsec
>

---

From: Pekka Riikonen <[email-masked]>
Date: Mon, 6 Sep 2010 19:48:02 +0200 (CEST)

On Mon, 6 Sep 2010, Raj Singh wrote:

: > The IKEv2 message id sync is definitely mandatory, but the IPSEC SA seqno
: > sync IMHO isn't.  Although, none of this would be an issue if IKEv2 would
: > allow initiator to move the window forward freely (that would be real
: > "fix").
: >
: > The IPsec SA replay counter sync is required for these reasons:
: 1.  In cluster environment, IPsec SA reply counter will get get updated from
: active to standby in
:      periodic manner. Suppose, we sync IPsec reply counter for every 1,000
: IPsec packets and
:      last sync happened at 30,000. So, the next sync will happen 31,000.
:      Now, say failover happened at 30, 500. So, standby member becomes
: active, and it start
:     using IPsec replay counter from 30, 000. It will be considered as Replay
: Attack and SA has to be destroyed.
: 
I don't think the picture is this bleak.  What needs to be done is to 
increase the sequence number when the failover happens with large enough 
value to avoid the replay errors.  This can be done by calculating the 
expected pps that can be sent by the active node during the sequence 
number sync period in the cluster.  If you sync once a second, then 
calculate the maximum pps you think/know what can be for that period, and 
increse the sequence number with that ammount.  In your example it would 
go as follows:

 active seq: 30500
 standby seq: 30000
 --failover--
 standby seq: 30000 + 1000

The window at the remote peer will move to 31000; no replay errors.

In this example we assume that maximum of 1000 packets can be sent during 
the sync period.

We've had this working for years without noticeable issues.  We sync 
faster than once a second because we support pretty high throughput 
(several gbit/sec), but we also increase it by quite large number to 
support that throughput also during failovers.

:     This applies to both outbound and inbound replay counters.
:
The issue is only with outbound sequence numbers, because as we know, the 
window can move always forward with incoming packets.  So even if the 
incoming window is lagging behind it will always move forward when the 
remote sends new packets to us.

The draft itself says that the sequence numbers are increased in failover:

   o  Active member dies and Stand-by member takes over.  Stand-by
      Member increments its values of Outbound SA Counters for each
      IPsec SA and sends them to the peer.

It just desn't define how much to increment.  If it increments them with 
large enough number there is no need to send them to the peer.  The window 
at the peer will move automatically to the new values when it receives 
ESP/AH packets with the new sequence numbers.

: - Simultaneous failover at both ends.  If failover happens at the same
: > time in both ends, implementations must be able to handle situation where
: > they receive SYNC_SA_COUNTER_INFO request before they receive response to
: > their own request (they may receive the response only after
: > retransmission).
: >
: > I don't think we have any solution for this and any solution is possible
: (?). This basically means that both sides has lost the SA.
: So we have to establish SA from scratch. This draft assumes, that both side
: has the SA and their message windows has mis-matched.
:
If both sides are clusters then naturally both ends have HA, and can 
handle failover gracefully.  Both ends retain the old SAs in the standby 
node, and need to just update the message id window.  The concern I had 
here is that implementation must be able to handle request also from the 
remote end while it is itself also doing the request.

	Pekka

---

From: Tero Kivinen <[email-masked]>
Date: Wed, 8 Sep 2010 14:22:44 +0300

> you agree to support both, which can mean you may receive the IPSEC SA 

That was my main request during the last IETF. I think this protocol
needs to be split to contain separate parts one for IKE SA and one for
IPsec SAs.

In many environments it is quite possible to keep the IKE SA state in
sync between the cluster members, especially if the liveness checks
are used correctly, i.e. only when needed, not every n seconds. I.e if
IKE SA messages are only sent when Child SAs are created, rekeyed, or
deleted, or when IKE SA is rekeyed, then syncing data between cluster
members is anyways something you want to do as you want to move that
new child SA state to other cluster member anyway and syncing the
message ids in that case is very small part of the syncing you do. If
you have 100,000 clients, and each have IPsec rekey interval of one
hour, that means you do 27 IPsec rekeys per second. The rekeys usually
dominate the IKE SA exchanges unless you use per flow Child SAs (I do
not think people will be using such things on this kind of setup). If
you add some liveness checks for completely silent connections, you
end up around 50-100 message IDs you need to sync from one cluster
member to another. Assuming you send IKE SPIs (16 bytes), window
information (32 bits for window, 32 bits for left and right pointer,
and 32 bits for next outgoing id) you send less than 100 bytes per IKE
SA, thus meaning you need to send one 1000 byte sync information
packet every 100 ms (i.e. ten times per second).

That should not be overwhelming the cluster sync channel even when you
are using state of the art 115200 bit/s serial connection....

If you have stupid implementation which will send liveness check
packet for every single IKE SA (regardless whether there has been
traffic or not in the IPsec SAs associated to them) every 10 seconds
then you will be updating all 100,000 IKE SAs every 10 seconds. This
means 10,000 updated IKE SAs per second, thus meaning 1 MB/s just for
those sync messages. This will overwhelm your serial connection, but
using trusted old 10 Mbit/s ethernet will be enough...

(as you can see I am mostly assuming you do not need the IKE SA
message id syncing at all, as syncing the peers for IKE SA message is
completely feasible).

Syncing the ESP sequence numbers for per packet is not feasible, thus
there it might be useful to have some protocol support, especially if
it is very important to protect against reply attacks. For outgoing
packets you can simply jump over enough packets, but for incoming
packets you cannot, and this means you might be taking in packets
which you have already processed before the crash, and allow those go
through. If protection against that is something you need in your
environment, then having protocol for syncing up the ESP sequence
numbers helps. Other option is simply to rekey the IPsec SA
immediately and delete old IPsec SA, and that will reset the sequence
numbers to "good" values. To protect against replayed old packets, you
need to throw away all incoming ESP packets, altought you can still
send packets out using "large enough" sequence numbers.
-- 
[email-masked]

---

From: Tero Kivinen <[email-masked]>
Date: Wed, 8 Sep 2010 15:09:29 +0300

Raj Singh writes:
>      Now, say failover happened at 30, 500. So, standby member
> becomes active, and it start using IPsec replay counter from 30,
> 000. It will be considered as Replay Attack and SA has to be
> destroyed.

If you have replayed incoming packets, you do consider that replay
attack, and you IGNORE those packets. You DO NOT destroy the SA
because of this.
-- 
[email-masked]

---

From: Tero Kivinen <[email-masked]>
Date: Wed, 8 Sep 2010 15:12:35 +0300

> The window at the remote peer will move to 31000; no replay errors.

That is true for outgoing packets. That is not possible for the
incoming packets, as attacker can replay packets sent to you just
before it caused you to crash, thus causing those replayed packets to
get through. 

> remote sends new packets to us.

But before there is any packets from the real peer having bigger
sequence numbers, the attacker can send old replayed packets from time
before crash, and cause those replayed packets to be accepted.

In most cases this does not matter, but in some environments it might
be very important that such replayed messages are not forwarded.
-- 
[email-masked]

---

From: Pekka Riikonen <[email-masked]>
Date: Wed, 8 Sep 2010 14:30:40 +0200 (CEST)

: Pekka Riikonen writes:
: > The window at the remote peer will move to 31000; no replay errors.
: 
: That is true for outgoing packets. That is not possible for the
: incoming packets, as attacker can replay packets sent to you just
: before it caused you to crash, thus causing those replayed packets to
: get through. 
: 
: > The issue is only with outbound sequence numbers, because as we know, the 
: > window can move always forward with incoming packets.  So even if the 
: > incoming window is lagging behind it will always move forward when the 
: > remote sends new packets to us.
: 
: But before there is any packets from the real peer having bigger
: sequence numbers, the attacker can send old replayed packets from time
: before crash, and cause those replayed packets to be accepted.
: 
I agree there is a small window for attacker to be able to replay old 
packets, but this would mean the attacker has to record old packets and 
has to know exactly when the failover happens (of course if attacker can 
induce the failover, this is serious).

In principal I'm not against having the IPSEC SA sync also in this 
protocol, but if it is left in, I think we have to have a way to negotiate 
which of the sync methods peer wants to use.  Either message id sync, or 
ipsec SA sequence number sync or both.

	Pekka

---

From: Pekka Riikonen <[email-masked]>
Date: Wed, 8 Sep 2010 17:16:11 +0200 (CEST)

On Wed, 8 Sep 2010, Tero Kivinen wrote:
: In many environments it is quite possible to keep the IKE SA state in
: sync between the cluster members, especially if the liveness checks
: are used correctly, i.e. only when needed, not every n seconds. I.e if
: IKE SA messages are only sent when Child SAs are created, rekeyed, or
: deleted, or when IKE SA is rekeyed, then syncing data between cluster
: members is anyways something you want to do as you want to move that
: new child SA state to other cluster member anyway and syncing the
: message ids in that case is very small part of the syncing you do. If
: you have 100,000 clients, and each have IPsec rekey interval of one
: 
: (as you can see I am mostly assuming you do not need the IKE SA
: message id syncing at all, as syncing the peers for IKE SA message is
: completely feasible).
: 
I think it is essential that this protocol provides way to rectify this 
situation, since IKEv2 doesn't allow moving the window forward freely, 
this protocol must provide mechanism for it.  This issue is much more 
serious than the possible replay attack with IPSEC sequence numbers; the 
message id _must_ be correct or the IKE SA is broken.

I think we should leave both sync types in the protocol and provide way to 
negotiate support for either or both.  I think this will make everyone 
happy and makes IKEv2 work in clusters (almost) as well as IKEv1.

	Pekka