Subject: Re: [IPsec] Motivation for ESP-null marking

From: "Grewal, Ken" <[email-masked]>
Date: Tue, 2 Sep 2008 15:19:20 -0600

Hi Tero,
Apologies for the delayed response, but looks like this slipped under the Radar.

A large part of your message is promoting a heuristics based approach, which was also suggested in hallway conversations in Dublin. In order to evaluate this further, including performance / cost / extensibility, I would encourage someone to submit a specification on the heuristics algorithms to employ and perhaps also the associated probability of false positives (where the same conclusion may be reached by running these heuristics on encrypted data). This way it can be evaluated on merit against the current proposal.

Other comments inline...

Thanks,
- Ken

>-----Original Message-----
>From: Tero Kivinen [mailto:[email-masked]]
>Sent: Tuesday, August 12, 2008 4:53 AM
>To: Grewal, Ken
>Cc: Yaron Sheffer; [email-masked]
>Subject: Re: [IPsec] Motivation for ESP-null marking
>
>Grewal, Ken writes:
>> >Let us ignore the details of the solution proposals for a while (1
>> >protocol number vs. 2, ESP wrapper vs. keeping the existing ESP
>> header),
>> >and consider whether there is a need to have any solution at all for
>> >this problem:
>> [Ken] I am surprised that there are questions being raised about this
>> goal, given that its presence in the charter indicates that it's a
>> worthy one. Are there any strong arguments for reversing that consensus?
>
>Standard track mechanism does not need to be protocol change, it can
>be also documentation of how to do the thing with heuristic checks.
>
>I think there is no point of doing this on the protocol level as to
>deploy it requires changes to ALL IPsec endpoints.

[Ken] It will only require changes to nodes that choose to utilize this functionality, which is probably the same statement for any other work item in IETF today. If it is not needed (e.g. remote access / VPN), then those nodes can operate as they do today.

>If we document and
>implement heuristic checks those can be done directly on the
>middleboxes, thus requiring only upgrading those small amount of
>boxes, not all the network endnodes. This means that the heuristic
>methods are deployable within one year from now or even faster
>(especially as they do not require interoperability testing or actual
>protocol specifications, thus vendors can start implementing them
>NOW).
>
>If we now define extension for IKEv2 to allow this (and I hope nobody
>is even going to suggest implementing anything to the now already
>obsoleted IKEv1 protocol), it will most likely take about year to
>finish the protocol document, and then vendors will start implementing
>it thus first implementations would most likely be end of 2009 or
>beginning of 2010, and then you need to wait for couple of more years
>before most of the operating systems support that new extension (i.e.
>around 2012-2014). After most of the operating systems start
>supporting it, then you still need to wait 3-5 years to make sure all
>of the old devices in the network has been updated to support that
>latest operating system, before you can make sure that all network
>endnodes support this. So in best you could be able to use it in 2012
>or so and in reality you would have to wait until 2015-2018...
>
>In the mean time you most likely still want to have solution that will
>work NOW, not in 10 years time, thus middle box vendors will implement
>the heuristic methods anyways regardless whether we define protocol
>for it or not.

[Ken] This argument is applicable to any charted work item and the deployment timelines will be dependent on many factors, including the perceived usefulness of the feature, as well as alternatives available. A heuristics based solution may be less desirable, as anytime a new protocol/extension is encapsulated in IPsec, the heuristics need to be updated to comprehend it.
Having said that, perhaps it is possible to devise a solution that will work for all protocols/payloads of today and tomorrow?  If so, we'd very much like to see it fully specified in a draft submission.  That will make it much easier to evaluate the relative feasibility and practicality of the heuristic approach, compared to our proposal.

>
>This protocol change is again something that will disrupt deployment
>of IPsec, i.e. there will always be implementations which do not
>support it, and which then cannot be used in some environments. It can
>also cause delays in deployment, when enterprices decide that they
>wait until they have this protocol ready for all platforms before even
>starting to use IPsec.

[Ken] Again, please see above. This is applicable to any change / new feature, so not unique to ESP extensions.

>
>I would much more like to see IPsec to spread out as fast as possible,
>and not waste time for something that can be done in the middle boxes
>quite easily with just heuristic checks.
>
>> >    * What is the security/threat environment that we are dealing with?
>> >      In other words, what is the security policy for which we want an
>> >      efficient enforcement?
>> [Ken] This solution is targeting an environment where intermediary
>> devices would like to inspect the packet on route and optionally make
>> access control decisions based on local policy and the results of the
>> packet inspection. Some examples of policies are: Dropping packets with
>> certain characteristics (based on protocol / port or deeper inspection).
>> E.g. Do not allow any encrypted data; Allow encrypted data to only these
>> systems; Do not allow FTP/TFTP traffic between these endpoints, but
>> allow for others. The list goes on...basically any 5-tuple firewall
>> rules + additional enforcement based on payload examination and
>> intrusion prevention.
>> In order to make these decisions, the packets needs to be inspected and
>> the current IPsec ESP spec does not allow you to infer if the packet is
>> encrypted or in the clear (ESP-NULL).
>
>I do not see any reason why both ESP-NULL and encrypted ESP shuld be
>allowed between 2 end nodes, i.e. I see only reason where there is
>either only ESP-NULL allowed or only any type of ESP allowed (i.e.
>encrypted or not, but not inspected at all).

[Ken] This is a matter of policy and we should not offer a pure black-or-white choice. Why don't we just remove NAT and use AH in these environments? Why did we propose a NAT-T solution at all? We cannot mandate how an organization will structure the network (hence the NAT-T solution) and similarly we cannot predict or dictate what protocols/encapsulations/algorithms it should run over that network.

>
>If both ESP-NULL and encrypted ESP is allowed between pair of end
>nodes, then anybody who wants to bybass middlebox checks will simply
>use encrypted ESP. If someone disagrees with this, I would like him to
>explain me why this is needed, and what kind of policy is used for
>that.

[Ken] This is negotiated via policy, which is under the admin's control. E.g. If the policy states that my company CEO (based on his credentials used during IKE) is able to connect to a particular server using encrypted ESP, whereas Joe Bloggs must use ESP-NULL to allow traffic scanning, why could this not be allowed?

>
>Using that premisis, that means that for every single ESP packet
>coming in to the device, we know immediately after matching the
>IP-addresses in the header to the policy whether ESP-NULL is mandated,
>or not. If it is not mandated, then packet is passed forward, as there
>is no point of inspecting encrypted packets.
>
>If ESP-NULL is mandated, then you simply assume packet is ESP-NULL and
>inspect the packet, and if it looks like encrypted ESP (or garbage)
>then you do the normal thing firewalls do, i.e drop it.
>
>> This has been a deployment barrier to using IPsec in Enterprise
>> environments as IT departments would like to employ E2E security, but
>> not at the cost of circumventing existing network monitoring /
>> diagnostics tools. This charter item is looking at removing this barrier
>> and allowing IPsec to be employed in these environments.
>
>It will be really, really expensive and time consuming to update all
>end nodes in the network to allow them to use this new protocol
>extension. It is much faster and cheaper to update those network
>monitoring / diagnostic tools with ones that support heuristic checks.
>
>Also if IT department has full control of the end node
>configurations (which they must have if they assume they can update
>all end node implementations at will) they can simply mandate them to
>use one specific ESP-NULL algorithim, i.e. simply say that no other
>ESP-NULL algoritm is allowed except AUTH_HMAC_SHA1_96, and that will
>remove all heuristic checks needed. This kind of solution is
>implementatable in extremely fast timeframe.

[Ken] This may be viable in small organizations, but is not practical in a large organization, unless a 'forklift upgrade' is used. Also, we need to consider legacy devices (e.g. secure printing using an older algorithm - Do I really want to upgrade all my printers because I choose to use GMAC for other secure comms and my printer is using HMAC-SHA-1-96?). This is just a simple example for illustration purposes.
The remainder of your comments below pertains to heuristics, so please see my comment at the end.

>
>> >    * Are there alternative ways to detect unencrypted traffic in the
>> >      existing ESP, e.g. heuristics? Are they easy to implement in
>> >      software or in hardware? How significant is the problem of false
>> >      positives?
>> [Ken] Heuristics will never be free of false positives, all we can do is
>> play the probability game and reduce false positives to a very small
>> value.
>
>Depends what you mean with false positives.
>
>If you mean that encrypted ESP packet will pass the ESP-NULL
>heuristics and pass the firewall, then the probability of that will go
>very quickly way below the 96 bits offered by the AUTH_HMAC_SHA1_96 if
>you do any kind of deep inspection. Also as there should not be any
>encrypted ESP packets between the nodes who by policy should use
>ESP-NULL, if they happen to negotiate encrypted ESP for some reason,
>they might get 15 packets through for every million packets they try
>to send, even when if you only check TCP checksum (16 bits). If yo do
>TCP flow checking (source and destination ports, sequence number,
>acknowledgement number) you do check close to the 96 bits worth of
>data already. For that to match from the random encrypted packet to
>some ESP-NULL flow is about the same probability that the randomly
>generated packet will pass your AUTH_HMAC_SHA1_96 check.
>
>> False positives have a high cost as they can lead to incorrectly
>> applying policies (e.g. because we falsely determine that an ESP
>> encrypted packet is ESP-NULL) and can cause almost
>> impossible-to-diagnose network glitches (e.g. which packet(s) out of the
>> millions being processed actually generated the false positives).
>
>No. For the end nodes point of view the encrypted ESP connection
>between the nodes is broken regardless whether there is one packet
>every 2^16 or 2^96 going through it. They will see that as broken link
>and user who misconfigured boths ends (note, that it is not enough for
>one user to misconfigure his node, but both ends must be misconfigured
>to allow enctypted ESP) to allow encrypted ESP will notice this and
>fix his configuration.
>
>For attacker doing the same thing, he can continue using encrypted
>packets but as 99.99999999999999999999999999% of his packets will get
>dropped by simple tcp flow checking his attacks will be detected very
>quickly.
>
>Note, that there cannot be valid encrypted ESP packet dropped because
>of these heuristic checks, as encrypted ESP packets are not inspected,
>but they are passed by (when allowed by policy).
>
>> Heuristics can also become very complex with numerous rules to cater for
>> different protocols / payloads (including catering for the protocols of
>> tomorrow). As for employing heuristics in SW Vs. HW, this is really
>> implementation dependent. Having said that, in the multi-core
>> environment of today, numerous vendors employ IO acceleration techniques
>> in HW for fast data movement between the HW (e.g. NIC) and SW
>> components. These techniques are needed to ensure that data is moved
>> directly to the correct CPU/NPU core, which is the consumer of that
>> data. These decisions are typically based on stream identifiers (e.g.
>> 5-tuples in the packet) to ensure full load-balancing capabilities
>> between the multiple cores. In order to make these decisions, the HW
>> needs to extract this data and hence any heuristics would need to run in
>> the HW. Implementing heuristics in HW is very costly, compared with our
>> proposed approach of an ESP wrapper.
>
>No. None of those heuristics needs to really run on the HW. Only the
>SPI checking needs to run on the HW. I.e. the hardware must understand
>that it is matching six tuple if IP header protocol is ESP, i.e.
>source ip/destination ip/spi/protocol number inside ESP-NULL/source
>port inside ESP-NULL/destination port inside ESP-NULL. Immediately
>when it sees the SPI it will know where the protocol number and actual
>header information is. If the SPI is uknown, then HW throws the packet
>to slowpath, which will do the heuristics, calculate the offsets for
>the actual protocol information and for the protocol number, and store
>that information along with the 6-tuple to the packet matching
>hardware and resume the packet.
>
>In almost all the hardware I have seen (our own SafeXcel-5160, Octeon)
>the flow creation is always done on the softpath anyways. For Intel
>Tolapai chips can do it both ways (i.e. either from the microcode
>controlled fastpath, or from the softpath).
>
>Doing flow creation on the fastpath is so complicated, that I expect
>all chips doing so are running some kind of microcode inside the chip
>doing the actual flow creation, thus chip vendors can most likely add
>support for IPsec SPI flow creation there in firmware updates when it
>is widely requested.
>
>Note, that if we change the ESP any way (i.e. define new protocol
>number, or add new bytes or whatever), all the existing hardware will
>need at minimum firmare updates anyways. In some cases like in Octeon
>it would be just writing new code, in some cases like in SafeXcel-5160
>or Tolapai chips it would require the chip vendor to update the
>microcode.
>
>So from hardware point of view it does not really matter what we do
>(i.e. heuristics or ESP-NULL marker) the firmware needs to be updated.
>
>In some chip architectures firmware require smaller updates for the
>heuristics method, as heuristics can be done on the softpath, as the
>full ESP-NULL marker processing needs to be done on the firmware (none
>of that processing can be pushed to the showpath).

[Ken] The remainder of your comments above relate to Heuristics and where these are implemented. This could be in HW or firmware or SW and not all implementations are equally amenable to upgrade options or tolerant of performance requirements. Also, the practically of heuristics based approaches at high speed line rates (10Gbps today, 40/100Gbps tomorrow) can be costly and dependent on the hardware architecture, whereas a new ESP encapsulation would work fine regardless. Furthermore, any changes to any
ESP encapsulated protocols will require changes / additions to the heuristics approach, making this approach a continuous, moving target.
A heuristics based approach was mentioned as an alternative during some hallway conversations in Dublin, so, perhaps it would nice to have a submission on the details. This way, the heuristics approach can be evaluated from the point of view of different hardware implementations and architectures.
_______________________________________________
IPsec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipsec

---

From: Tero Kivinen <[email-masked]>
Date: Fri, 5 Sep 2008 15:12:00 +0300

Grewal, Ken writes:
> [Ken] It will only require changes to nodes that choose to utilize
> this functionality, which is probably the same statement for any
> other work item in IETF today. If it is not needed (e.g. remote
> access / VPN), then those nodes can operate as they do today. 

If we are talking about the enterprice checking traffic in their
internal network, that most likely means that everything using ipsec
will needs to be updated, i.e. both ends of the IPsec connections.
Heuristics only require updating the middleboxes. This is bit similar
than IPv6 vs NAT. IPv6 require updating of all end nodes (and
infrastructure), as NAT only required middle boxes to be added, and no
changes at all to the end nodes. As you can see in the deployment
timeline of IPv6 vs NAT, you can see how much faster we can update
middle boxes... 

> [Ken] This argument is applicable to any charted work item and the
> deployment timelines will be dependent on many factors, including
> the perceived usefulness of the feature, as well as alternatives
> available. A heuristics based solution may be less desirable, as
> anytime a new protocol/extension is encapsulated in IPsec, the
> heuristics need to be updated to comprehend it.

Anytime a new protocol goes through any box doing deep inspection, the
box needs to be updated anyways, as otherwise the new protocol will
not get passed through that box (deep inspection boxes quite commonly
drop everything they will not understand). On the other hand
heuristics only needs to be updated if no known protocols will go
through the deep inspection box at all. I.e. if there is NO TCP or UDP
traffic going inside the ESP-NULL flow at all, then the deep
inspection box needs to understand the new protocol, so it can run
heuristics on that too.

In normal case there is still TCP and UDP traffic going, so it can
simply run the heuristics on those packets and find out the parameters
of the ESP-NULL flow from there. After that they are cached to the
box, and no heuristics is needed until the SA is rekeyed. 

> Having said that, perhaps it is possible to devise a solution that
> will work for all protocols/payloads of today and tomorrow?  If so,
> we'd very much like to see it fully specified in a draft submission.
> That will make it much easier to evaluate the relative feasibility
> and practicality of the heuristic approach, compared to our
> proposal.

Even when you are only checking the ESP padding, that already gives
you quite good probability for running the heuristics with even a
single packet with completely unknown protocol inside the ESP-NULL
packet. The worst case is when there is no padding, thus there is only
Pad-length field having value of 0, so every 1/256 packets goes
through this check. If there is for example 2 bytes of padding (it
needs to be padded to 4-byte boundary), then there is 2 bytes of
padding and 1 padding length byte, which all needs to have specific
values (i.e. 01 02 02). That makes it so that 1 packet out of 16777216
will pass that first check.

I am writing the pseudocode for the heuristics, but it might take
some more time before I get that one ready. 

> [Ken] This is a matter of policy and we should not offer a pure
> black-or-white choice. Why don't we just remove NAT and use AH in
> these environments? Why did we propose a NAT-T solution at all? We
> cannot mandate how an organization will structure the network (hence
> the NAT-T solution) and similarly we cannot predict or dictate what
> protocols/encapsulations/algorithms it should run over that network.

But if the idea is that the deep inspection boxes check the packets
going through them, then they cannot inspect those packets which are
encrypted, and if specific end node pairs are allowed to have both
encrypted and un-encrypted packets, why would the "attacker" use the
un-encrypted packets that will be inspected. He would of course use
the encrypted packets so they will bypass the inspection if he is
trying to do something that is not allowed.

> [Ken] This is negotiated via policy, which is under the admin's
> control. E.g. If the policy states that my company CEO (based on his
> credentials used during IKE) is able to connect to a particular
> server using encrypted ESP, whereas Joe Bloggs must use ESP-NULL to
> allow traffic scanning, why could this not be allowed?

So in this setup you are not doing security checking anymore in the
middlebox, but you are trusting your end SGW to enforce security
policy. If you are doing that, you might as well trust for the end SGW
to do full security policy checking.

Heuristics would work in that kind of situations too, but then the
default what happens when the heuristics fail to check that packet is
ESP-NULL is different (i.e. pass, instead of drop). Of course there is
no way to check that the Joe Bloggs is not using ESP encrypted traffic
in this setup, as you allow encrypteed ESP to go through. The SGW on
the other end might have policy which only allows ESP null for Joe
Bloggs, but if it is misconfigured for some reason, then Joe might be
able to bypass security checks. If the middlebox was put there to
enforce the security checks, then it cannot do that anymore.

> [Ken] This may be viable in small organizations, but is not
> practical in a large organization, unless a 'forklift upgrade' is
> used. Also, we need to consider legacy devices (e.g. secure printing
> using an older algorithm - Do I really want to upgrade all my
> printers because I choose to use GMAC for other secure comms and my
> printer is using HMAC-SHA-1-96?). This is just a simple example for
> illustration purposes. 

If you decide that GMAC is the only one offering enough security for
you, then you do want to upgrade your printers too. If you consider
HMAC-SHA-1-96 to be enough then you can use that for all of your
traffic. If you want to allow your printers to use the new ESP-NULL
protocol then you need to upgrade them anyways. 

> [Ken] The remainder of your comments above relate to Heuristics and
> where these are implemented. This could be in HW or firmware or SW
> and not all implementations are equally amenable to upgrade options
> or tolerant of performance requirements. Also, the practically of
> heuristics based approaches at high speed line rates (10Gbps today,
> 40/100Gbps tomorrow) can be costly and dependent on the hardware
> architecture, whereas a new ESP encapsulation would work fine
> regardless.

I do not really see any real performance differences between the new
ESP encapsulation vs heuristics in the hardware, especially anything
that does any kind of deep inspection. For boxes that do deep
inspection there is already flow tables for TCP flows and such, so
adding SPI flows for mappng SPI numbers to parameters is quite fast
thing to do. All of the complicated heuristics are usually run on the
slowpath anyways.

> Furthermore, any changes to any ESP encapsulated
> protocols will require changes / additions to the heuristics
> approach, making this approach a continuous, moving target. 

We can very easily make the heuristics extremely effective and fast by
saying that IPsec implementations should but extra 4 bytes of padding
to each ESP-NULL packet. This offers very fast and very easy way to
check whether packet is ESP-NULL or ESP encrypted packet without doing
actual protocol change. But deep inspection is always moving target
regardless what we dowith the ESP, thus the middle boxes needs to be
updated every time any protocols or additions are going to be made.
-- 
[email-masked]
_______________________________________________
IPsec mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ipsec