Subject: [nvo3] Security and encapsulation protocols

From: Tom Herbert <[email-masked]>
Date: Fri, 21 Mar 2014 09:25:16 -0700

Looking a some of the proposals related to nvo3, I am concerned that
assumptions are being baked in that the data center network will
provide adequate security of a protocol.

>From geneve draft:
"Within a particular security domain, such as a data center operated
by a single provider, the most common and highest performing security
mechanism is isolation of trusted components."

>From nsh draft:
"In many deployments, NSH will be used in a controlled environment,
with trusted devices (e.g. a data center) thus mitigating the risk of
unauthorized header manipulation."

This seem to be punting the security of a protocol to be someone
else's problem. Even in the data center, our security threats are
increasing and obviously the push to allow third party code to run in
that environment probably increases the threats by an order of
magnitude.

Here are some perspectives from a large deployment:

1) Given the value of the data that is being carried in packets, the
cost of a breach is potentially **very** high.
2) Any single device which is completely comprised should still only
have limited access and bad effects.
3) Inevitably, misconfiguration or bad routing will misdirect traffic
and inadvertently bypass network security for some packets.
4) In environments that allow third party VMs we must assume that
every hosting device is potentially untrusted.
5) Corollary to above, every host now must implement a security perimeter.
6) We will never completely trust third party devices for which we
don't own the code or implementation.
7) In a large network bit errors, HW failures, SW bugs are common
occurrences. It's problematic that in VXLAN and nvgre even a single
bit error in the vni could misdirect a packet to the wrong VM (no CRC
or checksum protects vni).
8) We know that threats to our network will only increase overtime, we
must be able to adapt accordingly. This becomes a big problem if we
became completely dependent on network HW for security (cost of
swapping out HW is prohibitive).

So fundamentally, we need end to end security within the protocols--
at least that covers any fields that are sensitive to corruption or
snooping. Mechanisms in the network are still very important, but in
themselves not adequate and should be complementary to security within
the protocol.

Please take this under consideration.

Thanks,
Tom

---

From: Fabio Maino <[email-masked]>
Date: Tue, 25 Mar 2014 11:34:52 -0700

Tom,
I'm not sure security should *always* be deployed end-to-end, as there 
might be use cases where it's perfectly acceptable, or it may even be 
required, to deploy security at the DC edge or ToR (e.g. for better 
troubleshooting, traffic inspection, or even just cost/operational 
reasons). I guess that, ultimately, it depends from the threat model 
you're trying to address.

As long as the architecture allows to combine security with the network 
virtualization function, it can allow different deployment models, 
including the one you describe.

This is actually an argument for proper layering, where a flexible 
architecture could separate the network virtualization layer, from the 
metadata layer (e.g. service function), and from the security layer. 
Different combinations of those layers may allow for different 
deployment models. Also it seems very likely that well known security 
mechanisms (such as IPsec) could be applied to address some of the 
security challenges you describe.

Fabio

On 3/21/14, 9:25 AM, Tom Herbert wrote:
> _______________________________________________
> nvo3 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nvo3

---

From: Tom Herbert <[email-masked]>
Date: Tue, 25 Mar 2014 12:06:53 -0700

On Tue, Mar 25, 2014 at 11:34 AM, Fabio Maino <[email-masked]> wrote:
> Tom,
> I'm not sure security should *always* be deployed end-to-end, as there might
> be use cases where it's perfectly acceptable, or it may even be required, to
> deploy security at the DC edge or ToR (e.g. for better troubleshooting,
> traffic inspection, or even just cost/operational reasons). I guess that,
> ultimately, it depends from the threat model you're trying to address.
>
> virtualization function, it can allow different deployment models, including
> the one you describe.
>
> Different combinations of those layers may allow for different deployment
> models. Also it seems very likely that well known security mechanisms (such
> as IPsec) could be applied to address some of the security challenges you
> describe.
>
Can you provide an example (actual packet formats) of how I might be
able to use SFC with something like VXLAN to implement security of the
vni as I described? I think this is pretty straightforward and very
efficient in GUE, and I think I can extrapolate how to do this in
geneve...

Thanks,
Tom

---

From: Jon Hudson <[email-masked]>
Date: Tue, 25 Mar 2014 16:52:50 -0700

Excellent points to discuss.

Thank you very much for posting!

> On Mar 21, 2014, at 9:25 AM, Tom Herbert <[email-masked]> wrote:
>

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 27 Mar 2014 11:34:40 -0700

Tom,

It sounds like we need some approach to be able to avoid misdelivering 
packets if there is a bit error in the vni field.
I suspect there might be multiple approaches (UDP-lite covering the NVO3 
header, header checksum field in the NVO3 header).

But my high-level question is whether the security requirements will be 
such that you are likely to want to use IPsec. If so adding a checksum 
over the vni wouldn't be needed.

Regards,
    Erik

---

From: Tom Herbert <[email-masked]>
Date: Thu, 27 Mar 2014 12:25:38 -0700

> header, header checksum field in the NVO3 header).
If it's UDP (e.g. VXLAN) enabling UDP checksum would be a good start,
but this is still weaker than the Ethernet CRC which protects VLAN.

> But my high-level question is whether the security requirements will be such
> that you are likely to want to use IPsec. If so adding a checksum over the
> vni wouldn't be needed.
>
IPsec is very desirable and I believe we want to get to encrypting all
data in flight. However there are some practical considerations:

- Lack of viable solutions that could work line rate (40, 100Gpbs).
- We still would want a UDP encapsulation to get hashing in networking
devices to work on inner flow.
- If vni is not in plain text headers we lose the ability to route or
filter in the network based on that (or other meta data).
- We would lose visibility in the network of the precise inner
4-tuple. We get this information now from sflow for diagnosing the
network. A long flow token in the plain text header might be useful
for this, and there may be other attributes of the encapsulated packet
we might with to expose.

Tom

---

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 27 Mar 2014 13:50:49 -0700

> header, header checksum field in the NVO3 header).

I must be missing something...why is the CRC of the received Ethernet frame
not enough?  That does cover the VNI along with the rest of the packet.
 When an NVE receives a packet, if it experiences a CRC error, it should be
discard the frame and not try to decap it.

---

From: Tom Herbert <[email-masked]>
Date: Thu, 27 Mar 2014 13:57:23 -0700

> discard the frame and not try to decap it.

This is a layer 3 protocol. A packet may routed across several hops
some of which might not even be Ethernet.

---

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 27 Mar 2014 14:18:34 -0700

> some of which might not even be Ethernet.

The CRC protects the entire frame till it gets to a router.  Once in the
router, it's the responsibility of the router to maintain packet integrity
and not introduce errors via an internal checksum or some other mechanism.
 A similar issue exists with MPLS labels used for tenant identification.

If we don't trust the router, then we have a problem.  IPv6 lacks a header
checksum and I'm not aware of many routers that will compute the TCP
checksum in order to validate the integrity of the IPv6 destination address
before delivering it to a host.

Anoop

---

From: Tom Herbert <[email-masked]>
Date: Thu, 27 Mar 2014 16:38:20 -0700

> If we don't trust the router, then we have a problem.  IPv6 lacks a header

Right, I don't trust routers :-). Hardware fails, cosmic radiation
zaps bits, software bugs occur, etc. A cursory look at our network
currently shows that TCP checksum errors between internal machines are
very rare but not zero (a couple hundred instances a day throughout
the network on average). However, on a bad day, when hardware or
software is failing, we have seen these numbers go way up. So, to be
robust, we still need to consider the network unreliable and not error
free-- this remains a fundamental operational characteristic of IP.

> before delivering it to a host.
The IP addresses are part of pseudo header (RFC 2460) and are
validated in the TCP checksum, so simple corruption of addresses would
be detected at the receiving host.

I would point out that we don't really trust the TCP checksum either!
For any serious data center application (like those that carry real
user data), we need a much stronger end to end CRC over the data (like
in iSCSI, RDMA) if not actual packet security like in IPsec. Because
of the sensitivity of vni, I tend to think it is critical to get right
so a stronger mechanism is warranted.

Tom

---

From: Anoop Ghanwani <[email-masked]>
Date: Thu, 27 Mar 2014 17:54:56 -0700

> be detected at the receiving host.

It would be (that was why I brought up the TCP checksum), but from a
security standpoint, unless routers validate it before delivering the
packet to the host, that's too late.

> so a stronger mechanism is warranted.

I don't know if I agree or not with respect to protecting just the VNI.
 Off the top of my head, I want to disagree, since MPLS VPNs seem to have
been doing OK.  But maybe things are different with this protocol...

Anoop

---

From: Tom Herbert <[email-masked]>
Date: Fri, 28 Mar 2014 08:19:51 -0700

On Thu, Mar 27, 2014 at 5:54 PM, Anoop Ghanwani <[email-masked]> wrote:
> security standpoint, unless routers validate it before delivering the packet
> to the host, that's too late.
>>
>>
> I don't know if I agree or not with respect to protecting just the VNI.  Off
> the top of my head, I want to disagree, since MPLS VPNs seem to have been
> doing OK.  But maybe things are different with this protocol...
>
I don't know about MPLS VPNs, but the cost of the security cookie in
GUE is 64 bits of header and negligible processing overhead. This
gives us strong protection against unintentional corruption of the vni
and a fair amount protection against vni being spoofed.  I think it's
a pretty good deal.

Tom