Operational Issues with Processing of the Hop-by-Hop Options Header

Abstract

This document describes the processing of the Hop-by-Hop Options Header (HBH) in current routers in the aspects of standards specification, common implementations, and default operations. This document outlines reasons why the Hop-by-Hop Options Header is rarely utilized in current networks. In addition, this document describes how HBH Options Header could be used as a powerful mechanism allowing deployment and operations of new services requiring a more optimized way to leverage network resources of an infrastructure. The purpose of this draft is to document reasons why HBH Options Header is rarely used within networks. It motivates the benefits and requirements needed to enable wider use of HBH Options.¶

1. Introduction

When IPv6 was first implemented on high-speed routers, Hop-by-Hop Options header (HBH) options were not yet well-understood and ASICs were not as capable as they are today. Due to these historical reasons, as well as limited IPv6 deployments and few service requirements, the early common IPv6 implementation was that the node sends the IPv6 packets with the HBH Options Header to the control plane of the node [RFC9098]. The option type of each option carried within the HBH Options Header is not even examined before the packet is sent to the control plane [RFC7045]. Such processing behavior is often the default configuration or, even worse, is the only behavior of the IPv6 implementation of the node.¶

Sending the HBH Options Header to the control plane for processing could result in Denial of Service (DoS) attack on the router control plane [RFC9288] and inconsistent packet drops. This is due to rate-limiting on the interface between the router control plane and forwarding plane, which impacts packet forwarding. This has in turn led to configuration of measures (such as infrastructure Access Control Lists (ACLs)) to mitigate this DoS vector.¶

This actually introduced a circular problem:¶

-> Implementations that send packets to the control plane for processing are at risk for Denial of Service attack.¶

-> To mitigate the risk of DoS attack, many network operators configured their nodes to discard all packets containing HBH Options Header.¶

-> Because of network operators' configurations that discard packets containing HBH Options Header, protocol designers stopped defining new HBH Options.¶

-> Because protocol designers stopped defining new HBH Options, the community was not motivated to change the conventional implementation that caused use of a HBH Option Header to become a DoS vector.¶

Driven by the wide deployments of IPv6 and ever-emerging new services in recent years, the HBH Options Header is a valuable container for carrying the information to facilitate these new services.¶

The purpose of this work is to:¶

• Break the endless cycle that resulted in HBH being a DOS vector.¶

• Enable the HBH options header to be utilized in a safe and secure way without impacting the control plane.¶

• Ease the deployments of the new network services that utilize the HBH Option Header in a multi-vendor scenario that can now be deployed without operational impact.¶

2. Terminology

The terms Forwarding Plane and Control Plane used in this draft refer to the terminologies as defined in [I-D.ietf-6man-hbh-processing].¶

3. New Services

As IPv6 is being rapidly and widely deployed worldwide, more and more applications and network services are migrating to or directly adopting IPv6. New services that require hop-by-hop processing are emerging and the HBH Options header is going to be utilized by the new services in various scenarios.¶

In-situ OAM (IOAM) with IPv6 encapsulation [RFC9486] is used to enhance diagnostics of IPv6 networks and complements other mechanisms, such as the IPv6 Performance and Diagnostic Metrics Destination Option described in [RFC8250]. The IOAM data fields are encapsulated in "option data" fields of the Hop-by-Hop Options header.¶

The Alternate Marking Option is defined as a new IPv6 Hop-by-Hop option to encode an alternate marking technique [RFC9343].¶

The Minimum Path MTU Hop-by-Hop Option is defined in [RFC9268] to record the minimum Path MTU along the forward path between a source host to a destination host. This Hop-by-Hop option is intended to be used in environments like Data Centers and on paths between Data Centers as well as other environments including the general Internet. It provides a useful tool to better take advantage of paths capable of supporting a large Path MTU.¶

A Virtual Transport Network (VTN) Option is used to carry the VTN related information, which could used to identify the set of network resources allocated to a VTN and the rules for packet processing. [I-D.ietf-6man-enhanced-vpn-vtn-id].¶

As more services start utilizing the HBH Options header, more packets containing HBH Options are going to be injected into the networks. A currently common configuration in deployed networks is for routers to send all the packets of the new services to the control plane of the nodes, with the possible consequence of resulting in a DoS vulnerability on the control plane. The packets will be dropped and the normal IP forwarding is severely impacted. The deployment of new network services involving multi-vendor interoperability is greatly impeded.¶

4. Modern Router Architecture

The architecture of modern routers deployed by Internet operators maintains a strict separation of the router control plane and its forwarding plane [RFC6192], as shown in Figure 1. Either the control plane or the forwarding plane is composed of both software and hardware, but each plane is responsible for different functions. In this document, we focus on only hardware routers following the architecture as shown in Figure 1 which are typical of core routers or datacenter routers, as opposed to software-based routers such as home routers or hotspot routers in a smartphone.¶

                             +----------------+
                             | Router Control |
                             |     Plane      |
                             +------+-+-------+
                                    | |
                                 Interface Z
                                    | |
                             +------+-+-------+
                             |   Forwarding   |
               Interface X ==[     Plane      ]== Interface Y
                             +----------------+

Figure 1. IP router architecture splitting the control and forwarding planes

Many of the packet-processing devices employed in current router designs are fixed-function ASICs that handle common functions, however there is a trend towards programmable device datapaths such as P4 [P4]. While these devices can be very efficient for the set of functions they are designed for, they can be very inflexible. There is a tradeoff of price, performance and flexibility when vendors make a choice to use a fixed function ASIC as opposed to Network Processing Unit (NPU). Due to the inflexibility of the fixed function ASIC, tasks that require additional processing such as IPv6 HBH header processing must be punted to the control plane.¶

This problem is still a challenge today and is the reason why operators drop or ignore HBH options to protect against control plane DOS attack. Semiconductor chip technology has advanced significantly in the last decade. An industry trend is for intelligent multi-core CPU hardware using modern NPUs for forwarding packets at line rate while still being able to perform other complex tasks such as HBH forwarding options processing without having to punt to the control plane increasing the viability of ubiquitous HBH use cases.¶

Parsing buffers with a limited size are commonly used in router implementations. The parsing buffer contains the headers of a packet that can be processed in the fast path and typically have sizes like 128, 256, 284 bytes. A source node is permitted to encode hundreds of options in 2,048 bytes [I-D.ietf-6man-eh-limits]. With today's technology it would be cost prohibitive to be able to process hundreds of options with either NPU or proprietary fixed function ASIC [I-D.ietf-6man-eh-limits].¶

IPv6 Extended Header limitations that need to be addressed to make HBH processing more efficient and viable in the forwarding plane.¶

[I-D.ietf-6man-eh-limits] defines the IPv6 router requirements and how to protect a router from excessive header chain and excessive header options with various limitations that can be defined on a router. Per [RFC8200] HBH options must be processed serially. However, an implementation of options processing can be done with more parallelism in serial processing by grouping similar options to be processed in parallel.¶

Each Option is encoded in a TLV and so processing of a long list of TLVs is expensive. Zero data length encoded options TLVs are a valid option with a minimum length of two bytes. A DOS vector could be easily generated by encoding around 700 HBH options (Zero data length) in a standard 1500 MTU packet. Limiting the number of options that are processed could avoid an arbitrary large amount of processing for a single EH.¶

5. Specification of RFC 8200

As specified in [RFC8200], the HBH Options header is used to carry optional information that will be examined and processed by every node along a packet's delivery path, and it is identified by a Next Header value of zero in the IPv6 header.¶

As per [RFC8200], it is now expected that nodes along a packet's delivery path only examine and process the Hop-by-Hop Options header if explicitly configured to do so. This means that the HBH processing behavior in a node depends on its configuration.¶

However, in the current [RFC8200], there is no explicit specification of the possible configurations leading to likely uncertain processing behaviors, that is, the nodes may be configured to ignore the HBH Options Header, drop packets containing a HBH Options Header, or assign packets containing a HBH Options Header to the control plane [RFC8200]. This specification is updated in [I-D.ietf-6man-hbh-processing].¶

6. Common Implementations

Current routers deployed by operators usually process an IPv6 packet that has a Next Header field set to 0 by directly forwarding it to the control plane of the node. With such implementations, the value of the Next Header field in the IPv6 header is the only trigger for the default processing behavior. The option type of each option carried within the HBH Options Header is not even examined before the packet is sent to the control plane.¶

Very often, such processing behavior is the default configuration on the node, which is embedded in the implementation and cannot be changed or reconfigured.¶

Another critical component of IPv6 HBH processing, in some cases overlooked, is that an operator core network can be designed to use the global Internet routing table for internet traffic and in other cases use an overlay MPLS VPN to carry Internet traffic.¶

In the global Internet routing table scenario where only an underlay global routing table exists, and no VPN overlay carrying customer Internet traffic, the IPv6 HBH options could be used as a DOS attack vector for both the operator nodes, adjacent inter-AS peer nodes, as well as customer nodes along a path.¶

In a case where the Internet routing table is carried in an MPLS VPN overlay payload, the HBH options header does not impact the operator underlay framework and only impacts the VPN overlay payload. Thus the operator underlay top most label global table routing FEC LSP instantiation is not impacted as the operator underlay is within the operators closed domain.¶

However, the HBH options DOS attack vector in the VPN overlay can still impact the customer end nodes as well as other adjacent inter-AS operators that only use the underlay global Internet routing table. In an operator domain where an MPLS VPN overlay is utilized to carry internet traffic, the operator has full control of the underlay and IPv6 Extension Header chain length as well as the number of HBH options encoded [I-D.ietf-6man-eh-limits].¶

In the global routing table scenario for Internet traffic there is no way to control the IPv6 Extended header chain length as well as the number of HBH options encoded.¶

When IPv6 was first implemented on high-speed routers, HBH options were not yet well-understood and ASICs were not as capable as they are today. So, early IPv6 implementations dispatched all packets that contain HBH options to their control plane. Such implementation introduces a risk of a DoS attack on the control plane of the node, and a large flow of IPv6 packets could congest the control plane, causing other critical functions (including routing and network management) that are executed on the control plane to fail. Rate limiting mechanisms will cause inconsistent packet drops and impact the normal IP forwarding of packets.¶

7. Typical Processing

To mitigate this DoS vulnerability, many operators have deployed Access Control Lists (ACLs) that discard all packets containing HBH Options [RFC9288].¶

[RFC6564] shows the Reports from the field indicating that some IP routers deployed within the global Internet are configured either to ignore or to drop packets having a hop-by-hop header. As stated in [RFC7872], many network operators perceive HBH Options to be a breach of the separation between the forwarding and control planes. Therefore, several network operators configured their nodes so as to discard all packets containing the HBH Options Extension Header, while others configured nodes to forward the packet but to ignore the HBH Options. [RFC7045] also states that hop-by-hop options are not handled by many high-speed routers or are processed only on a control plane. [I-D.vyncke-v6ops-james] shows that the HBH options header cannot reliably traverse the global Internet; only small headers with 'skippable' options have some chances.¶

Due to such behaviors observed and described in these specifications, [RFC8200] did not recommend new HBH Options, which limited the usability of HBH Options.¶

Besides service providers' networks, other sectors such as industrial IoT networks are slowly replacing a dozen of semi-proprietary protocols in industrial automation into IP. The proper processing of the HBH options header is also required [I-D.pthubert-detnet-ipv6-hbh].¶

8. Design Principles

Here are some design principles for HBH Options Header and its carrying Options.¶

• The HBH Options Header should not become a possible DDoS Vector. Therefore, the control plane needs to be preserved from unwanted incoming traffic due to HBH header present in the packet.¶

• HBH Options need to be designed in a manner so that they don't reduce the probability of packet delivery, especially in length limit.¶

• HBH processing implementations need to perform well at a reasonable cost without endangering the security of the router.¶

• The Router Alert Option needs not to impact the processing of other HBH Options that should be processed more quickly [I-D.ietf-6man-hbh-processing].¶

• HBH Options can influence how a packet is forwarded. However, with the exception of the Router Alert Option, an HBH Option must not cause control plane state to be created, modified or destroyed on the processing node. As per [RFC6398], protocol developers SHOULD avoid future use of the Router Alert Option.¶

• The limits specified in [I-D.ietf-6man-eh-limits] need to be configurable to manage HBH processing so it doesn't become a bottleneck.¶

• MLDv2 [RFC3810] should not be forbidden.¶

9. Migration Strategies

In order to make the HBH Options Header usable and facilitate the ever-emerging new services to be deployed across multiple vendors' devices, the new specifications related to the processing of HBH Options Header need to be designed to allow a smooth migration from old to new behavior without disruption time.¶

12. Acknowledgements

The authors would like to acknowledge Ron Bonica, Fred Baker, Bob Hinden, Stefano Previdi, and Donald Eastlake, Gorry Fairhurst, Nick Hilliard, Tom Herbert, Haisheng Yu, Nalini Elkins, and Alexandre Petrescu for their valuable review and comments.¶

13. References