Border Gateway Protocol 4 (BGP-4) Send Hold Timer
Job Snijders
Fastly
Amsterdam
Netherlands
job@fastly.com
Ben Cartwright-Cox
Port 179 Ltd
London
United Kingdom
ben@benjojo.co.uk
Yingzhen Qu
Futurewei Technologies
Santa Clara
United States
yingzhen.ietf@gmail.com
Routing
IDR
BGP
TCP
This document defines the SendHoldtimer, along with the SendHoldTimer_Expires event, for the Border Gateway Protocol (BGP) Finite State Machine (FSM). Implementation of the SendHoldTimer helps overcome situations where a BGP connection is not terminated after the local system detects that the remote system is not processing BGP messages. This document specifies that the local system should close the BGP connection and not solely rely on the remote system for connection closure when the SendHoldTimer expires. This document updates RFC4271.

Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Introduction
This document defines the SendHoldtimer, along with the SendHoldTimer_Expires event, for the Border Gateway Protocol (BGP) [RFC4271] Finite State Machine (FSM) defined in section 8.
Failure to terminate a blocked BGP connection can result in network reachability issues, and the subsequent failure to generate and deliver BGP UPDATE messages to another BGP speaker of the local system is detrimental to all participants of the inter-domain routing system. This phenomena is thought to have contributed to IP traffic blackholing events in the global Internet routing system [bgpzombies].
This specification intends to improve this situation by requiring that BGP connections be terminated if the local system has detected that the remote system cannot possibly have processed any BGP messages for the duration of the SendHoldTime. Through standardization of the aforementioned requirement, operators will benefit from consistent behavior across different BGP implementations.
BGP speakers following this specification do not rely exclusively on remote systems closing blocked connections, but will also locally close blocked connections.

Example of a problematic scenario
In implementations lacking the concept of a SendHoldTimer, a malfunctioning or overwhelmed remote speaker may cause data on the BGP socket in the local system to accumulate ad infinitum. This could result in forwarding failure and traffic loss, as the overwhelmed speaker continues to utilize stale routes.
An example fault state: as BGP runs over TCP [RFC9293], it is possible for a BGP speaker in the Established state to encounter a BGP speaker that is advertising a TCP Receive Window (RCV.WND) of size zero. This 0 window prevents the local system from sending KEEPALIVE, UPDATE, or any other critical BGP messages across the network socket to the remote speaker.
Generally BGP implementations have no visibility into lower-layer subsystems such as TCP or the speaker's current Receive Window size, and there is no existing BGP mechanism for such a blocked connection to be recognized. Hence BGP implementations are not able to handle this situation in a consistent fashion.
The major issue arising from a BGP speaker being unable to send a BGP message to a given remote speaker is that as a result that speaker subsequently is operating based on stale routing information. Failure of the BGP speaker to send (and thus the remote speaker to receive) BGP messages on a single BGP session can negatively impact the ability of an entire autonomous system (or even a group of autonomous systems) to converge.

Changes to RFC 4271 - SendHoldTimer
BGP speakers are implemented following a conceptual model "BGP Finite State Machine" (FSM), which is outlined in section 8 of [RFC4271]. This specification adds a BGP timer, SendHoldTimer, and updates the BGP FSM as follows:

Session Attributes
The following optional session attributes for each connection are added to Section 8, before "The optional session attributes support different features of the BGP functionality that have implications for the BGP FSM state transitions":
NEW
14) SendHoldTimer
15) SendHoldTime
The SendHoldTime determines how long a BGP speaker will stay in Established state before the TCP connection is dropped because no BGP messages can be transmitted to its peer. A BGP speaker can configure the value of the SendHoldTime for each peer independently.

Timer Event: SendHoldTimer_Expires
Another timer event is added to Section 8.1.3 of [RFC4271] as following:
NEW
Event 29: SendHoldTimer_Expires
Definition:
An event generated when the SendHoldTimer expires.
Status:
Optional

Changes to the FSM
The following changes are made to section 8.2.2 in [RFC4271].
In "OpenConfirm State", the handling of Event 26 is revised as follows:
OLD
If the local system receives a KEEPALIVE message (KeepAliveMsg (Event 26)), the local system:
- restarts the HoldTimer and
- changes its state to Established.
NEW
If the local system receives a KEEPALIVE message (KeepAliveMsg (Event 26)), the local system:
- restarts the HoldTimer,
- starts the SendHoldTimer if the SendHoldTime is non-zero, and
- changes its state to Established.
The following paragraph is added to section 8.2.2 in "Established State", after the paragraph which ends "unless the negotiated HoldTime value is zero.":
NEW
If the SendHoldTimer_Expires (Event 29) occurs, the local system:
  - (optionally) sends a NOTIFICATION message with the BGP Error Code "Send Hold Timer Expired" if the local system can determine that doing so will not delay the following actions in this paragraph,
  - logs an error message in the local system with the BGP Error Code "Send Hold Timer Expired",
  - releases all BGP resources,
  - sets the ConnectRetryTimer to zero,
  - drops the TCP connection,
  - increments the ConnectRetryCounter by 1,
  - (optionally) performs peer oscillation damping if the DampPeerOscillations attribute is set to TRUE, and
  - changes its state to Idle.
Each time the local system sends a BGP message, it restarts the SendHoldTimer unless the SendHoldTime value is zero or the negotiated HoldTime value is zero, in which cases the SendHoldTimer is stopped.
The SendHoldTimer is stopped following any transition out of the Established state as part of the "release all BGP resources" action.

Changes to BGP Timers
Section 10 of [RFC4271] summarizes BGP Timers. This document adds another optional BGP timer: SendHoldTimer.
NEW
SendHoldTime is an FSM attribute that stores the initial value for the SendHoldTimer. If SendHoldTime is non-zero then it MUST be greater than the value of HoldTime, see Section 5 of [I-D.ietf-idr-bgp-sendholdtimer] for suggested default values.

Send Hold Timer Expired Error Handling
If the local system does not send any BGP messages within the period specified in SendHoldTime, then a NOTIFICATION message with the "Send Hold Timer Expired" Error Code MAY be sent and the BGP connection MUST be closed. Additionally, an error MUST be logged in the local system, indicating the Send Hold Timer Expired Error Code.

Implementation Considerations
Due to the relative rarity of the failure mode that this specification is designed to address, and also the fact that network operators may be unfamiliar with the formal specification of BGP fault detection mechanisms such as HoldTimer, it is likely that a large number of operators are unaware of the necessity of an additional mechanism such as SendHoldtimer.
Accordingly, it is RECOMMENDED that implementations of this specification enable SendHoldtimer by default, without requiring additional configuration of the BGP speaking device.
The default value of SendHoldTime for a BGP connection SHOULD be the greater of:
- 8 minutes; or
- 2 times the negotiated HoldTime
Implementations MAY make the value of SendHoldTime configurable, either globally or on a per-peer basis, within the constraints set out in Section 3.4 above.
The subcode for NOTIFICATION message "Send Hold Timer Expired" is set to 0 and is not used, no additional data is to be appended to the end of a "Send Hold Timer Expired" NOTIFICATION message.

Operational Considerations
When the local system recognizes a remote speaker is not processing any BGP messages for the duration of the SendHoldTime, it is likely that the local system will not be able to inform the remote peer through a NOTIFICATION message as to why the connection is being closed. This documents suggests that an attempt to send a NOTIFICATION message with the "Send Hold Timer Expired" error code is still made, if doing so will not delay closing the BGP connection. Meanwhile an error message is logged into the local system.
Other mechanisms can be used as well, for example BGP speakers SHOULD provide this reason as part of their operational state; e.g. bgpPeerLastError in the BGP MIB.

Security Considerations
This specification does not change BGP's security characteristics. Implementing the BGP SendHoldTimer as specified in this document will enhance network resilience by terminating connections with malfunctioning or overwhelmed remote peers.

IANA Considerations
IANA has registered code 8 for "Send Hold Timer Expired" in the "BGP Error (Notification) Codes" registry in the "Border Gateway Protocol (BGP) Parameters" registry group.

Acknowledgements
The authors would like to thank William McCall, Theo de Raadt, John Heasley, Nick Hilliard, Jeffrey Haas, Tom Petch, Susan Hares, Keyur Patel, Ben Maddison, Claudio Jeker, and John Scudder for their helpful review of this document.

Normative References
[RFC9293]

Transmission Control Protocol (TCP)
Wesley M. Eddy, Ed.
August 2022
RFC 9293
DOI 10.17487/RFC9293
[I-D.ietf-idr-bgp-sendholdtimer]

Border Gateway Protocol 4 (BGP-4) Send Hold Timer
Job Snijders
Fastly
Ben Cartwright-Cox
Port 179 Ltd
Yingzhen Qu
Futurewei Technologies
Internet-Draft draft-ietf-idr-bgp-sendholdtimer

Informative References
[openbgpd]

bgpd send side hold timer
Claudio Jeker
OpenBSD
December 2020
[frr]

bgpd: implement SendHoldTimer
David Lamparter
NetDEF
May 2022
[neo-bgp]

What does bgp.tools support
Ben Cartwright-Cox
Port 179 Ltd
Aug 2022
[BIRD]

BIRD Internet Routing Daemon
Katerina Kubecova
CZ.NIC
Oct 2023
[bgpzombies]

BGP Zombies
Romain Fontugne
IIJ Research Lab
april 2019
[RFC2119]
[RFC8174]
[RFC4271]

Implementation status - RFC EDITOR: REMOVE BEFORE PUBLICATION
This section records the status of known implementations of the protocol defined by this specification at the time of posting of this Internet-Draft, and is based on a proposal described in RFC 7942. The description of implementations in this section is intended to assist the IETF in its decision processes in progressing drafts to RFCs. Please note that the listing of any individual implementation here does not imply endorsement by the IETF. Furthermore, no effort has been spent to verify the information presented here that was supplied by IETF contributors. This is not intended as, and must not be construed to be, a catalog of available implementations or their features. Readers are advised to note that other implementations may exist.
According to RFC 7942, "this will allow reviewers and working groups to assign due consideration to documents that have the benefit of running code, which may serve as evidence of valuable experimentation and feedback that have made the implemented protocols more mature. It is up to the individual working groups to use this information as they see fit".
- OpenBGPD [openbgpd]
- FRRouting [frr]
- neo-bgp (bgp.tools) [neo-bgp]
- BIRD [BIRD]
Patches to recognize error code 8 were merged into OpenBSD's and the-tcpdump-group's tcpdump implementations.