Subject: RE: My BGP Route Update Pacing Draft

From: "Abarbanel, Benjamin" <[email-masked]>
Date: Thu, 18 Jul 2002 16:47:59 -0400

Vrishab:

-----Original Message-----
From: vrishab sikand [mailto:[email-masked]]
Sent: Thursday, July 18, 2002 4:37 PM
To: Abarbanel, Benjamin; [email-masked]'
Subject: Re: My BGP Route Update Pacing Draft

Specific comment: 

Unless the rate of pacing is defined in absolute terms (as opposed to the a
subjectively for e.g 

yellow 4: reduce update message traffic by 40% of your normal rate). It will
be very difficult to verify compliance. 

Also in my experience, the "normal rate" varies drastically from vendor to
vendor. 

 What I meant here, whatever the message/second or route/second transmission
was as an average, reduce it by 40%. So if source router sent 100
messages/second it should now send 60 messages/second. That implies that the
sending router is keeping average statistics of its transmission rate to the
given peer on a message or route per second basis. I dont think this will be
too costly to do. 

General comment: 

 It appears to me that here is an attempt to solve an internal scheduling or
horse power problem with the help of external protocol.   

Yes in way it does, but it could also help scale older routers when they are
placed in a new environment where the loads exceed their previous tuned
expectations. You can think of this as a gracefull way to deal with abnormal
loading conditions without emploding internally.  

This mechanism may put undue burden on the transmitting router, which is in
the best interest of quick convergence in mind is sending prefixes as fast
as possible.   

I am not sure this is the case, it all depends on how people implement the
solution.  

Ben 

  "Abarbanel, Benjamin" <[email-masked]> wrote: 

Hi all:

Our recent discussions on this list and my recent work 
experiences have led me to write this draft and offer it 
to the IETF community. I would appreciate any comments 
anyone has to make.

Thanks in advance,
Ben

Network Working Group Ben Abarbanel
Internet Draft Marconi Communicatons
Expiration Date: December 2002 

BGP Route Update Pacing

draft-abarbanel-bgp-route-update-pacing-00.txt

1. Status of this Memo

This document is an Internet-Draft and is in full conformance with
all provisions of Section 10 of RFC2026 except that the right to
produce derivative works is not granted.

Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as Internet-
Drafts.

Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as ``work in progress.''

The list of current Internet-Drafts can be accessed at
http://www.ietf.org/ietf/1id-abstracts.txt

The list of Internet-Draft Shadow Directories can be accessed at
http://www.ietf.org/shadow.html.

2. Abstract

This document defines a mechanism for controlling or limiting the rate at 
which BGP update messages are sent from one peer to the next when a BGP peer

experiences internal congestion. With the introduction of new dynamic BGP 
protocol capabilities [CAP] message or other none BGP session destructive 
messages, it is necessary to limit the rate at which the BGP update messages

are sent without affecting the entire BGP session and without relying on the

transport (TCP) layer to do so as a normal reaction to data congestion of
the 
TCP session across the network.

3. Specification of Requirements

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", 
"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to
be 
interpreted as described in RFC 2119 [RFC2119].

Internet Draft draft-abarbanel-bgp-route-update-pacing-00.txt [Page 2]

4. Introduction

This document defines a mechanism for controlling or limiting the rate at 
which BGP update messages are sent from one peer to the next when a BGP peer

experiences internal congestion. With the introduction of new dynamic BGP 
protocol capabilities [CAP] message or other none BGP session destructive 
messages, it is necessary to limit the rate at which the BGP update messages

are sent without affecting the entire BGP session and without relying on the

transport (TCP) layer to do so as a normal reaction to data congestion of
the 
TCP session across the network.

When a router enters a state where either its CPU utilization is maximized 
(reaches close to 100%) or its memory is nearly depleted (less than 10% of 
memory left), it cannot handle new or heavy streams of updates from its
peers 
and at times unable to send messages to its peer in a timely manner (within 
30 seconds). As a consequence it cannot keep current with the topological 
state. In some scenarios, a non-congested peer might want to negotiate new 
capabilities with a congested peer. The congested peer is so degraded that 
its TCP session goes into significant flow control off conditions and is 
unable to see the new BGP messages. Peer to peer communication is severely 
hampered and as a result the uncongested peer will take corrective action 
when its hold timer expires and drops the session. The uncongested peer 
computes alternate routing paths that are suboptimal in distance or
attribute 
and thus affect the forwarding decisions of all routers in this network. It 
is possible that the congested peer's routing (control) plane is badly 
degraded but its forwarding plane is at normal working level. The mistake is

made by the uncongested peer since it does not see any Keepalive/Update 
messages before its Hold timer expires and drops the session thereby
dropping 
all its associated routes. After routes are dropped, network instability 
occurs and suboptimal paths are used by the remaining peers.

The MinRouteAdvertisementInternal and MinAsOriginationInterval timers are 
inadequate since they are mostly implemented on a peer session basis and 
studies have shown when they are used they severely degrade route
convergence 
time. The problem with statically defined timers (initialized during system 
load or session establishment phase) is that they do not adjust to peer 
internal dynamically changing congestion conditions. The problem with the 
congested peer using TCP flow control to reduce its congestion condition, is

that it completely stops all incoming session traffic and thus preventing
any 
messages of high priority nature from being seen.

TCP Out of Band Data or Urgent Message is one way to bypass the TCP flow 
control condition and allow high priority BGP messages to get to the 
congested peer, assuming it is able to read the session socket queue. This 
solution has its drawbacks as described in [UNIX-NET] chapter 21, p. 568. 

Internet Draft draft-abarbanel-bgp-route-update-pacing-00.txt [Page 3]

This document presents a mechanism by which the BGP session of a congested 
router need not be degraded to the level that communication is broken with 
its peers. By using a peer to peer pacing mechanism which allows one peer to

rate limit the number of update messages per second received from another 
peer, it can avoid the severe congestion conditions and process these 
messages at a manageable level. In addition, the uncongested peer has the 
knowledge to send high priority messages ahead of or in place of the normal 
high volume of update messages.

Usually, topological disturbances are spiky in nature and once they subside,

the network returns to its optimum path oriented level. Whatever caused the 
network to become unstable, such as routers handling too much data in a
small 
period of time or routers loosing their sessions or their links going up and

down, occurs in most live network on an infrequent basis. By using the
pacing 
mechanism as outlined in this draft, a BGP peer can prevent the serious 
congestion long before it is in trouble and thus ride through topological 
disturbances and still regain its stability without causing its sessions to 
drop or its routes/paths to be discarded and recomputed.

The assumption in this spec is that the source of most or all of the
internal 
BGP router congestion is due to the heavy reception of update messages from 
neighboring peers containing large number of routes.

5. BGP Update Pacing Mechanism

The BGP Update Message Pacing Mechanism is used to slow down the rate at 
which a peer sends update messages to another. This extension to the BGP 
protocol is simplified to use session non-destructive messages such as
INFORM 
as described in [INFORM]. The pacing is performed dynamically upon
congestion 
detection and subsidence and thus needs to use the new [INFORM] message that

will not infringe on the underlying BGP protocol or all its semantics and 
rules.

5.1 BGP Update Message Pacing (PACE) Dynamic Capability

The BGP Update Message Pacing capability is dynamically negotiated with all 
BGP speakers as type code=PACE (where, PACE=TBD) per TLV structures as 
defined in [BGP-CAP] of the OPEN message or anytime after session is 
established using the Dynamic Capability Message as defined [DYN-CAP] 
specification. All those peers that accept the PACE capability will be 
expected to support the new INFORM message as defined in [INFORM] 
specification to carry the pacing TLV structure. 

Internet Draft draft-abarbanel-bgp-route-update-pacing-00.txt [Page 4]

All PACE capable routers will provide a configuration option to their 
operator to enable the BGP Update Message Pacing mechanism on a per peer 
basis. 

Any peer wishing to withdraw the PACE capability can do so dynamically using

the Dynamic Capability message as outlined in [DYN-CAP] specification. Once 
withdrawn, affected peering session will remain intact but will not benefit 
from the performance improvements offered by the pacing mechanism.

5.2 Use of INFORM Message

The INFORM message as described in [INFORM] is used to carry the rate 
limiting (pacing TLV) control structure to neighboring peers. This 
information informs the peer that the current BGP router has entered a 
congestion state and it is to rate limit its transmission to the level 
specified. 

The INFORM message contains the following PACE TLV structure:

Type = Pacing Information, type=TBD 
Length = 2
1st Byte of Value = Cmd as shown below 
2nd Byte of Value = Level as shown below

Cmd Description 
------- -----------
11 Request to Pace update messages
12 ACK Response to Pace Update messages
13 NACK Response to Paace Update Messages

A. Request to Pace Update Messages Level Indication (code=11)

The rate is divided into sub levels in term of categories (Gray, Yellow, 
Orange, Red, and Green) to denote the level of pacing which is directly 
related to the level of congestion experienced within the congested peer.
The 
colors are used as simple handles used for flagging the severity of the 
condition. The colors are also used as indicators for display by the NMS to 
identify the rate limiting levels from any neighboring peer to the operator.

Associated Level Management MIBs are defined in section 6. 

The actual pacing control is done via the sub levels within these colors. 

Level Description
------ -----------
Gray (Entering minor congestion state)
1 Reduce update message traffic by 10% your normal rate.
2 Reduce update message traffic by 20% your normal rate.
3 Reduce update message traffic by 30% your normal rate.

Internet Draft draft-abarbanel-bgp-route-update-pacing-00.txt [Page 5]

Yellow (Entering medium congestion state)
4 Reduce update message traffic by 40% your normal rate. 
5 Reduce update message traffic by 50% your normal rate.
6 Reduce update message traffic by 60% your normal rate.

Orange (Entering major congestion state)
7 Reduce update message traffic by 70% your normal rate. 
8 Reduce update message traffic by 80% your normal rate.
9 Reduce update message traffic by 90% your normal rate

Red (Entering critical congestion state)
10 Complete cessation of all update message traffic (flow off). 
At this Level the receiving peer might decide to bypass the 
Congested Router and pick another less optimal router for the
affected Routes.

Green (Exiting any congestion state)
0 Restore update message traffic to your normal level (flow on).
Routes redirected can be resumed to the original peer.

The receiving peer will send an INFORM message with an ACK or NACK
Indicating 
it has received and understood the pacing request and will either comply
with 
the it or forbid it. If the congested peer receives a NACK, it should remove

that peer from its list of PACE capable peers but still maintaining its 
session without the use of Pacing. If the NACKing peer decides at a future 
date to re-enable the Pacing with the local peer, it will renegotiate the 
PACE capability with the local peer at that time.

B. Response Message Error Indication:

Level Description
------ -----------
0 ACK indication. The peer will comply with the 
Pacing level request.
1 NACK indication. The peer is unable to perform Pacing or comply 
with the pacing level requested.

5.3 INFORM Message Response Timer

When a congested peer sends an INFORM message with a "Request to Pace Update

Messages (code=11)" to all peers that support the pacing feature, it will 
also start an associated INFORM Message Response Timer. Any peer that does 
not respond within a 30 second timeout period with either an ACK/NACK INFORM

Response message, its associated session will be dropped. This is done to 
clear any PACE capable peers that are also congested to the point where
their 
communication with the local congested peer is severed.

Internet Draft draft-abarbanel-bgp-route-update-pacing-00.txt [Page 6]

5.4 Congestion Detection Within the Router

There are at least two ways congestion could be detected and measured by a 
BGP router.

- A high CPU utilization condition 

- A Lack of available memory to accept incoming BGP messages or inability 
to get memory to successfully complete BGP processing. 

5.4.1 CPU Utilization Based Level Computation

It is recommended that when the congested router detects its inability to 
perform route calculations or accept new BGP session messages at a normal 
rate meaning the current router's CPU utilization is more than 49%, it
should 
inform all its peers of a rate limiting level and slow them down
accordingly. 

The recommended way for computing the Level, based on percent CPU 
Utilization, is done using the following:

Level = (((%CPU Utilization - 50) x 2) + 5) / 10).
Where, % CPU Utilization is a whole integer number. Use Integer math 
here to remove any fractional value.

Note: If computed Level is negative, go to Green and set Level = 0. If 
Level = 0 send INFORM message with pacing Level = 0, implying return 
to normal (100%) update rate.

Level implies, whatever number of messages you transmitted per second 
before, transmit (100 - (Level x 10))% of that now. 

e.g. If you transmitted 100 messages/second before. A level 2 (20%) 
reduction will cause you to transmit 80 messages/second now.
Assumption is that these messages have an average size (1500 bytes).

5.4.2 Memory Allocation Based Level Computation

It is recommended that when the congested router detects its inability to 
perform route calculations or accept new BGP session messages at a normal 
rate meaning the current router's memory allocation is more than 69%, it 
should inform all its peers of a rate limiting level and slow them down 
accordingly. 

The recommended way for computing the Level, based on percent Memory 
Allocation, is done using the following:

Level = (((%Memory Allocated - 70) x 2) + 5) / 10).
Where, % Memory Allocated is a whole integer number. Use Integer math 
here to remove any fractional value.

Internet Draft draft-abarbanel-bgp-route-update-pacing-00.txt [Page 7]

Note: If computed Level is negative, go to Green sub-group and set 
Level = 0. If Level = 0 send INFORM message with pacing Level = 0, 
implying return to normal (100%) update rate.

5.5 INFORM Message Throttling

Once the congested peer receives acknowledgement from another peer, it will 
send a modification INFORM message with a new Level to that peer after the 
computed pacing Level changes by at least 1 value. This will amount to no 
more than one INFORM modification message every 5 seconds. This is done to 
debounce any spiky bursts of INFORM messages to all PACE negotiated peers 
each time the computed pacing Level changes. Depending on vendor 
implementations, the internal utilization levels could change at the 
Microsecond or Millisecond rate.

6. Implementation Specific Mechanisms

The Memory Allocation and CPU Utilization Level detection algorithms 
discussed in section 5.4.1 and 5.4.2 are suggested ways one can implement 
these solutions. However, each vendor can implement a unique Memory 
Allocation and CPU Utilization Level detection algorithms that best suits 
his/her needs and will not negatively impact the overall BGP Route Update 
Pacing mechanism described in this spec. Any issues relating to internal 
implementation algorithms are outside the scope of this document.

7. Level Indicators Management MIBs

TBD

8. Security Considerations

This extension to BGP does not change the underlying security issues.

9. References

[BGP-CAP] Chandra, R., Scudder, J., "Capabilities Advertisement with 
BGP-4", draft-ietf-idr-rfc2842bis-02.txt

[DYN-CAP] Chen E., Sangli S., "Dynamic Capability for BGP-4", 
draft-ietf-idr-dynamic-cap-02.txt, October 2002.

[INFORM] Nalawade G., Scudder J., "BGPv4 INFORM Message",
draft-nalawade-bgp-inform-00.txt, December 2002

[BGP-4] Rekhter, Y., and T. Li, "A Border Gateway Protocol 4 
(BGP-4)", RFC 1771, March 1995.

Internet Draft draft-abarbanel-bgp-route-update-pacing-00.txt [Page 8]

[BGP-4-DRAFT] Rekhter, Y. and T. Li (editors), "A Border Gateway Protocol 4
(BGP-4)", Internet Draft draft-ietf-idr-bgp4-18.txt, 
January 2002.

[UNIX-NET] Stevens, R., "UNIX Network Programming, Vol 1", Second Edition
1998 Prentice Hall, Inc.

10. Author Information

Ben Abarbanel
Marconi Communications
1595 Spring Hill Road, 5th Floor
Vienna, VA 22182
Email: [email-masked]

   _____  

Do You Yahoo!?
Yahoo! Autos <http://autos.yahoo.com/>  - Get free new car price quotes

---

From: "Abarbanel, Benjamin" <[email-masked]>
Date: Tue, 23 Jul 2002 10:44:50 -0400

Manav:

Response below.

> From: Manav Bhatia [mailto:[email-masked]]
> Sent: Tuesday, July 23, 2002 8:02 AM
> To: Mareline Sheldon
> Cc: [email-masked]
> Mareline,
> | | What happens say when an uncongested router receives 
> pacing information
> | | telling it to reduce the advertisement rate by some 70%. 
> In that case
> there
> | | will be many UPDATEs lying with it remaining to be yet 
> announced to the
> | | remote peer which is experiencing congestion. Now what if the
> uncongested
> | | network receives withdrawals (both implicit and explicit) 
> for certain
> | | prefixes which have already been announced or remain yet to be
> announced?
> | | You open up a can of worms now !!
> | |
> | | You have to now flush all your output buffers regardless of the
> congestion
> | | state your remote peer lies in!
> |
> | what exactly do you mean by this?
> 
> If you have some UPDATEs to announce and if you are holding them back
> because of the congestion you perceive in the network and then if you
> recieve a WITHDRAW for one of those same prefixes then you 
> have to first
> advertise the UPDATEs because it is invalid to announce the 
> same address
> prefix in the WITHDRAWN routes and the NLRI field of an 
> UPDATE message. You
> thus end up flushing all the BGP UPDATEs you had piled up 
> waiting for the
> congestion to clear out at your remote end. The better 
> solution is to let
> TCP take care of the congestion internally as it has always 
> been OR if you
> want to dynamically adjust to the congestion then you may 
> re-negotiate the
> HoldTimer using the Dynamic Capabilities.
> 
> I hope i make myself clear.
> 
Manav:

Whether you use the pacing method or TCP flow control the updates will be
roadblocked in the source router and the end result will be the same. 
Remember TCP in the source router will only hold a window full of packets
and the rest of the UPDATE routes will be left in the RIB-OUT area of BGP.
Unless you
have implementation that moves 100,000+ routes into some transmit queue that
sits
above TCP. In which case, I doubt you would update this queue of update
messages
when new withdrawls come in affecting those routes.

So if your implementation leaves it in the RIB-out, the new Route withdrawls
coming in the source router could cause updates that have not made it to the
destination router to be affected. 

So there is no difference between the two mechanism in this case.

Ben

---

From: "Abarbanel, Benjamin" <[email-masked]>
Date: Tue, 23 Jul 2002 11:02:14 -0400

Brgds:

   for case 1 : 

Could the graceful restart mechanism be a solution to this pblm . 

Say R1 is peering with N peer . R1 is under stress 

R1 select to close 1..m  TCP session with peer 1 to m , deal with the
remaining  session (m+1 ...N) and clear his congestion (now R1 have 

enough HP ) . R1 initiae a session to peer 1 ...m with restart bit set and f
bit set  .  

Closing down a peer session just because you cannot handle its load is a bad
idea.  And then using some

gracefull session restart mechanism to control the load. I dont see how that
will work efficiently and keep routes up to date?

 Here we assume that R1 could sort his stress within a restart time value.

  What if the stress does not go away after a time period?

Ben

   _____  

Chat with friends online, try MSN Messenger: Click
<http://g.msn.com/1HM1ENXX/c144??PS=47575> Here

---

From: "Natale, Jonathan" <[email-masked]>
Date: Tue, 23 Jul 2002 18:06:37 -0400

TCP URG ???
2nd session ???

-----Original Message-----
From: Dennis Ferguson [mailto:[email-masked]] 
Sent: Tuesday, July 23, 2002 5:48 PM
To: Manav Bhatia
Cc: Abarbanel, Benjamin; [email-masked]
Subject: Re: My BGP Route Update Pacing Draft 

Hello,

> The crux of your draft is that if a peer gets so congested that it is not
> able to send any UPDATE/KEEPALIVE message to the remote uncongested peer
> then the session can break down because of the Hold Timer expiring and it
> can be prevented by sending some pacing information which will inform the
> uncongested peer to limit its route advertisement rate.

Note that, right now, it is impossible to send BGP updates to a peer any
faster than the peer is willing to process them.  Period.  TCP flow control
ensures this.  It hence isn't possible for update senders to put any more
load on a peer than that peer itself chooses to accept.  If a BGP peer
wants to ensure that its CPU is never more that 50% busy it can do
that already by not spending more than 50% of its CPU processing BGP
updates,
there is no way its neighbours can make it work harder than it wants to.
And the peer is free to stop processing update messages whenever it needs
to, like when it needs to send KEEPALIVEs, so a peer which voluntarily keeps
processing incoming updates which it should have been sending KEEPALIVEs is
just broken and needs to be fixed.

I hence don't think this draft has anything to do with fixing busy
BGP speakers, they are already in full control of their load.  The
draft seems to in fact have something to do with the inconvenience
an idle remote peer might suffer if the busy guy takes a long time
to respond to, say, a Capability message, but since the guys who are
inconvenienced are the guys who have available CPU cycles it seems to me
to be appropriate for those guys to spend the cycles dealing with this
themselves.

Dennis Ferguson

---

From: "Abarbanel, Benjamin" <[email-masked]>
Date: Wed, 24 Jul 2002 10:05:56 -0400

Dennis:
  I respect your knowledge and have read your books. You have said a
mouthfull
here. I have no ground to respond so I wont. Thanks for the long
explanation.
I hope you are right and I am wrong and we are not loosing a good idea.
Anyway
I have my thoughts and doubts about things, but that is a matter of opinion.

Thanks again,
Ben

> Sent: Tuesday, July 23, 2002 5:42 PM
> To: Abarbanel, Benjamin
> Cc: [email-masked]'
> Subject: Re: FW: My BGP Route Update Pacing Draft 
> 
> 
> Hello,
> 
> > > 
> > > Why not relly on the transport. Flow control is after all
> > > traditionally the function of the transport. What makes 
> you think it
> > > needs to be reinvented... and how is your version of flow control
> > > supperior to the current one ?
> > >
> [...]
> > The other end that is being flow controlled does not know why 
> > or by how much he
> > will be flow controlled off. Remember on a long session with 
> > many hops it takes significant time to communicate the flow 
> > control condition by ACKing or not ACKing messages. For me 
> > this is not a direct but indirect mechanism.
> > In the interim, the pipe will be full of data.
> 
> I think you are confused about this.  TCP flow control does 
> not work by
> withholding ACKs. TCP always ACKs messages that are received, and flow
> control is instead implemented by directly telling the other end
> not to send more.  The sender is informed of this condition as soon as
> it occurs, and will be informed that the condition has cleared in the
> time it takes for a packet to transit the path between the 
> peers (which
> is why it is probably good to have a TCP window of outstanding data
> at the receiver that it can process until the sender gets the
> message to send more, and noting that the receiver is in full control
> of just how much or little data that is).  It doesn't get much better
> than that.
> 
> Now a sender who has had all outstanding packets ACK'd but which
> is flow controlled off knows precisely why this has occurred; it
> has been sending updates faster than the receiver was willing to
> process them.  There is no ambiguity, this is the only way this
> situation could occur.
> 
> Therefore, out of everything you said above, the only thing I can
> see which is perfectly accurate is that the flow-controlled sender
> won't know how long it will be flow-controlled off.  This will depend
> on what other things the receiving router has to do which it finds
> more important than processing BGP messages from this particular
> peer.  It is hard to see how the receiver could know how long that
> is going to take until it actually does it, however, so it seems
> that telling the sender this would require the receiver to be able
> to predict the future (the sender knows all about the past already
> so there isn't much use having the receiver tell the sender about
> that).  I can't dispute that both prescient routers and 
> fortune tellers
> might be useful, but I'm not sure that either is a practical
> possibility in real life.
> 
> While I doubt the implementability of the future-prediction you
> seem to be looking for here, however, I think there is another
> more fundamental issue which says your approach is incorrect.  That
> is, TCP flow control already puts the congested receiver in full
> control of its load.  No one can send data faster than the congested
> receiver is willing to process it.  The congested receiver directly
> controls how much data is outstanding when flow control is asserted.
> The sender can't do anything to the receiver that the receiver doesn't
> explicitly permit.  With TCP flow control the congested receiver
> has full control of its resources.  If there is any problem at all
> here, then, it is that the sender has no control at all over the
> receiver's resources and hence may be inconvenienced by a congested
> receiver which chooses to take its time responding.  And, in fact,
> if we're talking about Capability messages in particular, it is of
> no advantage in general at all to the busy receiver to get this
> message any earlier than updates or anything else; it doesn't 
> even know
> the message is coming until it sees it.  Only the uncongested sender
> knows there is an outstanding capability message.
> 
> So if there is a problem here at all it is a problem for the
> idle sender, which might have to wait for a response to a Capability
> message, rather than the busy receiver.  Yet your proposed solution
> for this is to have the busy receiver do yet more work, continually
> predicting the future and sending Inform messages about it to all of
> its (possibly many) peers, on the off chance that a sender which
> might someday have a Capability message to send, and which has CPU
> cycles to burn, isn't inconvenienced.  Since it is the guy who has
> CPU cycles available which has the problem in the first place it seems
> utterly backwards to make the guy who has no CPU cycles free do more
> work to fix it.  The guy with the CPU cycles should be the one to fix
> this, i.e. the sender should spend his free cycles doing whatever it
> takes to wait until the congested guy gets around to responding.  Then
> everything works fine with no protocol changes.
> 
> > > In BGP a receiver can choose to accept a anyrate of updates 
> > it wants.
> > 
> > Yes, it can by reading or not reading the TCP socket queue. 
> > Which gets us
> > back to the original point I mentioned earlier. TCP FLOW 
> > Control is not
> > perfect.
> 
> Your original point seems to have been based on a faulty understanding
> of how TCP flow control works.
> 
> > > Please do offer more to substantiate this other than your 
> claim that
> > > the problem exists.
> > >
> >  
> > By the fact that the pacing control is more direct and implicit to a
> > particular
> > traffic type (in this case update messages). The pacing mechanism is
> > deterministic, whereas TCP flow control is not. You should 
> talk to people
> > who build IPC mechanisms to replace TCP about this issue. 
> 
> Or the guys that built deterministic FDDI or Token Ring or Token Bus
> or ATM LANs to replace non-deterministic Ethernets.  We have networks
> where non-deterministic stuff just happens despite our best efforts to
> avoid it, so you might be better off to write code which deals with
> that rather than depending on some other box to deterministically
> predict the future so you aren't inconvenienced.  While you 
> are correct
> that this would be easier for your box if other routers could predict
> their future for you with 100% reliability, I don't think this is
> likely to be possible in any case.
> 
> Dennis Ferguson
>

---

From: "Abarbanel, Benjamin" <[email-masked]>
Date: Wed, 24 Jul 2002 10:16:18 -0400

Dennis:
  I agree we should not send updates that are flow controlled that have 
new changes imposed on them by new events in the network. Only send the 
current state of all routes. It is the responsibility of the sending
router to report current and accurate information to its peers.

As we can see flow control conditions are a messy subject to deal with 
and does often ripple back to the application to take new (current) 
actions.

Ben

> Sent: Tuesday, July 23, 2002 6:36 PM
> To: Manav Bhatia
> Cc: Mareline Sheldon; [email-masked]
> > congestion to clear out at your remote end.
> 
> Why would you do this?  If you receive a withdrawal for a 
> route for which
> you haven't sent an UPDATE yet, why wouldn't you forget about 
> the UPDATE
> and just send the withdrawal (or, if you've never advertised the route
> to the peer, just send nothing at all)?  If a route is flapping and
> goes up and down 62 times while your peer has you flow 
> controlled, would
> you really send the 62 flaps when the peer allows you to send 
> again?  Why
> wouldn't you just send the final result?
> 
> You need to do routing protocol implementations in a way which avoids
> unbounded state, and doing this means that you shouldn't be recording
> all route transitions to play back to a peer which isn't listening at
> the moment.  If the only state you keep is what you advertised to the
> peer before and what the state of the route is now, then the amount
> of state you need to keep remains proportional to the number of routes
> you have and you only need to advertise the current state (if 
> it doesn't
> match what the peer already knows, otherwise you send nothing) when
> the peer starts taking updates again.
> 
> > The better solution is to let
> > TCP take care of the congestion internally as it has always been
> 
> I don't think this has much to do with TCP, TCP flow control works
> the same way.
> 
> Dennis Ferguson
>

---

From: "Abarbanel, Benjamin" <[email-masked]>
Date: Wed, 24 Jul 2002 10:23:00 -0400

Its 16K per session correct. If I have 500 sessions that is 8 million bytes.
 
Ben 

-----Original Message-----
From: vrishab sikand [mailto:[email-masked]]
Sent: Wednesday, July 24, 2002 12:59 AM
To: Manav Bhatia; Dennis Ferguson
Cc: [email-masked]
Subject: Re: My BGP Route Update Pacing Draft

  Manav Bhatia <[email-masked]> wrote: 

Hi Dennis,

| Why would you do this? If you receive a withdrawal for a route for which
| you haven't sent an UPDATE yet, why wouldn't you forget about the UPDATE
| and just send the withdrawal (or, if you've never advertised the route
| to the peer, just send nothing at all)? If a route is flapping and

Depends on whether your BGP UPDATE packets waiting to be announced are in
some transit queue above the TCP or lying below it. If you have already
given the UPDATE to the TCP then there isnt much that you can do upon
receiving the WITHDRAW. If its the former case (above TCP) then i have yet
to see any implementation which scans through all the pending BGP UPDATEs
it has in its transmit queue upon receiving the withdraw to remove the NLRI
if it exists in the withdraw also OR scanning through all the WITHDRAW
messages when it recieves any new NLRI to advertise.

Such  implemenations exist. and it does not require scanning any update
list. The route entry itself can have enough information in it to indicate
to which neighors it has been sent and to which neighors it has been queued
to be sent to. Off course once it is given to TCP it is as good as sent but
thats not a long list as TCP buffers are not VERY large @16K in most
implemenatations.

Manav

   _____  

Do You Yahoo!?
Yahoo! Health <http://health.yahoo.com/>  - Feel better, live better

---

From: "Abarbanel, Benjamin" <[email-masked]>
Date: Wed, 24 Jul 2002 10:28:50 -0400

I will agree the issue with keeping updates current might become stale when
flow control is activated, because each implementation might be different
and
thus may or may not clear up the updates before they leave the box. So the 
effect of historical replay will take place at the destination router. Even
if its physically wrong it will adjust and resolve the state to the current
levels. The problem comes in when the flow control is constantly engaged and
the sending and receiving routers start to drift out of phase from reality.
That is when we will see forwarding problems, due to inaccurate routes.

I will say again, it is the responsiblity of the sending router to send the
best and most current route information available to it, even if it means 
it has to go through some session queue and cleanup the updates, which I
am sure most implementations dont do. As for the TCP queue it is as good as
gone, but as other's have said that is only 8k bytes.

Ben

> Sent: Wednesday, July 24, 2002 12:07 AM
> To: Dennis Ferguson
> Cc: [email-masked]