Subject: Re: Evaluation of Gateway Location proposals

From: "Hussein F. Salama" <[email-masked]>
Date: Tue, 04 May 1999 11:32:57 -0700

Matt,

Please find some comments and questions on your analysis below.

Hussein

Matt Squire wrote:
> 
> >
> >2. Matt raised numerous transport issues (TCP or not TCP). I think its
> >fair to say we want to avoid designing a new transport protocol. There
> >is work underway elsewhere (sigtran and slums) to do this, which we can
> >leverage. It would seem to me that if something new is needed, this
> >would also be independent of whether the protocol baseline is BGP or
> >SCSP, since presumably one could use this new transport protocol with
> >either.
> >
> 
> I certainly agree that we don't want a new transport protocol.  But well
> short of designing a new transport is to design reliability into our
> application protocol (ie with acks, timeouts, re-transmissions, etc), just
> like most other application protocols.

I prefer to view TBGP/GLP as a call routing protocol not as an
application protocol. A call routing protocol bears closer resemblance
to layer 3 routing protocols than it has to application protocols. In
order to scale well and not to overload the network with too much
control traffic, routing protocols may have to tolerate a lower level of
reliability than some application protocols. 

> 
> >3. To me, the MAIN differences between a BGP based approach and SCSP
> >based approach are:
> >  a. how the data is synchronized between peers
> >  b. the intra-domain topology
> >
> >I think we are clear on (b), so (a) is the issue to explore. Hussein had
> >made reference to a 2-stage exchange only being needed with BGP, while
> >its three with SCSP. These are the kind of differences I think we need
> >to explore here. I'd invite both authors to comment on the differences
> >in how the data is exchanged and updated between peers. Lets stick with
> >inter-domain for the moment...
> >
> 
> Here's a brief synopsis of the data exchanges for each alternative.  In
> both cases, the exchanges are just peer-to-peer.
> 
> BGP
> ***
> 
>   BGP1) OPEN.  OPEN messages are exchanged to bring the link up.
> Basically, two peers connect and send an open message to the other.  A peer
> either acks the OPEN with a KEEP-ALIVE, or nacks the OPEN with a
> NOTIFICATION.  When your OPEN gets ack'd, the peer connection is considered
> to be established (ie have bi-directional connectivity).  BGP can also
> perform version negotiation and capabilities exchanges with the OPEN
> messages.
> 
>   BGP2) UPDATE.  Once a peer connection is established (via ack'd OPENs),
> BGP peers exchange UPDATE messages.  UPDATE messages are the messages that
> contain the routing information.  Each UPDATE message may contain a number
> of withdrawn routes and a single feasible route.  A withdrawn route is a
> route that should be removed from the peer's RIB-IN.  A withdrawn route is
> defined by a address/prefix-length.  A feasible route is defined as the
> NLRI (network layer reachability information, ie a list of
> address/prefix-length combinations) and its attributes.   A key mandatory
> attribute is the NEXT-HOP attribute which gives the next-hop for the NLRI.
> Basically, you send an UPDATE for each possible NEXT-HOP in your routing
> database.

You are talking here strictly about BGP for layer 3 routing. Even in
this case, a single UPDATE message may contain multiple NLRI
(destination prefixes) under the condition that they share the same
attributes (NEXT_HOP, AS_PATH, ...) . TBGP uses the MP_REACH_NLRI
attribute to advertise prefixes. Similar to BGP, the MP_REACH_NLRI
contains a list of NLRIs which share the same attributes.

The original BGP NEXT_HOP attribute will not be used in call routing.

The above discussion brings up the question I had for some time: Does
GLP/SCSP define a mechanism for explicitly withdrawing a route and
removing it from the call routing databases?

> 
> KEEP-ALIVE messages are periodically going in the background to ensure that
> the link is up.
> 
> BGP UPDATE and NOTIFICATION messages are not acknowledged as TCP
> acknowledgment is considered sufficient.

I'll comment on this further down.

> 
> SCSP
> ****
>   SCSP1) Hello.  Peers exchange Hello messages to ensure bi-directional
> connectivity between peers.  This is equivalent to the BGP OPEN exchanges
> but without anything like version negotiation or capabilities exchange.
> Hello messages are also periodically exchanged in the background to ensure
> the link us still up.
> 
>   SCSP2) Cache Aligment.  SCSP distinguishes between the state where two
> peers have had their databases synchronized and when they have not.  When a
> peer connection first comes up (via Hello), the peers enter the Cache
> Alignment phase where the entire databases must be exchanged.  First, there
> is a negotiation to determine who is the master and slave of the peering
> relationship.  Next, the peers exchange summary records with each other,
> where a summary record is a short-version of the record - enough to
> identify it, but doesn't have the real data.  This phase has no parallel in
> BGP.

And I don't see what advantage call routing can gain from this phase?

> 
>   SCSP3) Cache State Update.  A server maintiains a list of the records
> that it needs to get from its peer (the summary records in the previous
> phase).  A server requests these records via a Solicit message (ie I need
> the full record for these summary records).  The response to the the
> request is a Update Request message containing some/all of the records.  A
> server continues to send further Solicit messages for records that it has
> not yet received.  A server ack's that the records have been received with
> an Update Reply containing the summaries of recieved records.  The Update
> Request messages are similar to the UPDATE messages of BGP.

From reading the SCSP spec, I thought that in the Cache State Update
phase no CSA
Solicit messages are required? Just CSA Request/CSA Reply messages. If a
CSA Solicit message is required, this makes SCSP's overhead three
messages per update as compared to BGP's one message per update.

> 
> Acknowledgements in SCSP are done per-record (ie requests for a record are
> acknowledged by sending the record, record transmissions are acknowledged
> by sending a record summary in an Update Reply).
> 
> SOME DIFFERENCES
> ****************
> 
> I think Hussein's "2 vs 3" stages are identified above.  SCSP has the added
> stage of Cache Alignment where database summaries are exchanged.
> 
> BGP uses TCP, SCSP uses X (pick your transport).
> 
> BGP exchanges and message formats are certainly more efficient, SCSP more
> generic.  In the peer-to-peer case (ie inter-domain), some SCSP header
> fields are redundant (example: reciever and sender ID).
> 
> SCSP does per-record acks and requests, which is flexible, but can be a
> pain in the ass and inefficient.  BGP UPDATE/NOTIFICATION messages are
> unacknowledged and assume TCP provides reliability for the application.
> 

BGP does not provide a mechanism to positively Ack UPDATE messages.
However the NOTIFICATION message is used to provide a negative Ack in
response to errors in the contents of, or in the processing of, an
UPDATE message. This may be looked at as a more efficient mechanism for
reliability than positive Acks. Since transport reliability is already
guaranteed by TCP, error Notification usually indicates fatal problems
with the BGP FSM, and therefore is followed by closing the peer
connection. I can see situations where, in response to an error in
processing an UPDATE message, it should be sufficient to send an error
NOTIFICATION and to ignore that UPDATE, but not closing the entire
connection. But that's the way BGP works.

> SCSP Update Requests can contain many records.  BGP UPDATEs contain a
> single route (maybe multiple withdrawn routes).

I commented on this earlier.

> 
> SCSP bases record identification on record summaries (source identifier,
> cache 'key' for record), so to replace a record you keeep the record
> summary and replace the data contents.  BGP identifies records based on the
> contents (ie NLRI), so the data content is examined to determine new vs
> replacement.
> 

From discussions with people who implemented IDRP in the past, my
understanding is that
it relied on a key for record location, which proved to be "difficult to
implement with little practical gain."

-- 
Hussein F. Salama
Cisco Systems
Mail Stop SJC6/3, 170 W. Tasman Drive, San Jose, CA 95134
Voice: +1 (408) 527-7147, Fax: +1 (408) 527-1714

---------
This message came from the IETF IPTEL Working Group Mailing List.

---

From: Jose Costa Requena <[email-masked]>
Date: Thu May 06 10:22:53 1999

Hi Hussein,

Thanks for your comments.
I have implemented the SCSP and actually I am testing the first version in 
the Lab., I am glad to receive some comments to change some details and 
polish the implementation.

>Assume server A advertises a record X to server B. Then, at some later
>point in time, record X is deleted. Server B won't learn that record X
>has seized to exist until it solicits an update from server A for that
>record? This could be a very slow operation. How frequently should B
>solicit updates from A, in order to minimize the probability of using
>outdated or deleted records?

You are right, it is a good point because in the case that we fixe a short 
time out in the cache entry, all the database will be updated in a short 
interval but the drawback is that it will increase the traffic.Otherwise, if 
we fixed a long time out it will take a long time to update all the records.

Professor Kantola and me are working in the feature provided by the Family 
Identifier to split the database up in different sections. The part of the 
routing table that is changing dynamically will be fixed with a shorter time 
out than the other.Thus, the former part will be updated more frequently. 
With the Family ID we can distinguish two instances of the same SCSP running 
in the same Sever Group and maybe that is the key to differentiate a section 
in the routing table that is more dynamic than another section that is more 
static and requires less maintenance.

>Advertising route withdrawals to a server which is multiple hops away
>will be very slow using this mechanism. Let's say every server
>periodically solicits updates every time interval t. It will take a
>server, C, n*t time units, in the worst case, to learn that a record has
>been withdrawn, where n is the number of hops between server C and the
>server which has withdrawn the record. 

>What's wrong with conventional routing logic: when a server detects a
>change to the state of a record, it immediately triggers an unsolicited
>update to its peers. When a peer receives the update, if it causes any
>changes to that peer's records, it immediately triggers an update to the
>peer's peers. And so on. Wouldn't that be faster?

>The SCSP RFC says: "The "N" bit signifies that this CSAS Record is
>actually a Null record.  This bit is only used in a CSAS Record
>contained in a CSU Request/Reply which is sent in response to a CSUS
>message." This means that an update advertising a record deletion must
>be solicited. How about other changes to the attributes of an existing
>record? Can they trigger an unsolicited update, CSU Request?

>Hussein

I guess that the conventional routing logic that you mention in the last 
paragraph is performed by the SCSP on its last phase. When the peer has 
reached the Cache State Update phase they will perform a mirroring state.It 
means that when any cache entry has timed out, or it has been modified, this 
LS will send a CSU Request to each of its DCSs. In this Request we can 
advertise either that this record is newest or is going to be deleted 
setting the bit N. Thus, we don't have to wait to Solicit and update message 
for that record because is just the LS that has the record which is going to 
be removed who will send the Request to all the DCSs advertising that this 
record is going to be deleted from the routing table using the N bit.

Thanks for your comments, I am looking forward to hear from you about my 
ideas and what is your opinion.

Helsinki University of Technology
Laboratory of Telecommunications Technology
Jose Costa Requena
Research Scientist
Room: SG 227B
Tel: + 358 9 451 2475
email(internet): [email-masked]
http://keskus.tct.hut.fi/tutkimus/ipana/pers/jose.shtml

---------

---

From: Jose Costa Requena <[email-masked]>
Date: Sat May 15 17:58:25 1999

Hello,
 
> Matt Squire wrote:
> > 
> > >I think in call routing, when the connection between LS A and LS B is
> > >disconnected (due to any reason), LS A should immediately delete any
> > >call routes it has previously learned from LS B. As a result, when
>  the
> > >connection between LS A and LS B is restored, both LSs will have to
> > >exchange their entire call routing databases.
> > >
> > 
> > If the next hop on a call route received from B is B, then the route
>  should
> > be deleted when B goes away.  But in when the next hop is not B, we
>  might
> > consider having LS A keep the route around for "a little while." 
>  Haven't
> > really given this enough thought yet, but it might help with transient
> > failures (ie B is just being rebooted and will be up again in two
>  minutes).
> 
> What if B advertises the addresses of the gateways in its domain as the
> next hops, not its own address. When B goes away, there will be no way
> to advertise to other domains updated information about these gateways
> and the prefixes they can reach (assuming B is the only LS in that
> domain).
> 
> This is an issue where we can present good examples for both approaches.
> I think deleting all call routes learned from B immediately when B is
> disconnected is the safer approach. We will probably re-learn the routes
> which are still valied through alternate paths.
> 
> Hussein
> 
There are something that I cannot understand:

If the LS B is the unique server in that domain which can advertise about 
the gateways in its domain as the next hops, then if it goes down there will 
be no other way to advertise about the gateways of that domain. If I'm right 
in that supposition we cannot re-learn the routes, that are still valid, but 
they cannot be advertised through the B LS because actually it is down.

I think that it would be wiser if the last updates from the B LS are 
maintained until they will be updated from other paths, because as Matt 
wrote it could be either a transient failure of the B LS or maybe the 
connection to this domain through the B server is going to be replaced with 
another path or LS which will advertise about the same gateways that was 
done by the B before.

Best Regards.
Jose

*******************************************************
Jose Costa Requena
Research Scientist
Helsinki University of Technology
Laboratory of Telecommunications Technology
Room: SG 227B, Tel: + 358 9 451 2475
email(internet): [email-masked]
http://keskus.tct.hut.fi/tutkimus/ipana/pers/jose.shtml

---------

---

From: Matt Squire <[email-masked]>
Date: Tue, 18 May 1999 09:05:26 -0400

>> 
>> And if the route process is busy or its queues are full or for some reason
>> it just can't take the socket upcall from the TCP layer at this time, then
>> the information is lost.  And that is a problem.  Hence one has to hack the
>> TCP stack and the OS to ensure this never happens.
>> 
>
>I'd appreciate pointers to specific implementations where the OS and the
>TCP stack has been hacked. I am not aware of any. Please also check the
>the NSFNET backbone (phase II) for BGP implementations on standard OSes
>and TCP stacks. And what about the Gated implementation? 
>

I'm aware of these implementations.  In routers, the routing process is the
highest priority process in the box, because you can't forward the packets
without the control.  But in call routing, GLP may not be the most
important thing in the box.  Call routing is a different problem.

Many devices are going to handle media, call routing (SIP/323), GLP, and
more.  In this case, existing media connections have to have the highest
priority.  When resources are tight, existing media streams have to get
whats available so that already accepted calls get the quality they need.
Other things going on in the device may also have precedence over GLP.  

When a protocol is not the most important thing going on in a device,
things can happen inside the device so that TCP acks something but the
information doesn't get processed by its application.  Queues fill up, you
run out of memory, etc.  It happens all of the time.  

Reliability (timeouts, retries, etc.) is just a good thing to build into a
protocol, regardless of its transport.  

- Matt 

---------