Subject: New Draft- BFD Express Path

From: Spencer Giacalone <[email-masked]>
Date: Tue, 23 Nov 2010 10:35:33 -0500

Dear WG,

A new draft, called “BFD Express Path”
(draft-giacalone-bfd-express-path-00) has been submitted for review.
The primary intent of this draft is to mitigate capability gaps and
exploit opportunities in certain performance sensitive networks. BFD
Express Path creates a scalable, flexible, and integrated framework
for gathering detailed network performance information. This
information could be used to influence routing or traffic engineering,
or for other network management purposes. It is assumed that further
drafts will provide routing protocol extensions. BFD Express Path is
focused on real world issues, and is aimed at (for example) the
production network that I am responsible for. I invite discussion,
comments, or questions. The abstract is as follows, and the draft
itself is posted on the WG page:

“In certain networks, such as, but not limited to, financial
information networks (e.g. stock market data providers), network
performance criteria (e.g. latency) have become (or are becoming) as
(or more) critical to data path selection than other metrics. This
document describes extensions to the BFD protocol, such that
network performance information can be gathered in a scalable fashion,
and subsequently used (by other protocols) to make path selection
decisions. These extensions will also provide granular performance
monitoring information.”

Regards,

Spencer

---

From: Vishwas Manral <[email-masked]>
Date: Tue, 23 Nov 2010 16:23:44 -0800

Spencer,

I have some top level comments:

1. Do we need the functionality on a continuous basis or on a demand
basis, when such delay measurements are required?
2. What happens to the authentication as with the A bit the packet
seems to have the authentication field after the defined BFD packet.
3. Backward compatibility with existing implementations?
4. Is 1-way delay required or do we also need 2-way delay?

There seem to be quite some work on the delay measurements in IPPM and
MPLS-TP, you could try to use that effort.

Thanks,
Vishwas

---

From: Mach Chen <[email-masked]>
Date: Wed, 24 Nov 2010 15:53:29 +0800

Hi Spencer,

I think using BFD for performance monitoring is a reasonable way. Two years 
ago, we wrote a draft for the same purpose with relevant BFD extensions, it 
could be found at: http://tools.ietf.org/html/draft-guo-bfd-pm-extension-01. 
The reason for choose BFD for performance monitoring is because BFD is a 
protocol independent technology, BFD-based solution would provide a general 
mechanism for performance monitoring, no matter what the path is IP, MPLS or 
other types.

As Vishwas said in another email, you should consider the compatibility 
issue especially when you want to deploy BFD Authentication at the same 
time.

Best regards,
Mach
--------------------------------------------------
From: "Spencer Giacalone" <[email-masked]>
Sent: Tuesday, November 23, 2010 11:35 PM
To: <[email-masked]>
Subject: New Draft- BFD Express Path

---

From: Mach Chen <[email-masked]>
Date: Wed, 24 Nov 2010 15:56:34 +0800

Hi Vishwas,

At present, IPPM is charterd to do active measurement, and MPLS-TP relevant 
meachanisms only apply to LSP.

Best regards,
Mach

--------------------------------------------------
From: "Vishwas Manral" <[email-masked]>
Sent: Wednesday, November 24, 2010 8:23 AM
To: "Spencer Giacalone" <[email-masked]>
Cc: <[email-masked]>
Subject: Re: New Draft- BFD Express Path

---

From: Vishwas Manral <[email-masked]>
Date: Wed, 24 Nov 2010 10:18:33 -0800

Hi Mach,

I agree to the charters of IPPM and MPLS-TP, however looking at the
draft I see they are passing a lot more details than just originator
and receiver timestamps.

My comment was that it could be looked at in greater detail.

Thanks,
Vishwas

---

From: Spencer Giacalone <[email-masked]>
Date: Wed, 24 Nov 2010 23:36:38 -0500

Hi Vishwas,

Thanks for your comments. My responses are inline.

>basis, when such delay measurements are required?
BFD Express Path is meant to be an optional feature, to be enabled as
needed. Certainly, not all users will need this functionality at all.
When BFD Express Path is enabled, my assumption is that all BFD
packets between two peers will include the Express Path extensions.
Does that answer your question?

>From a backwards compatibility perspective, would it be easier to
leave the auth extension as the next field after the control packet (I
am thinking the answer is yes)?

I would prefer to auth the whole packet, for obvious reasons. From a
hashing perspective, are there any issues with hashing the entire
packet, including the Express Path extension, even if that extension
is unknown (I don’t see one)?

>3. Backward compatibility with existing implementations?
Section 7 in the draft deals with remote systems that do not support
Express Path or a clock sync protocol (when the local system does).
However, the reception of BFD Express Path packets on incompatible
implementations is a valid point. There are two issues there- the new
diag codes, and the extension itself. I don’t see anything in 5880 on
how the reception of unknown diag codes or extensions are to be
handled. I was assuming that conditions like this are ignored,
however, if this is implementation specific, of if I missed something,
perhaps someone can shed light on this.

That said, if we are using up diag codes in this manor, it might be
better to use a single diag code (or some other bit) to denote the
next header as a generic TLV. I am happy to write something up if the
WG concludes that is the way to go.

>4. Is 1-way delay required or do we also need 2-way delay?
My thoughts are that different users will have different requirements,
depending on what they are trying to do. My interpretation is that
users who want 2-way delay on each node can use BFD Express Path in
Echo Mode or set the P bit. The draft supports both (but not sure if
all implementations can handle both). For control plane purposes, I am
inclined to think 1-way delay is what is needed.

>MPLS-TP, you could try to use that effort.
IPPM is an interesting point. To the degree that the IPPM WG has
spelled out metrics, issues, and terms, BFD Express Path should/could
be a complimentary framework for actually providing performance
measurements. This is an example of why I left the actual timestamp
field format TBD. Outside of that, I would prefer to use existing
protocols, which we already run at granular packet intervals, to
introducing a new protocol in our network. Since we already run BFD, I
feel that it is almost perfectly placed to do this. In addition,
having experience with end-to-end “mesh based” monitoring protocols, I
feel that the link-by-link approach in BFD Express Path is superior in
a number of ways (though BFD Multihop, Express Path is mentioned in
the draft).  I am happy to expand on this last point. As a result of
your comment, perhaps I will reach out to the IPPM chairs.

Although my network is an in-sourced MPLS core, MPLS-TP is not on my
deployment roadmap, as we do not require that functionality.

I am happy to further this discussion. Let me know what you think,

Spence

---

From: Mach Chen <[email-masked]>
Date: Fri, 26 Nov 2010 15:26:21 +0800

Hi Spence,

Some comments about your draft!

Seems that the most concern of your network is the delay metric. You also 
mentioned in the draft that other measurements (jitter, loss) could be 
supported later, but based on the current definitions (fixed fields, no 
other flags for indication), seems that it is difficut to extend to support 
other measurements, so why not consider using TLV-based mechanisms hence to 
make it more extensible?

Your draft proposes to use diag codes for indicating whether BFD express 
path function is enabled and/or the clock is synchronized to the same soure. 
>From the semantic of diag codes that are originaly designed for diagnotic 
purpose, I am not sure this is the right to go. By the way, why need to 
communicate whether clock is synchronized? IMHO, if someone wants to do 
one-way delay measurement, clock synchroniztion is the precondition, it 
should be done before the measurement.

As for the auth compatible issue, in our previous draft, it proposed to use 
TLV-based mechanism. And the A(Authentication) bit is re-used to indicate 
the existing of the performance monitoring TLVs and/or the Authentication 
TLVs, and then subsequent processing are based on the actual types of the 
TLVs, so no changes will be made to the Authenctiaction relevant processing 
and it is compatible. So if we could change the semantic of the A bit, for 
example: the bit is used to indicate whether Optional sections exist, it 
will make BFD more extensible.

Best regards,
Mach

---

From: Spencer Giacalone <[email-masked]>
Date: Sat, 27 Nov 2010 21:48:58 -0500

When using BFD Express Path in Echo mode or with the P bit set, it's
important to know that the remote system's response timestamps are
compatible with the ones being sent. We can do that with a bit setting
(in the control packet or elsewhere- see TLV discussion below), or by
making a sync protocol mandatory. I take it that you feel making it a
precondition is the way to go (and therefore the sync bit is
redundant). Is there further comment from the WG on whether it makes
sense to make a sync protocol a precondition from an implementation
standpoint?

I agree that the use of TLVs is really the right way to go for
extending BFD's functionality. Is there any opinion from the WG on
whether it's easier to change the purpose of the A bit to indicate
that generic TLVs follow, or whether it's easier to assign something
like a diag code for this purpose? It seems like either one would do,
and that it comes down to backwards compatibility. If most
implementations ignore unexpected auth extension formats/types when
the A bit is set, then that is a "cleaner" option from a protocol
design standpoint. If not, and most implementations ignore unknown
diag codes or unknown extension formats, then this should be easier to
code and to deploy in production (no wholesale, "big-bang" network
deployments required).

There is no reason why Express Path cannot use TLVs. However, since
TLVs have implications beyond Express Path, I think we should view
them and Express Path as separate, but complimentary.

Spence

---

From: David Ward <[email-masked]>
Date: Sun, 28 Nov 2010 08:19:29 -0800

Spencer -

You'd want to use the "A" bit to state that TLVs follow. If one is not passing auth information you'd have to carry some extra bits but, the A bit is the way to do it not in a diag code.

-DWard

---

From: Dave Katz <[email-masked]>
Date: Mon, 29 Nov 2010 12:10:20 -0800

There have been a number of extensions proposed for BFD over the years.  My standard response is always as follows:

Is this extension intimately associated with the core goal of BFD?  One can make the case that performance measurement is at least conceptually related to liveness detection, but I think the overlap is weak.

Does it make sense for the extension to be sent at 100pps or more?  If it doesn't make sense to send it in *every* packet, it probably doesn't belong in BFD.  I don't think that it is terribly useful to measure the path latency at 100Hz.

Is the extension's use of BFD as a transport impacted by the transmission patterns of BFD?  (Or conversely, does it impact it?)  BFD is a really crappy general-purpose transport, as it may have very bizarre latency characteristics as the transmit and receive intervals are changed, and if demand mode is in use, it shuts down altogether for indefinite periods of time.  It doesn't seem that path latency measurement is very compatible with the dynamics of BFD--if it *is* interesting to measure path latency at 100Hz, BFD may change that to 10Hz, or 0Hz, in order to satisfy its own needs.

Are there other protocols that would be a better fit, or should the extension be a standalone protocol by itself?  It sure looks like NTP already does everything that you need (and more), and it is already widely implemented in both hosts and routers.

As for the particulars of the proposed extensions, the attempted use of the Diag field changes its semantics.  Here's how the field is defined in RFC 5880:

  A diagnostic code specifying the local system's reason for the
  last change in session state.

The field is also unreliable as a signaling mechanism.  The following from RFC 5880 also gives a hint:

  Note that if the BFD session subsequently fails, the diagnostic code will be
  overwritten with a code detailing the cause of the failure.

Although it isn't fully explicated in the RFC (my bad, as this keeps coming up), from the standpoint of the protocol, the Diag field is a *write-only* field--there is no normative text that interprets the Diag field as received from the remote end of a session.  The primary intent of the field is so that the operator of the remote system can get an idea (via management) as to what is happening when the session goes down.  A secondary intent is to give the receiving *clients* some additional indication as to what is happening in order to improve the heuristics of the client/BFD interaction.

The primary takeaway is that the value of the Diag field cannot be relied upon for anything requiring determinism, because it will be overwritten at the drop of a hat.

As DaveW pointed out (perhaps more politely), when we defined BFD, we specifically made it hard to extend.  This was in part because we expected it to make its way into silicon and wanted to keep it as simple and stable as we could, and in part because we knew that folks would want to extend it in various ways with varying levels of appropriateness.  As DaveW has also pointed out, we did, however, leave the Echo packet format completely unspecified;  one of the first uses folks came up with for it was to do a basic form of latency measurement.  While using the Echo packet doesn't allow the remote system to add any timestamp information, it can be asserted that the processing time in the remote system is uninteresting, and if done in hardware is probably smaller than the order of magnitude of jitter resulting from having to specifically process the packet and add the timestamps (and ultimately all you do with that information is try to eliminate the remote-end processing time from the total round-trip time.)

While I see latency measurement as a quite reasonable thing to do, I don't see BFD as the place to do it.  Inevitably, it will end up compromising either BFD or the latency measurement function, as they will almost certainly end up tripping over one another.  "Because it's there" isn't a great reason to put this into BFD.  Assuming that you can't use NTP for some reason, and seeing as how there are other efforts going on in the IETF to achieve similar goals, and given that you're inventing a new protocol anyhow (new fields, new semantics) it would seem much cleaner to define a separate protocol that has the characteristics that you like about BFD (encapsulation independence, though it has thus far only found traction within IP;  whatever else) but does exactly what you need, and do the work in a WG that presumably has more expertise in this area.

My $.02...

--Dave

---

From: John E Drake <[email-masked]>
Date: Mon, 29 Nov 2010 13:03:23 -0800

Dave,

I couldn't agree more.  Also, there already is an working group I-D in MPLS defining a protocol for delay and loss measurement (http://datatracker.ietf.org/doc/draft-ietf-mpls-tp-loss-delay/).

Thanks,

John 

Sent from my iPhone

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Dave Katz
> Sent: Monday, November 29, 2010 12:10 PM
> To: Spencer Giacalone
> Cc: [email-masked]
> There have been a number of extensions proposed for BFD over the years.
> My standard response is always as follows:
> 
> 
> Is this extension intimately associated with the core goal of BFD?  One
> can make the case that performance measurement is at least conceptually
> related to liveness detection, but I think the overlap is weak.
> 
> Does it make sense for the extension to be sent at 100pps or more?  If
> it doesn't make sense to send it in *every* packet, it probably doesn't
> belong in BFD.  I don't think that it is terribly useful to measure the
> path latency at 100Hz.
> 
> Is the extension's use of BFD as a transport impacted by the
> transmission patterns of BFD?  (Or conversely, does it impact it?)  BFD
> is a really crappy general-purpose transport, as it may have very
> bizarre latency characteristics as the transmit and receive intervals
> are changed, and if demand mode is in use, it shuts down altogether for
> indefinite periods of time.  It doesn't seem that path latency
> measurement is very compatible with the dynamics of BFD--if it *is*
> interesting to measure path latency at 100Hz, BFD may change that to
> 10Hz, or 0Hz, in order to satisfy its own needs.
> 
> Are there other protocols that would be a better fit, or should the
> extension be a standalone protocol by itself?  It sure looks like NTP
> already does everything that you need (and more), and it is already
> widely implemented in both hosts and routers.
> 
> 
> As for the particulars of the proposed extensions, the attempted use of
> the Diag field changes its semantics.  Here's how the field is defined
> in RFC 5880:
> 
> Although it isn't fully explicated in the RFC (my bad, as this keeps
> coming up), from the standpoint of the protocol, the Diag field is a
> *write-only* field--there is no normative text that interprets the Diag
> field as received from the remote end of a session.  The primary intent
> of the field is so that the operator of the remote system can get an
> idea (via management) as to what is happening when the session goes
> down.  A secondary intent is to give the receiving *clients* some
> additional indication as to what is happening in order to improve the
> heuristics of the client/BFD interaction.
> 
> The primary takeaway is that the value of the Diag field cannot be
> relied upon for anything requiring determinism, because it will be
> overwritten at the drop of a hat.
> 
> 
> As DaveW pointed out (perhaps more politely), when we defined BFD, we
> specifically made it hard to extend.  This was in part because we
> expected it to make its way into silicon and wanted to keep it as
> simple and stable as we could, and in part because we knew that folks
> would want to extend it in various ways with varying levels of
> appropriateness.  As DaveW has also pointed out, we did, however, leave
> the Echo packet format completely unspecified;  one of the first uses
> folks came up with for it was to do a basic form of latency
> measurement.  While using the Echo packet doesn't allow the remote
> system to add any timestamp information, it can be asserted that the
> processing time in the remote system is uninteresting, and if done in
> hardware is probably smaller than the order of magnitude of jitter
> resulting from having to specifically process the packet and add the
> timestamps (and ultimately all you do with that information is try to
> eliminate the remote-end processing time from the total round-trip
> time.)
> 
> 
> While I see latency measurement as a quite reasonable thing to do, I
> don't see BFD as the place to do it.  Inevitably, it will end up
> compromising either BFD or the latency measurement function, as they
> will almost certainly end up tripping over one another.  "Because it's
> there" isn't a great reason to put this into BFD.  Assuming that you
> can't use NTP for some reason, and seeing as how there are other
> efforts going on in the IETF to achieve similar goals, and given that
> you're inventing a new protocol anyhow (new fields, new semantics) it
> would seem much cleaner to define a separate protocol that has the
> characteristics that you like about BFD (encapsulation independence,
> though it has thus far only found traction within IP;  whatever else)
> but does exactly what you need, and do the work in a WG that presumably
> has more expertise in this area.
> 
> My $.02...
> 
> --Dave
> 
> 
> 
> 
> > A new draft, called "BFD Express Path"
> > "In certain networks, such as, but not limited to, financial
> > monitoring information."

---

From: Spencer Giacalone <[email-masked]>
Date: Mon, 29 Nov 2010 16:25:16 -0500

As I have mentioned, we do not plan to run, nor do we need TP. We also
need link by link mon, and want to inject into a routing protocol.

Spence

---

From: John E Drake <[email-masked]>
Date: Mon, 29 Nov 2010 13:44:22 -0800

OAM MPLS-TP is not monolithic so you could deploy just the delay/loss measurement protocol, and it does measure links as well as LSPs.  Injecting this type of information into a routing protocol is probably covered by the Composite Link work in the RTG WG.   

Sent from my iPhone

> From: Spencer Giacalone [mailto:[email-masked]]
> Sent: Monday, November 29, 2010 1:25 PM
> To: John E Drake
> Cc: Dave Katz; [email-masked]
> > I couldn't agree more.  Also, there already is an working group I-D
> in MPLS defining a protocol for delay and loss measurement
> (http://datatracker.ietf.org/doc/draft-ietf-mpls-tp-loss-delay/).
> >
> > Thanks,
> >
> > John
> >
> > Sent from my iPhone
> >
> >
> >> The field is also unreliable as a signaling mechanism.  The
> following
> >> from RFC 5880 also gives a hint:
> >>
> >>   Note that if the BFD session subsequently fails, the diagnostic
> code
> >> will be

---

From: Spencer Giacalone <[email-masked]>
Date: Mon, 29 Nov 2010 17:29:29 -0500

Dave & Dave,

-What I propose may not be in line with the original intent, but I
don’t see why that in and of itself should limit innovation. The fact
that people keep wanting to extend seems to indicate demand and lost
opportunity

-As far as the points about control channel and state machine, would
using TLVs make that easier?

-Packet rates at 10ms could be valid issue, but I am not sure I agree
overall. The Juniper transmit interval range may be 1-255,000ms, but
as far as I know, other popular implementations set the minimum
transmit interval to 50ms. Perhaps we limit the min transmit interval
when Express Path is enabled or we tag every nth packet (but this
might negate loss monitoring). Maybe this should be implementation
specific?

-Transmits/receive intervals:  For latency monitoring, at an abstract
level, not sure it matters if the intervals are changed/jittered, so
long as they don't vary above a certain threshold. My main interest is
the send and receive timestamp diff per packet, not necessarily the
interval (for latency monitoring, anyway). Anyway, as you point out,
less might be "more".

-If demand mode is an issue, simple, don't include it in/permit it
with Express Path

-If diagnostic codes are an issue, simple, use the A bit, as mentioned

-NTP: I need link by link, not server to "client". Express Path needs
something like NTP. But, as far as I can see, NTP does not replace
Express Path.

-Echo mode is the least interesting mode for my application. Some
vendors don’t support it (at all), and it would double the overhead
for the same result (assuming you could make it work for my purpose).
However, you mentioned that people have tried to extend it for latency
monitoring. If so, what happened, and does that not further the
perceived client demand?

-Again, cannot use underlying transport level OAM- I need to verify
the underlying services, and want to inject that data into other
protocols. Something ubiquitous like BFD makes more sense for this,
and lets me do SLA verification. Not planning to run TP & have no
need. Also need link by link, not LSP level. Want to control setup,
not just do OAM.

Spence

---

From: Vishwas Manral <[email-masked]>
Date: Mon, 29 Nov 2010 14:35:19 -0800

Hi Spencer,

> Does that answer your question?
It does answer my question. My question however is why would an on
demand mode work, which could use Ping extensions for the purpose.
Though I agree Delay measurement could be done on a continuous basis
like in CCM packets (Layer-2 version of BFD).

> is unknown (I don’t see one)?
No the problem is that if the A bit is set the packet does not look
like the way you have mentioned it. The other issue is that the
current implementations will not look at BFD after the header you
mention and as there is no way to signal before using the extension,
you cannot use the same. Though I would have preferred some means of
doing extension for BFD, i notice that not even 1-bit remains which
can be overloaded.

However a new authentication type could be defined. This could contain
the information you have mentioned.

> perhaps someone can shed light on this.
I agree unknown diag code should be ignored on receipt, and guess that
is how implementations would behave too.

> WG concludes that is the way to go.
I prefer not using TLV's. It would be better to use bit fields as far
as possible and fixed location information. with TLV's packet needs to
be parsed completely to find some information does not exist.

> inclined to think 1-way delay is what is needed.
Echo mode packets are never defined. Are you saying this draft is
going to define it? If something can be done without peer intervention
I think we do not need to define it at all.

> deployment roadmap, as we do not require that functionality.
I am not stating that the network use MPLS-TP. However what I am
statis is that TP defines drafts that do delay measurements. You could
try to extrapolate data from that.

> I am happy to further this discussion. Let me know what you think,
I think a delay measurement tool could be added like in CCM, it
however may need to be more comprehensive. I think we should not make
too many changes to BFD and allow too many extensions. It will then
become fast path incapable.

Thanks,
Vishwas

---

From: Vishwas Manral <[email-masked]>
Date: Mon, 29 Nov 2010 14:47:52 -0800

Hi,

Just to add Echo mode is supported for only single hop sessions, so it
cannot be used for 2-way delay for multihop sessions.

Thanks,
Vishwas