Subject: Re: WGLC for draft-ietf-bfd-large-packets

From: "Reshad Rahman (rrahman)" <[email-masked]>
Date: Mon, 9 Sep 2019 14:55:44 +0000

BFD WG, reminder that WGLC is ongoing for this document.

Regards,
Reshad.

From: Rtg-bfd <[email-masked]> on behalf of "Reshad Rahman (rrahman)" <[email-masked]>
Date: Tuesday, August 27, 2019 at 12:34 PM
To: "[email-masked]" <[email-masked]>
Subject: WGLC for draft-ietf-bfd-large-packets

BFD WG,

As was mentioned at IETF105, this document is stable and there was an interop test done between FRR and Junos VMX.

Please provide comments/feedback on the document. The deadline for last call is September 13th.

Regards,
Reshad & Jeff.

---

From: <[email-masked]>
Date: Wed, 11 Sep 2019 11:33:45 +0800 (CST)

Hi Reshad,

I support this draft to be published, it provides a simple and elegant extension for BFD to verify path MTU.

By the way, if further we'd like to achieve a fine-grained process control while using BFD to verfiy or detect path MTU, with the cost of a more complex extention to BFD, then draft-mirmin-bfd-extended potentially can provide such a solution.

Best Regards,

Xiao Min

原始邮件

发件人：ReshadRahman(rrahman) <[email-masked]>
收件人：[email-masked] <[email-masked]>;
日 期 ：2019年09月09日 23:14
主 题 ：Re: WGLC for draft-ietf-bfd-large-packets

 

Regards,

Reshad.

 

BFD WG,

 

 

Regards,

Reshad & Jeff.

---

From: Mahesh Jethanandani <[email-masked]>
Date: Thu, 12 Sep 2019 11:32:27 -0700

I support the publication of this draft. It is useful, and addresses a problem in a network that we know has different MTU values.

> On Aug 27, 2019, at 9:33 AM, Reshad Rahman (rrahman) <[email-masked]> wrote:
> 
> BFD WG,
>  

Mahesh Jethanandani
[email-masked]

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 13 Sep 2019 02:56:02 +0000

Thanks for the reminder, Reshad. I support publication of this document, short and useful.

I only have one comment in regards to:

   It is also worthy of note that even if an implementation can function
   with larger transport PDUs, that additional packet size may have
   impact on BFD scaling.  Such systems may support a lower transmission
   interval (bfd.DesiredMinTxInterval) when operating in large packet
   mode.  This interval may depend on the size of the transport PDU.

Instead of (or in addition to) a lower transmission interval, why not add flexibility to not *have* to send large packets every packet, and instead send every n paks or so?

Thanks!

Carlos.

On Sep 9, 2019, at 10:55 AM, Reshad Rahman (rrahman) <[email-masked]<mailto:[email-masked]>> wrote:

Regards,
Reshad.

From: Rtg-bfd <[email-masked]<mailto:[email-masked]>> on behalf of "Reshad Rahman (rrahman)" <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, August 27, 2019 at 12:34 PM
To: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: WGLC for draft-ietf-bfd-large-packets

BFD WG,

Regards,
Reshad & Jeff.

---

From: "Ketan Talaulikar (ketant)" <[email-masked]>
Date: Fri, 13 Sep 2019 03:55:22 +0000

Hi All,

I would like to ask some questions and seek clarifications on this draft.

  1.  I am aware that this draft originates from practical pain points at a specific operator. During the adoption calls, the scenarios were debated in detail. It was basically a L2 WAN circuit service over a provider network and the challenge was that the PMTU of the underlying path in the SP network changes dynamically. However, from the Enterprise POV, the L2 circuit is seen as a single link and the BFD runs in directly connected mode. The draft however, discusses BFD multi-hop which is an entirely different use-case. When doing multi-hop, the BFD packet could go over entirely different paths in the network with different PMTUs (especially different from the application sending large packets) – this makes things flaky? So shouldn’t this mechanism be actually focussed on the single hop/directly connected mode instead?

  1.  There are implementations with BFD offload to H/W out there. What happens when a particular implementation cannot handle such large packet sizes (and I am not specifically aware of one such)? Like other aspects of monitoring like the intervals, would there be a value in indicating a support for large packets during the signalling? The draft does raise the need for it but doesn’t seem to do anything about it – why? Either we need it and this draft should define it. Or we don’t need it and it would placing the onus on the operator to enable this when they know both ends support it. Then it is something for operational consideration section perhaps?

  1.  There was a discussion on the list about whether this needs to be done for every packet or not. I don’t find that discussion or the result captured in the draft. The draft just says that perhaps longer intervals should be applied to BFD when doing large packets – but this will defeat the purpose of fast-detection. What would be good is we have both fast-detection and slow PMTU validation? Perhaps we need some analysis of whether large packets should be used always or intermittently and the pros/cons or guidelines for the same?

  1.  The draft is missing an operational considerations and manageability considerations sections. Some of this info is placed in sec 3, but would help if things were elaborated in their individual sections. It would provide more insight into how exactly this mechanism in BFD is envisaged to be actually deployed and used. More importantly, perhaps how it should NOT be used?

Can the authors and WG discuss the above? I think it seems too rushed to go for WGLC just as yet?

Thanks,
Ketan

From: Rtg-bfd <[email-masked]> On Behalf Of Reshad Rahman (rrahman)
Sent: 09 September 2019 20:26
To: [email-masked]
Subject: Re: WGLC for draft-ietf-bfd-large-packets

Regards,
Reshad.

BFD WG,

Regards,
Reshad & Jeff.

---

From: Mahesh Jethanandani <[email-masked]>
Date: Thu, 12 Sep 2019 21:13:39 -0700

Hi Carlos,

> On Sep 12, 2019, at 7:56 PM, Carlos Pignataro (cpignata) <[email-masked]> wrote:
> 

Wouldn’t that have a problem that one packet lost every n packets would still keep the BFD session up?

Cheers.

> 
> Thanks!
> 
> Carlos.
> 
>> On Sep 9, 2019, at 10:55 AM, Reshad Rahman (rrahman) <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> From: Rtg-bfd <[email-masked] <mailto:[email-masked]>> on behalf of "Reshad Rahman (rrahman)" <[email-masked] <mailto:[email-masked]>>
>> To: "[email-masked] <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 13 Sep 2019 04:21:11 +0000

Hi, Mahesh,

On Sep 13, 2019, at 12:13 AM, Mahesh Jethanandani <[email-masked]<mailto:[email-masked]>> wrote:

Hi Carlos,

On Sep 12, 2019, at 7:56 PM, Carlos Pignataro (cpignata) <[email-masked]<mailto:[email-masked]>> wrote:

Unless you also measure large-packets lost :-)

Thanks,

Carlos.

Cheers.

Thanks!

Carlos.

Regards,
Reshad.

BFD WG,

Regards,
Reshad & Jeff.

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 13 Sep 2019 08:10:04 -0400

Carlos,

> On Sep 12, 2019, at 10:56 PM, Carlos Pignataro (cpignata) <[email-masked]> wrote:
> 

Such a thing could be done.  A form of this sort of thing for "probing" was discussed in a prior draft I did with Xiao Min, but that was for Echo mode.

But since this in BFD async mode, the timers are likely chosen for the worst case.  So, if we have the usual Detect Multiplier of 3 and we can support sending a large packet every (e.g.) 25ms, and our implementation would normally send smaller packets every 10ms, we'd need to negotiate for some range between 10 and 25 knowing that the occasional large packet may itself perturb timing enough it might be counted as loss.

So, such a thing could be done by an implementation, but timing is messy.  If you want something a bit more occasional, I suggest a reasonable set of timers in async mode and more clever procedures in Echo mode, if you support that.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 13 Sep 2019 08:33:50 -0400

Ketan,

Thanks for your comments.

> On Sep 12, 2019, at 11:55 PM, Ketan Talaulikar (ketant) <[email-masked]> wrote:
> 
> Hi All,
>  
> 	• I am aware that this draft originates from practical pain points at a specific operator. During the adoption calls, the scenarios were debated in detail. It was basically a L2 WAN circuit service over a provider network and the challenge was that the PMTU of the underlying path in the SP network changes dynamically. However, from the Enterprise POV, the L2 circuit is seen as a single link and the BFD runs in directly connected mode. The draft however, discusses BFD multi-hop which is an entirely different use-case.

This is perhaps some poor wording choices in the Introduction section.  The intent is really that this is usable for all cases where BFD may be used.  The Introduction text was intended to note that for single-hop scenarios, BFD Echo procedures may be sufficient to solve the problem.  (See comment to Carlos elsewhere in the thread.)  However, for any form of multi-hop, you can't solve this via Echo.

General note about the "pain points at a specific author": While Albert certainly was the first customer I'd had willing to help stamp their name on such a draft, I'd been approached in my chair capacity both in IETF and at my employer over the years about similar headaches many times.  This is sadly not a rare problem.

> When doing multi-hop, the BFD packet could go over entirely different paths in the network with different PMTUs (especially different from the application sending large packets) – this makes things flaky? So shouldn’t this mechanism be actually focussed on the single hop/directly connected mode instead?

Even for single-hop this may be slightly flakey.  Consider "directly connected" links that are composed from LAGs.  

The draft doesn't try to completely address all such forms of multipath problems, even for directly connected solutions.  When we were doing BFD for LAGs, we realized that trying to get overly specific in terms of the implementation not only got ourselves perilously into the specifics of a given vendor, but potentially into conflict with IEEE in terms of discussions about distributing traffic.  To that end, we've been silent on the matter in this draft.

It's reasonable in any BFD implementation dealing with multipath issues to potentially interact with the load balancer and decide to exercise specific links.  I believe it's the case for some scenarios that your own employer may do this in pre BFD-on-LAG implementations for single hop BFD based on some old mailing list discussion.  However, it's proven unreasonable over the life of the BFD protocol to try to get too pedantic over how to do such a thing; some hardware may simply not support it.

>  
> 	• There are implementations with BFD offload to H/W out there. What happens when a particular implementation cannot handle such large packet sizes (and I am not specifically aware of one such)? Like other aspects of monitoring like the intervals, would there be a value in indicating a support for large packets during the signalling? The draft does raise the need for it but doesn’t seem to do anything about it – why? Either we need it and this draft should define it. Or we don’t need it and it would placing the onus on the operator to enable this when they know both ends support it. Then it is something for operational consideration section perhaps?

This is effectively a criticism of BFD in general rather than specifically this feature.  Part of the motivation for RFC 7419, Common Interval Support in Bidirectional Forwarding Detection, was to specify a number of common implementation timings.  But even then, that document doesn't talk much about scaling considerations for a given set of intervals.  Common considerations for implementors and operators tend to be "my implementation can support X sessions at 3.3ms, Y and 10ms, and Z much slower if we leave the fully distributed to the line card mode".  This just simply means the product sheet gets another axis in its listings.

What you're asking for in the protocol otherwise is a mechanism to negotiate or renegotiate a set of timers over a given set of interfaces based on existing or future re-provisioning of session quantity and parameters.  I think you'll find operators aren't particularly supportive of that.

FWIW, we expect exactly these same questions during IESG review.  BFD had exactly this type of question when doing even the base RFC 5880 work.  (BFD tends to make the current Transport AD unhappy.)

>  
> 	• There was a discussion on the list about whether this needs to be done for every packet or not. I don’t find that discussion or the result captured in the draft. The draft just says that perhaps longer intervals should be applied to BFD when doing large packets – but this will defeat the purpose of fast-detection. What would be good is we have both fast-detection and slow PMTU validation? Perhaps we need some analysis of whether large packets should be used always or intermittently and the pros/cons or guidelines for the same?

See my comment to Carlos in-thread about why this makes the timers tricky.  See also how BFD Echo can solve this problem in some circumstances without negotiation.

It's worth noting that BFD is quite capable of dynamically re-negotiating its timers.  This means an implementation could have a behavior such as:

1. Start by bringing up its session with timers for regular sized packets, and 3.3ms
2. Decide to shift to large packets, but renegotiate to 10ms timers first.

The "signaling" that a system is willing to accept the large packets is shifting to a longer timer.

>  
> 	• The draft is missing an operational considerations and manageability considerations sections. Some of this info is placed in sec 3, but would help if things were elaborated in their individual sections. It would provide more insight into how exactly this mechanism in BFD is envisaged to be actually deployed and used. More importantly, perhaps how it should NOT be used?

Is there specific information you think should go into such a section that isn't otherwise present in the draft?  The "how it should NOT" be used is a challenging section to write - somewhat similar to proving negatives - without specific content.

> Can the authors and WG discuss the above? I think it seems too rushed to go for WGLC just as yet?

The purpose of WGLC is to provide the working group an opportunity to drive out commentary like that above when things are otherwise quiet. :-)  Thanks for providing it.

-- Jeff

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 13 Sep 2019 15:48:44 +0000

Jeff,

Right. Or a burst of large packets and a set of bursts of “regular sized” ones. My main point is that I’d like the spec to allow for flexibility in usage if you think it makes sense, and not be all-or-nothing. 

Thanks!

Thumb typed by Carlos Pignataro.
Excuze typofraphicak errows

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 13 Sep 2019 14:08:02 -0400

Carlo,s

> On Sep 13, 2019, at 11:48 AM, Carlos Pignataro (cpignata) <[email-masked]> wrote:

No matter what, you'd have to accommodate for the timers.

If your timers will accommodate small and large within the same interval, go for it. If you need to renegotiate to do a periodic check, go for it.  BFD is permissive about such things. :-)

Even better, suggest text that makes you think such a thing is explicitly permitted rather than implicitly permitted.  I treat this as simply varying bfd.PaddedPduSize on an established session.  

-- Jeff

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Wed, 18 Sep 2019 05:24:17 +0000

A few apologies:

1)Sorry to be late in responding - but just back from vacation.
2)As a number of my comments overlap with Ketan's, I decided to respond to this email - apologies if that adds confusion.
3)I will top post my reply in the hopes it will more universally readable - apologies if the lack of context makes things harder to understand.

With that out of the way...

I support this draft - but I do not think it is ready for last call.

There are very legitimate concerns about the impact supporting padded BFD PDUs may have on existing hardware implementations - both functionally and in terms of scalability.
As a standards track document I think more needs to be said about operational considerations - and I think there may be legitimate reasons to consider negotiation of the capability.
In particular the statement in Section 3:

    "Additional changes to the base BFD   protocol may be required to permit negotiation of this functionality
         and the padding value."

If these protocol changes are to be made, shouldn't they be specified in this document?? Otherwise the document would seem just informational.

Section 3 also suggests 

      "support[ing] a lower transmission  interval (bfd.DesiredMinTxInterval)"

as a means of minimizing scalability impacts. But in early discussions of this draft it had been suggested that rather than use BFD for PathMTU validation we could use another protocol (e.g., IS-IS hellos) to do this. But that was rejected because it was stated that the deployment requirement required much faster detection. If that argument holds, then the suggestion in the draft to use longer timers seems inappropriate - or at least without sufficient context.

(BTW, by "lower transmission interval" I assume you mean a longer interval (i.e., a larger value for bfd.DesiredMinTxInterval)??)

I think these points need to be addressed in the draft before it is can be considered complete.

    Les

> -----Original Message-----
> From: Rtg-bfd <[email-masked]> On Behalf Of Jeffrey Haas
> Sent: Friday, September 13, 2019 5:34 AM
> To: Ketan Talaulikar (ketant) <[email-masked]>
> Cc: Reshad Rahman (rrahman) <[email-masked]>; [email-masked]
> > 	• I am aware that this draft originates from practical pain points at a
> specific operator. During the adoption calls, the scenarios were debated in
> detail. It was basically a L2 WAN circuit service over a provider network and
> the challenge was that the PMTU of the underlying path in the SP network
> changes dynamically. However, from the Enterprise POV, the L2 circuit is
> seen as a single link and the BFD runs in directly connected mode. The draft
> however, discusses BFD multi-hop which is an entirely different use-case.
> 
> This is perhaps some poor wording choices in the Introduction section.  The
> intent is really that this is usable for all cases where BFD may be used.  The
> Introduction text was intended to note that for single-hop scenarios, BFD
> Echo procedures may be sufficient to solve the problem.  (See comment to
> Carlos elsewhere in the thread.)  However, for any form of multi-hop, you
> can't solve this via Echo.
> 
> General note about the "pain points at a specific author": While Albert
> certainly was the first customer I'd had willing to help stamp their name on
> such a draft, I'd been approached in my chair capacity both in IETF and at my
> employer over the years about similar headaches many times.  This is sadly
> not a rare problem.
> 
> > When doing multi-hop, the BFD packet could go over entirely different
> paths in the network with different PMTUs (especially different from the
> application sending large packets) – this makes things flaky? So shouldn’t this
> mechanism be actually focussed on the single hop/directly connected mode
> instead?
> 
> The draft doesn't try to completely address all such forms of multipath
> problems, even for directly connected solutions.  When we were doing BFD
> for LAGs, we realized that trying to get overly specific in terms of the
> implementation not only got ourselves perilously into the specifics of a given
> vendor, but potentially into conflict with IEEE in terms of discussions about
> distributing traffic.  To that end, we've been silent on the matter in this draft.
> 
> It's reasonable in any BFD implementation dealing with multipath issues to
> potentially interact with the load balancer and decide to exercise specific
> links.  I believe it's the case for some scenarios that your own employer may
> do this in pre BFD-on-LAG implementations for single hop BFD based on
> some old mailing list discussion.  However, it's proven unreasonable over the
> life of the BFD protocol to try to get too pedantic over how to do such a thing;
> some hardware may simply not support it.
> 
> 
> >
> > 	• There are implementations with BFD offload to H/W out there.
> What happens when a particular implementation cannot handle such large
> packet sizes (and I am not specifically aware of one such)? Like other aspects
> of monitoring like the intervals, would there be a value in indicating a support
> for large packets during the signalling? The draft does raise the need for it
> but doesn’t seem to do anything about it – why? Either we need it and this
> draft should define it. Or we don’t need it and it would placing the onus on
> the operator to enable this when they know both ends support it. Then it is
> something for operational consideration section perhaps?
> 
> This is effectively a criticism of BFD in general rather than specifically this
> feature.  Part of the motivation for RFC 7419, Common Interval Support in
> Bidirectional Forwarding Detection, was to specify a number of common
> implementation timings.  But even then, that document doesn't talk much
> about scaling considerations for a given set of intervals.  Common
> considerations for implementors and operators tend to be "my
> implementation can support X sessions at 3.3ms, Y and 10ms, and Z much
> slower if we leave the fully distributed to the line card mode".  This just
> simply means the product sheet gets another axis in its listings.
> 
> What you're asking for in the protocol otherwise is a mechanism to negotiate
> or renegotiate a set of timers over a given set of interfaces based on existing
> or future re-provisioning of session quantity and parameters.  I think you'll
> find operators aren't particularly supportive of that.
> 
> FWIW, we expect exactly these same questions during IESG review.  BFD had
> exactly this type of question when doing even the base RFC 5880 work.  (BFD
> tends to make the current Transport AD unhappy.)
> 
> 
> >
> > 	• There was a discussion on the list about whether this needs to be
> done for every packet or not. I don’t find that discussion or the result
> captured in the draft. The draft just says that perhaps longer intervals should
> be applied to BFD when doing large packets – but this will defeat the purpose
> of fast-detection. What would be good is we have both fast-detection and
> slow PMTU validation? Perhaps we need some analysis of whether large
> packets should be used always or intermittently and the pros/cons or
> guidelines for the same?
> 
> See my comment to Carlos in-thread about why this makes the timers tricky.
> See also how BFD Echo can solve this problem in some circumstances without
> negotiation.
> 
> It's worth noting that BFD is quite capable of dynamically re-negotiating its
> timers.  This means an implementation could have a behavior such as:
> 
> > 	• The draft is missing an operational considerations and
> manageability considerations sections. Some of this info is placed in sec 3, but
> would help if things were elaborated in their individual sections. It would
> provide more insight into how exactly this mechanism in BFD is envisaged to
> be actually deployed and used. More importantly, perhaps how it should
> NOT be used?
> 
> Is there specific information you think should go into such a section that isn't
> otherwise present in the draft?  The "how it should NOT" be used is a
> challenging section to write - somewhat similar to proving negatives - without
> specific content.
> 
> >
> The purpose of WGLC is to provide the working group an opportunity to drive
> out commentary like that above when things are otherwise quiet. :-)  Thanks
> for providing it.
> 
> -- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 18 Sep 2019 11:28:17 -0400

Les,

> 1)Sorry to be late in responding - but just back from vacation.

I wouldn't know anything about that sort of problem. :-)

> If these protocol changes are to be made, shouldn't they be specified in this document?? Otherwise the document would seem just informational.

No.  There's not really any room in BFD v1 for negotiation.  That would take
something like BFDv2, as we've started discussing in the working group.

Your point about the document being potentially informational is certainly
possible, albeit probably not what you're really intending.  As
implementation has already shown us, one side can simply start using this
procedure without any support required on the other side.  I.e. there's no
protocol changes we're doing.

> as a means of minimizing scalability impacts. But in early discussions of this draft it had been suggested that rather than use BFD for PathMTU validation we could use another protocol (e.g., IS-IS…

More context could be added, if there's appropriate text to add.
Suggestions are welcome.

However, this isn't particularly different from any other BFD considerations
we've had over the years across all uses of BFD.  Your scale for sessions
vs. timers will depend on transport, whether it's distributed to the line
card or not, hardware support vs. general purpose CPU, etc.  BFD
implementors and users expect to do some level of tuning.   This is normal.

What you otherwise seem to be implying is that some operational forum should
mandate a profile for number of sessions vs. timers vs. MTU size vs.
transport on a particular set of hardware.  And even then, that's not
impacting on this document's procedures.

> (BTW, by "lower transmission interval" I assume you mean a longer interval (i.e., a larger value for bfd.DesiredMinTxInterval)??)

Yes.

-- Jeff

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Thu, 19 Sep 2019 00:06:13 +0000

Jeff -

> From: Jeffrey Haas <[email-masked]>
> Sent: Wednesday, September 18, 2019 8:28 AM
> To: Les Ginsberg (ginsberg) <[email-masked]>
> Cc: Ketan Talaulikar (ketant) <[email-masked]>; Reshad Rahman
> (rrahman) <[email-masked]>; [email-masked]

[Les:] Sounds like something for you to discuss with HR. 😊

> 
> > There are very legitimate concerns about the impact supporting padded
> BFD PDUs may have on existing hardware implementations - both
> functionally and in terms of scalability.
> > As a standards track document I think more needs to be said about
> operational considerations - and I think there may be legitimate reasons to
> consider negotiation of the capability.
> > If these protocol changes are to be made, shouldn't they be specified in
> this document?? Otherwise the document would seem just informational.
> 

[Les:] I am not fully comfortable with the notion that it is OK to deploy this w/o regard for whether existing implementations can handle it. I think there is some risk here and I would like the WG to discuss this point and - if agreed to - have the draft explicitly state that the risks of deploying this w/o negotiation have been considered and deemed acceptable. The fact that one implementation had no issue with it isn't compelling.

> protocol changes we're doing.

[Les:] Understood - but I am still concerned.

> > as a means of minimizing scalability impacts. But in early discussions of this
> draft it had been suggested that rather than use BFD for PathMTU validation
> we could use another protocol (e.g., IS-IS hellos) to do this. But that was
> rejected because it was stated that the deployment requirement required
> much faster detection. If that argument holds, then the suggestion in the
> draft to use longer timers seems inappropriate - or at least without sufficient
> context.
> 

[Les:] The point I am highlighting is that we already have (at least for some deployments) a way to detect this within a scale of several seconds (or at least 10s of seconds).
If the argument is that this is not fast enough, what is fast enough? The conclusion I drew from the earlier discussion was that existing detection times for BFD were what was wanted. But the text in the draft suggests that the addition of MTU checking might justify accepting a slower  detection time. I don’t think this point has been explicitly discussed in the WG. For me, if slower detection is acceptable then I would like to reopen the discussion as to why other detection mechanisms are not acceptable. What is the real requirement here?

> impacting on this document's procedures.
[Les:] Absolutely not!! For the record, I am not a fan of RFC 7419 - please do not put me in that camp.
But I do think we need to clearly understand the desired behavior for this extension. Based on WG discussion I did not think that slower detection time was a goal.
 
    Les

> > (BTW, by "lower transmission interval" I assume you mean a longer interval
> (i.e., a larger value for bfd.DesiredMinTxInterval)??)
> 
> Yes.
>

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 18 Sep 2019 22:01:28 -0400

Les,

> [Les:] I am not fully comfortable with the notion that it is OK to deploy this w/o regard for whether existing implementations can handle it. I think there is some risk here and I would like the WG…

FWIW, I'm not blowing off your considerations.  Mostly, I'm highlighting
that we get them regularly.  TSV review of BFD in every flavor over the
entire life of the protocol has been some variety of this conversation.  We
just get to replay the entire life of BFD RFCs within such considerations
with every new TSV area director. :-)

Obviously it's up to Reshad as the shepherding chair how this process plays
forward, but I suspect it'll resemble:
- Getting further consensus from the rest of the WG whether this
  consideration is a sticking point.
- If it is, do we force this to be deferred to BFD v2?
- Alternatively, can we cleverly figure out some way to wedge negotiation on
  top of the existing feature in a backward compatible way?  (My answer:
  possibly.  Whether it's worth the complexity bump is the arguable component
  since simplicity of the protocol has been a stronger driving factor for
  implementors than scaling considerations.)

I'd like to hear from other implementors whether they share your concerns.
But either way, this exact conversation will play out again with the TSV ADs
during IESG review.  This is expected.

> [Les:] The point I am highlighting is that we already have (at least for some deployments) a way to detect this within a scale of several seconds (or at least 10s of seconds).

You're over-simplifying your point slightly.  That's true for one protocol,
IS-IS, where the padding can be done.  For those that can benefit from that
scenario, great!  But I suspect you'd agree that not everyone is going to
enable IS-IS on a link just to test MTU liveness.

> If the argument is that this is not fast enough, what is fast enough?

Much as you aren't a fan of RFC 7419, that's exactly why that document
existed.  Some level of commonality was desired in order to develop more
uniform scaling comparisons.

It's entirely possible that some implementations will take zero hit to run
BFD at exactly the same wire rate.  A BFD PDU every 3.3ms is not that much
in the way of pps.  And, if your forwarding silicon does its job of
isolating IP validation in the pipeline before punting to the BFD code, BFD
itself may not take any hit in terms of cycles.  However, if you're software
only, you may now be congesting a host path.

We already run into flavors of these considerations for regular 5880 BFD
over various transports.  The timer and session granularities vary.  No one
freaks out at this.

>  The conclusion I drew from the earlier discussion was that existing detection times for BFD were what was wanted. But the text in the draft suggests that the addition of MTU checking might justify accepting a slower  detection time.

I'm going to simply point out that such considerations apply to built-in BFD
features such as authentication.  If you enable some classes of
authentication, you've impacted your detection interval.

I'd like to know why you think that this feature is any scarier than the
impact of authentication.

-- Jeff

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Thu, 19 Sep 2019 02:37:10 +0000

Jeff -

First I would like to reemphasize that I support the draft - so we aren't on opposite sides here. It is just that Last Call seems premature.

As for the comparison with authentication...

Authentication added new functionality which cost CPU time. The tradeoff there was clear - performance/scale vs security. But there was no concern that the addition of the authentication bytes in and of itself might introduce a problem.

Large packets introduces MTU sized packets - which in and of itself is unlikely to cause a performance issue. But, having spent a fair number of hours debugging MTU related issues of various flavors, I do think it is likely to expose bugs in packet processing related to size. It shouldn't - in a perfect world - but what chips/software does with sub-MTU sized packets doesn't always translate to MTU size packets. And since the definition of what MTU is isn't consistent across vendors (let alone even within a single vendor's products) there are many ways to screw up configuration here. Throw in all the various flavors of encaps...I think we can expect deployment issues - and maybe bugs as well.

Just my thoughts...

   Les

> Sent: Wednesday, September 18, 2019 7:01 PM
> > [Les:] I am not fully comfortable with the notion that it is OK to deploy this
> w/o regard for whether existing implementations can handle it. I think there
> is some risk here and I would like the WG to discuss this point and - if agreed
> to - have the draft explicitly state that the risks of deploying this w/o
> negotiation have been considered and deemed acceptable. The fact that one
> implementation had no issue with it isn't compelling.
> 
> > [Les:] The point I am highlighting is that we already have (at least for some
> deployments) a way to detect this within a scale of several seconds (or at
> least 10s of seconds).
> 
> >  The conclusion I drew from the earlier discussion was that existing
> detection times for BFD were what was wanted. But the text in the draft
> suggests that the addition of MTU checking might justify accepting a slower
> detection time.
>

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 19 Sep 2019 08:16:41 -0400

Les,

> On Sep 18, 2019, at 10:37 PM, Les Ginsberg (ginsberg) <[email-masked]> wrote:
> 

The purpose of WGLC is to shake out final comments when things have otherwise stalled.  If it's not ready, we're not bothered by that. :-)

That said, we need to match concerns with something actionable.  That's what I'm hunting for.

> Authentication added new functionality which cost CPU time. The tradeoff there was clear - performance/scale vs security. But there was no concern that the addition of the authentication bytes in a…

Most of your commentary was focused around the impact on detection time, hence the response you got regarding other features that have similar impact.

So, if we've moved on from detection time impact of features to other issues, that's fine.

> Large packets introduces MTU sized packets - which in and of itself is unlikely to cause a performance issue. But, having spent a fair number of hours debugging MTU related issues of various flavor…

I think Albert is happy to see this text. :-)

The primary purpose of this feature is to shake loose issues related to MTU sized packets and try to remove such paths from service when we can't forward through them.  The path detection at speed is a general good fit for BFD.

What I'm concerned about is you're trying to point out "when there are issues, this doesn't help - and at best exacerbates them".  If so... well, this feature isn't intended to be a debugging layer for those sort of issues.  However, since BFD is riding on top of various transports to do this job, if the encapsulation can carry additional information that permits such debugging information, we're amenable to discussing what to do with it.

-- Jeff

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Tue, 24 Sep 2019 22:48:51 +0000

Jeff -

A few more thoughts - maybe these are more helpful than my previous comments - maybe not. I am sure you will let me know.

Protocol extensions allowing negotiation and/or advertisement of support for larger PDUs may well be useful - but let's agree that it is desirable to deploy this without protocol extensions just to keep the interoperability bar low.

My primary angst is with the following paragraph in Section 3:

"It is also worthy of note that even if an implementation can function
   with larger transport PDUs, that additional packet size may have
   impact on BFD scaling.  Such systems may support a lower transmission
   interval (bfd.DesiredMinTxInterval) when operating in large packet
   mode.  This interval may depend on the size of the transport PDU."

Given long experience that CPU use correlates more highly with number of packets than with number of bytes, the first sentence would seem to be weakly supported.
Given the previously mentioned concerns about detection time, the second sentence seems to compromise the value of the extension.

What might be better?

1)Some statement that MTU isn't necessarily a consistent value for all systems connected to an interface - which can impact the results when large BFD packets are used. Implementations might then want to consider supporting "bfd-mtu" configuration and/or iterating across a range of packet sizes to determine what works and what doesn't.

2)Use of both padded and unpadded packets in combination with draft-ietf-bfd-stability to determine whether a BFD failure is due to padding or a generic forwarding failure.

Either of these suggestions are really "diagnostic modes" which may help diagnose a problem but aren't meant to be used continuously as part of fast failure detection.

   Les

> Sent: Thursday, September 19, 2019 5:17 AM
> > First I would like to reemphasize that I support the draft - so we aren't on
> opposite sides here. It is just that Last Call seems premature.
> 
> The purpose of WGLC is to shake out final comments when things have
> otherwise stalled.  If it's not ready, we're not bothered by that. :-)
> 
> > Authentication added new functionality which cost CPU time. The tradeoff
> there was clear - performance/scale vs security. But there was no concern
> that the addition of the authentication bytes in and of itself might introduce a
> problem.
> 
> Most of your commentary was focused around the impact on detection time,
> hence the response you got regarding other features that have similar
> impact.
> 
> > Large packets introduces MTU sized packets - which in and of itself is
> unlikely to cause a performance issue. But, having spent a fair number of
> hours debugging MTU related issues of various flavors, I do think it is likely to
> expose bugs in packet processing related to size. It shouldn't - in a perfect
> world - but what chips/software does with sub-MTU sized packets doesn't
> always translate to MTU size packets. And since the definition of what MTU is
> isn't consistent across vendors (let alone even within a single vendor's
> products) there are many ways to screw up configuration here. Throw in all
> the various flavors of encaps...I think we can expect deployment issues - and
> maybe bugs as well.
> 
> The primary purpose of this feature is to shake loose issues related to MTU
> sized packets and try to remove such paths from service when we can't
> forward through them.  The path detection at speed is a general good fit for
> BFD.
> 
> What I'm concerned about is you're trying to point out "when there are
> issues, this doesn't help - and at best exacerbates them".  If so... well, this
> feature isn't intended to be a debugging layer for those sort of issues.
> However, since BFD is riding on top of various transports to do this job, if the
> encapsulation can carry additional information that permits such debugging
> information, we're amenable to discussing what to do with it.
> 
> -- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Thu, 26 Sep 2019 15:49:48 -0400

Les,

> Given the previously mentioned concerns about detection time, the second sentence seems to compromise the value of the extension.

My experience is largely identical to yours.

The motivation for mentioning anything at all here is TANSTAAFL[1], and
we've already had people ask about possible impacts.  And, as we discussed
previously in the thread we shall inevitably get asked about it during TSV
review in IESG.

The primary reason this is a "may" in the non-RFC 2119 sense is that our
experience also suggests that when the scaling impacts are primarily pps
rather than bps that this feature will likely have no major impact on
implementations beyond your valid concerns about exercising bugs.

I suspect had this not been mentioned at all, you would have been happier.
But you're not the target audience for this weak caveat.

> 1)Some statement that MTU isn't necessarily a consistent value for all systems connected to an interface - which can impact the results when large BFD packets are used. Implementations might then w…

I'm not clear what you intend by this statement.

Are you asking that we emphasize the use case in a different way?  The
Introduction currently states: 
  "However,
   some applications may require that the Path MTU [RFC1191] between
   those two systems meets a certain minimum criteria.  When the Path
   MTU decreases below the minimum threshold, those applications may
   wish to consider the path unusable."

I'm also unclear what "Implementations" may refer to here.  BFD?  An
arbitrary user application?  If the latter, the application may not have
strict control over the generation of a given PDU size; e.g. TCP
applications.

> Either of these suggestions are really "diagnostic modes" which may help diagnose a problem but aren't meant to be used continuously as part of fast failure detection.

We could certainly add a paragraph or two as an application note about using
this for BFD stability purposes as well.

-- Jeff

---

From: "Les Ginsberg (ginsberg)" <[email-masked]>
Date: Fri, 27 Sep 2019 21:14:08 +0000

Jeff -

Inline.

> Sent: Thursday, September 26, 2019 12:50 PM
> > A few more thoughts - maybe these are more helpful than my previous
> comments - maybe not. I am sure you will let me know.
> >
> > Protocol extensions allowing negotiation and/or advertisement of support
> for larger PDUs may well be useful - but let's agree that it is desirable to
> deploy this without protocol extensions just to keep the interoperability bar
> low.
> >
> > Given long experience that CPU use correlates more highly with number of
> packets than with number of bytes, the first sentence would seem to be
> weakly supported.
> > Given the previously mentioned concerns about detection time, the
> second sentence seems to compromise the value of the extension.
> 

[Les:] I am not opposed to a discussion of potential issues in the draft - rather I am encouraging it. But the current text isn't really on the mark as far as potential issues - and we seem to agree on that. It also suggests lengthening detection time to compensate - which I think is not at all what you want to suggest as it diminishes the value of the extension. It also isn't likely to address a real problem.

For me, the potential issues are:

a)Some BFD implementations might not be able to handle MTU sized BFD packets - not because of performance - but because they did not expect BFD packets to be full size and therefore might have issues passing a large packet through the local processing engine.

b)Accepted MTU is impacted by encapsulations and what layer is being considered (L2 or L3). And oftentimes link MTUs do not match on both ends ("shudder"), so you might end up with unidirectional connectivity. I appreciate that this is exactly the problem that the extensions are designed to detect. I am just asking that these issues be discussed more explicitly as an aid to the implementor. If that also makes Transports ADs happier that is a side benefit - but that's not my motivation.

> > What might be better?
> >
> > 1)Some statement that MTU isn't necessarily a consistent value for all
> systems connected to an interface - which can impact the results when large
> BFD packets are used. Implementations might then want to consider
> supporting "bfd-mtu" configuration and/or iterating across a range of packet
> sizes to determine what works and what doesn't.
> 

[Les:] I am talking about BFD implementations.
I suppose one can imagine each BFD client requesting a certain MTU value - but that wouldn't be my choice.
I would think the value we want is really the maximum L3 payload that the link is intended to support - which should be independent of the BFD client. This might be larger than any client actually uses - but that seems like a good thing.

   Les

> > 2)Use of both padded and unpadded packets in combination with draft-
> ietf-bfd-stability to determine whether a BFD failure is due to padding or a
> generic forwarding failure.
> >
> > Either of these suggestions are really "diagnostic modes" which may help
> diagnose a problem but aren't meant to be used continuously as part of fast
> failure detection.
>

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 28 Sep 2019 22:04:39 +0200

Hi Jeff,

Imagine two scenarios which were already highlighted as justification for
this work:

*Scenario 1 -* IGP with nodes interconnected with ECMP links

*Scenario 2 -* IGP nodes interconnected with L2 emulated circuits which in
turn are riding on telco IP network with ECMPs or LAGs.

*Questions Ad 1 - *

Is the idea to use in those cases "ECMP-Aware BFD for LDP LSPs" vendor's
feature to be able to detect MTU issues on any of the L3 paths ? Is there
feature extension to accomplish the same without LDP just when using ECMP
with OSPF ?

How do you solve this when there is L2 LAG hashing across N links enabled ?

*Question Ad 2 - *

How do you detect it if your L2 circuit provider maps BFD flows to one
underlay path and some encapsulated data packets is hashed to traverse the
other path(s) ? Clearly running multiple BFD sessions is not going to help
much in this scenario .... For example if someone is using v6 flow label it
may be directly copied to the outer service header.

Many thx,
Robert.

On Thu, Sep 26, 2019 at 9:46 PM Jeffrey Haas <[email-masked]> wrote:

> for larger PDUs may well be useful - but let's agree that it is desirable
> to deploy this without protocol extensions just to keep the
> interoperability bar low.
> >
> > Given the previously mentioned concerns about detection time, the second
> sentence seems to compromise the value of the extension.
>
> supporting "bfd-mtu" configuration and/or iterating across a range of
> packet sizes to determine what works and what doesn't.
>
> > 2)Use of both padded and unpadded packets in combination with
> draft-ietf-bfd-stability to determine whether a BFD failure is due to
> padding or a generic forwarding failure.
> >