Subject: [GROW] Handling of LAGs in Mitigating Negative Impact of Maintenance through BGP Session Culling

From: Thomas King <[email-masked]>
Date: Tue, 9 Jan 2018 11:30:35 +0000

Hi all,

we at DE-CIX are currently implementing BGP Session Culling and we hit the question how to handle LAGs (e.g. LACP, Static configured). The Internet Draft is not covering this question yet, however, from our point of view it is worth discussing it.

Our suggestion for handling LAGs looks like this:
Typically, a minimum number of port members can be defined for a LAG being up. The LAG is not touched by BGP Session Culling during a maintenance unless this number is undercut. If the number if undercut the LAG is handled by BGP Session Culling as defined in the Internet Draft.

If no value for the minimum number of active port members is defined for a LAG, the value 1 should be used as this is the behaviour of LAGs today already.

Best regards,
Thomas

---

From: Gert Doering <[email-masked]>
Date: Tue, 9 Jan 2018 12:34:10 +0100

Hi,

> Typically, a minimum number of port members can be defined for a LAG being up. The LAG is not touched by BGP Session Culling during a maintenance unless this number is undercut. If the number if un…

This assumes that this is a LAG of which only *some* of the ports would
be affected by a planned maintenance, right?

Like, customer has 4x 10GE, and you reboot one of the switches that 
has 2 of these, so 20 GE remain and the customer traffic could just use
these, with reduced bandwidth.

So, if my reading of the scenario is right, your proposal makes sense -
let customers decide what minimum bandwidth they need, and if the "4x 10GE"
customer says "less than 30GE is not useful for me!", cull his sessions
as well, even if he could go on.

thanks for bringing this up,

Gert Doering
        -- NetMaster
-- 
have you enabled IPv6 on something today...?

SpaceNet AG                        Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14          Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen                   HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444           USt-IdNr.: DE813185279

---

From: Job Snijders <[email-masked]>
Date: Tue, 9 Jan 2018 11:35:07 +0000

Dear Thomas,

On Tue, Jan 09, 2018 at 11:30:35AM +0000, Thomas King wrote:
> we at DE-CIX are currently implementing BGP Session Culling and we hit
> the question how to handle LAGs (e.g. LACP, Static configured). The
> Internet Draft is not covering this question yet, however, from our
> point of view it is worth discussing it.
> 
> Our suggestion for handling LAGs looks like this: Typically, a minimum
> number of port members can be defined for a LAG being up. The LAG is
> not touched by BGP Session Culling during a maintenance unless this
> number is undercut. If the number if undercut the LAG is handled by
> BGP Session Culling as defined in the Internet Draft.
> 
> If no value for the minimum number of active port members is defined
> for a LAG, the value 1 should be used as this is the behaviour of LAGs
> today already.

Is this in context of multi-chassis LAG?

Kind regards,

Job

---

From: "Will Hargrave" <[email-masked]>
Date: Tue, 09 Jan 2018 15:34:46 +0000

> Is this in context of multi-chassis LAG?

I think if we include anything about LAGs we should make it very clear 
that you must apply the culling ACL to *either* all ports of a LAG *or* 
none. Applying it to half of an MCLAG could be disastrous.

I didn’t realise there were IXPs using MC-LAG. Discovering this maybe 
surprise some members.

-- 
Will Hargrave
Technical Director
LONAP Ltd
+44 114 303 4444

---

From: Job Snijders <[email-masked]>
Date: Tue, 9 Jan 2018 15:50:39 +0000

On Tue, Jan 09, 2018 at 03:34:46PM +0000, Will Hargrave wrote:
> > > Our suggestion for handling LAGs looks like this: Typically, a
> > > minimum number of port members can be defined for a LAG being up.
> > > The LAG is not touched by BGP Session Culling during a maintenance
> > > unless this number is undercut. If the number if undercut the LAG
> > > is handled by BGP Session Culling as defined in the Internet
> > > Draft.
> > > 
> > > If no value for the minimum number of active port members is
> > > defined for a LAG, the value 1 should be used as this is the
> > > behaviour of LAGs today already.
> >
> that you must apply the culling ACL to *either* all ports of a LAG
> *or* none.  Applying it to half of an MCLAG could be disastrous.

Will, my reading of Thomas' message is slightly different, I don't think
he is proposing to apply culling ACLs on half the ports of a LAG, he
proposes that culling ACLs are only applied (to all ports) when more
than X members of a LAG will be impacted by the maintenance (where X is
an ixp-participant configurable number).

> I didn’t realise there were IXPs using MC-LAG. Discovering this maybe

Thomas, in terms of IETF logistics, the RFC-To-Be draft-ietf-grow-bgp-session-culling
document has already been submitted to the RFC Editor and is in the
queue, information on LAGs cannot be added at this point to the
publication.

However, since this is a BCP, the next iteration could include
additional information as our understanding of the culling practise
improves, and the BCP number wouldn't change of course.

>From what I read in your message your organisation is still in the early
phases of applying the 'culling' mechanism. I recommend you to (over
time) carefully take notes of the interaction between LAG and culling,
and whatever arises as the best current practise is documented in the
next revision of the BCP.

Kind regards,

Job

---

From: Joel Jaeggli <[email-masked]>
Date: Tue, 9 Jan 2018 08:33:07 -0800

Sent from my iPhone

> On Jan 9, 2018, at 07:50, Job Snijders <[email-masked]> wrote:
> 

This sounds like an implementation specific detail to me. I can think of several variants on  selecting which bgp sessions to cull based on criterion that might be exchange specific. Such as when inter-switch/site bandwidth drops below a certain threshold cull peers accordingly. 

I’m not sure that I would recommend any of these however the specifics of a service offering are up to an operator.

> an ixp-participant configurable number).

The same approach applies across multiple line cards in a large switch, or indeed might involve the loss of ports to which the client is not directly connected.

> _______________________________________________
> GROW mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/grow
>

---

From: Thomas King <[email-masked]>
Date: Thu, 11 Jan 2018 21:26:13 +0000

Hi Job,

On 09.01.18, 12:35, "Job Snijders" <[email-masked]> wrote:

    Dear Thomas,

This is in the context of single- and multi-chassis LAGs.

Best regards
Thomas

---

From: Thomas King <[email-masked]>
Date: Thu, 11 Jan 2018 21:27:23 +0000

Hi Will,

I know of many IXPs utilising Multi-Chassis LAGs.

Best regards,
Thomas

From: Will Hargrave <[email-masked]>
Date: Tuesday, 9. January 2018 at 16:34
To: Job Snijders <[email-masked]>
Cc: Thomas King <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [GROW] Handling of LAGs in Mitigating Negative Impact of Maintenance through BGP Session Culling

I think if we include anything about LAGs we should make it very clear that you must apply the culling ACL to either all ports of a LAG or none. Applying it to half of an MCLAG could be disastrous.

I didn’t realise there were IXPs using MC-LAG. Discovering this maybe surprise some members.

--
Will Hargrave
Technical Director
LONAP Ltd
+44 114 303 4444

---

From: Thomas King <[email-masked]>
Date: Thu, 11 Jan 2018 21:34:06 +0000

Hi Job,

On 09.01.18, 16:50, "Job Snijders" <[email-masked]> wrote:

This is what I meant!

Really, my understanding is it could be added. It is not typical but doable.

From my point of view the culling mechanism is very import for LAGs and should be clearly defined in order to be useful und comprehensive. As we see more and more IXP peers moving to LAGs (n times 10GE and also n times 100GE) this topic is already very important. It will gain more importance in the future as traffic volumes increase.

Best regards,
Thomas

---

From: Thomas King <[email-masked]>
Date: Mon, 15 Jan 2018 18:11:26 +0000

Hi guys,

any update on this?

Best regards,
Thomas

On 11.01.18, 22:34, "GROW on behalf of Thomas King" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Job,

        
    
    This is what I meant!

    

    
    
    Best regards,
    Thomas

---

From: Job Snijders <[email-masked]>
Date: Mon, 15 Jan 2018 18:34:09 +0000

> any update on this?

Hi Thomas,

I hope you realize that pulling the culling document out of the RFC
editor queue, and going again through GROW, IETF and IESG review, is a
very significant amount of work you'd be dumping in our laps. I'm not
sure I'd agree to that. :-)

In my January 9th message https://mailarchive.ietf.org/arch/msg/grow/1H30kxE7e1QN_5bRi3EfOoX9d-U
I mentioned that perhaps a next revision of the BCP can include text
specific to Multi-Chassis LAGs. You are free to propose text for the
next revision, you have not done so yet. I recommend that any proposed
text is based on actual operational experience with the combination of
'culling' and 'MC LAG'.

I indicated that your organisation appears to be in the early stages of
applying the 'culling' mechanism. I recommend you to carefully take
notes on the interaction between LAG and culling, and whatever arises as
the best practise, can maybe be documented in the next revision of the BCP.

Kind regards,

Job

---

From: Thomas King <[email-masked]>
Date: Tue, 16 Jan 2018 14:28:13 +0000

Hi Job,

to be clear, my suggestion covers MC-LAGs and more importantly LAGs. Both is missing in the current document. I think the working group should decide if the current document is good to go or whether it is missing important parts and should be revised.

If we as a working group decide to move ahead and add the proposed changes on an update of the BCP I will definitely contribute.

Best regards,
Thomas

On 15.01.18, 19:34, "Job Snijders" <[email-masked]> wrote:

    Hi Thomas,

    
    
    Kind regards,
    
    Job

---

From: "Will Hargrave" <[email-masked]>
Date: Tue, 16 Jan 2018 15:27:44 +0000

Hello,

We can always think of enhancements and this is part of the natural 
process of documenting best practice. At some point (some time ago) a 
consensus was reached, and we moved forward to publication. Without such 
a watershed no work can actually ever be completed.

We don’t have a concept of ‘minimum links’; when the member buys 
the ports they expect them all to work :) Selecting such a figure 
implies an understanding of their network that I don’t possess without 
further technical communication. This technique is designed to help with 
those communication difficulties not increase their complexity.

Will

On 16 Jan 2018, at 14:28, Thomas King wrote:

> Hi Job,
>
> to be clear, my suggestion covers MC-LAGs and more importantly LAGs. 
> Both is missing in the current document. I think the working group 
> should decide if the current document is good to go or whether it is 
> missing important parts and should be revised.
>
> If we as a working group decide to move ahead and add the proposed 
> changes on an update of the BCP I will definitely contribute.
>
> Best regards,
> Thomas
>
>
>     In my January 9th message 
> https://mailarchive.ietf.org/arch/msg/grow/1H30kxE7e1QN_5bRi3EfOoX9d-U

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 16 Jan 2018 15:40:16 +0000

> to be clear, my suggestion covers MC-LAGs and more importantly LAGs.

reading between the lines, in the case of single-device LAGs, I'm
guessing that you're talking about LAGs deployed over different line
cards on larger chassis based network devices.  On devices with a
single/unified data plane, conditionally applying session culling to LAG
interfaces based on the number of active bearer ports isn't an issue
because if you perform maintenance on that sort of device, all ports are
affected device-wide so the point is moot.

So the concerns you have seem only to be an issue in situations where
there are either MC-LAGs or else single-device LAGs configured on
different line cards in the same chassis and the maintenance operation
only applies to a single line card.  In either case, the LAGs would be
configured for throughput rather than resilience.

Regarding implementation, this mostly falls into "don't shoot yourself
in the foot" category.  Any network operator who rolls either MCLAG or
LAGs on multiple line cards will be aware that this is an area where
basic due diligence needs to be applied.

There are plenty of ways of shooting yourself in the foot at an IXP.
draft-ietf-grow-bgp-session-culling doesn't attempt to enumerate all of
them.

I wouldn't see a problem mentioning LAGs in a future -bis, but as others
have noted, there doesn't seem to be a compelling reason to pull the
document out of the rfc editor queue at this point.

Nick

---

From: Gert Doering <[email-masked]>
Date: Tue, 16 Jan 2018 22:36:54 +0100

Hi,

> document out of the rfc editor queue at this point.

+1

Gert Doering
        -- NetMaster

---

From: Thomas King <[email-masked]>
Date: Wed, 17 Jan 2018 17:14:51 +0000

Thanks for the feedback!

Let’s move ahead.
Thomas

On 16.01.18, 22:36, "Gert Doering" <[email-masked]> wrote:

    Hi,

    +1
    
    Gert Doering
            -- NetMaster

---

From: Randy Bush <[email-masked]>
Date: Wed, 17 Jan 2018 18:56:10 -0800

really do not want to get drawn into this.  but while looking into lags
a few years back, we found a lot of them split across line cards, with
resilience to line card failure being the stated reason.

randy

---

From: joel jaeggli <[email-masked]>
Date: Wed, 17 Jan 2018 21:35:29 -0800

> resilience to line card failure being the stated reason.

even in fixed configuration 1 and 2ru switches you may split them across
asics for survivability and fabric / forwarding capacity reasons.

---

From: Randy Bush <[email-masked]>
Date: Thu, 18 Jan 2018 10:45:27 -0800

> asics for survivability and fabric / forwarding capacity reasons.

we think we saw more results of hashing spread in those splits