Subject: [Bier] draft-ietf-bier-use-cases: BUM use-case questions considerations

From: Toerless Eckert <[email-masked]>
Date: Thu, 29 Sep 2016 03:47:17 +0200

During the BIER interim, we discussed the VXLAM BUM use case for BIER.
This would be similar to the EVPN use-case mentioned in the use-case draft.

Q: Should VXLAN be added beside EVPN to the use case draft ?

Q: I haven't tried to read up on EVPN in detail: Is EVPN always 
"broadcasting" Unknown packets ? Isn't there a signaling option as well ?

Ask: If would be great if the description of the use-case would consider
example scale considerations, eg: Can we predict if, and how efficient
BIER would be ?

That efficiency is somrthin i am mostly worried about:

- Data centers may have as much as 10,000 TORs. Which i guess would be
  the BFER. 256 BitString length == 40 SIs. Aka: Worst case, if i have
  randomn allocation of 10,000 BFR-IDs, i would need to send up to
  40 BIER packets for a BUM packet. 
- Would VLANs have randomnly distributed VMs associated with them ?
  If not, eg: if they would stay usually within smaller PODs, that would
  be good to describe in the use-case text.
- Would i achieve better replication efficiency if i would start
  to plan the bits more, eg: instead of SI use subdomains for PODs or
  the like...

Aka: without being able to examplify the efficiency we can achieve with
BIER for this BUM use case its really hard to predict how useful
it would be for this use-case!

Cheers
    Toerless

---

From: Tony Przygienda <[email-masked]>
Date: Wed, 28 Sep 2016 19:54:37 -0700

> Q: Should VXLAN be added beside EVPN to the use case draft ?

IMO yes ...

> "broadcasting" Unknown packets ? Isn't there a signaling option as well ?

it does for BU_, for multicast a solution is currently worked on. Overlay
signaling for EVPN multicast with BIER has already been proposed.

> example scale considerations, eg: Can we predict if, and how efficient

There are numbers I saw from providers but I'm not free to disclose them.
They looked pretty impressive @ 256 bit mask.

>   40 BIER packets for a BUM packet.

40 copies to serve 10K destinations is  99.6% compression rate. Your
complaint being?

>   to plan the bits more, eg: instead of SI use subdomains for PODs or

which we have the architecture for ...  255 subdomain should give you pretty
good scaling if you exploit the locality on the topology. Inter-subdomain
forwarding
is a different issue outside BIER scope for the moment I'd say ...

-- tony

---

From: Alia Atlas <[email-masked]>
Date: Thu, 29 Sep 2016 09:39:29 -0400

A point of VXLAN and other NVO3 encapsulations is that there is a virtual
network (L2 in the case of BUM being interesting) that is being emulated.
That means that the BUM traffic is sent to the other VMs in the same VNI -
which should be significantly lower than all of the VMs in the data-center.

Of course, many data-centers are L3 based and many may still not be
virtualized.

Regards,
Alia

On Wed, Sep 28, 2016 at 10:54 PM, Tony Przygienda <[email-masked]>
wrote:

> _______________________________________________
> BIER mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bier
>
>

---

From: "Acee Lindem (acee)" <[email-masked]>
Date: Thu, 29 Sep 2016 13:53:57 +0000

Hi Alia,

From: BIER <[email-masked]<mailto:[email-masked]>> on behalf of Alia Atlas <[email-masked]<mailto:[email-masked]>>
Date: Thursday, September 29, 2016 at 9:39 AM
To: Tony Przygienda <[email-masked]<mailto:[email-masked]>>
Cc: Toerless Eckert <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Bier] draft-ietf-bier-use-cases: BUM use-case questions considerations

A point of VXLAN and other NVO3 encapsulations is that there is a virtual network (L2 in the case of BUM being interesting) that is being emulated.  That means that the BUM traffic is sent to the other VMs in the same VNI - which should be significantly lower than all of the VMs in the data-center.

Isn’t the scaling for BUM traffic more a question of the number PEs in the underlay rather than the volume of traffic in any particular VNI or EVPN Instance?

Thanks,
Acee

Of course, many data-centers are L3 based and many may still not be virtualized.

Regards,
Alia

On Wed, Sep 28, 2016 at 10:54 PM, Tony Przygienda <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, Sep 28, 2016 at 6:47 PM, Toerless Eckert <[email-masked]<mailto:[email-masked]>> wrote:
During the BIER interim, we discussed the VXLAM BUM use case for BIER.
This would be similar to the EVPN use-case mentioned in the use-case draft.

IMO yes ...

it does for BU_, for multicast a solution is currently worked on. Overlay signaling for EVPN multicast with BIER has already been proposed.

There are numbers I saw from providers but I'm not free to disclose them. They looked pretty impressive @ 256 bit mask.

40 copies to serve 10K destinations is  99.6% compression rate. Your complaint being?

which we have the architecture for ...  255 subdomain should give you pretty
good scaling if you exploit the locality on the topology. Inter-subdomain forwarding
is a different issue outside BIER scope for the moment I'd say ...

-- tony

_______________________________________________
BIER mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/bier

---

From: Alia Atlas <[email-masked]>
Date: Thu, 29 Sep 2016 09:58:29 -0400

Hi Acee,

On Thu, Sep 29, 2016 at 9:53 AM, Acee Lindem (acee) <[email-masked]> wrote:

> Hi Alia,
>
> From: BIER <[email-masked]> on behalf of Alia Atlas <
> [email-masked]>
> To: Tony Przygienda <[email-masked]>
> Cc: Toerless Eckert <[email-masked]>, "[email-masked]" <[email-masked]>,
> "[email-masked]" <[email-masked]>
> Isn’t the scaling for BUM traffic more a question of the number PEs in the
> underlay rather than the volume of traffic in any particular VNI or EVPN
> Instance?
>

True - I was thinking of the number of receivers, but the underlay would
need to be able to address each hypervisor or ToR.  Depending on how the
bits were assigned and applications were distributed onto physical
machines, the benefits from BIER in the underlay might not be that good.

Interested to hear others thoughts on the scaling and different places it
is beneficial.

Thanks for the correction!

Alia

Thanks,

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Thu, 29 Sep 2016 15:07:56 +0000

Provider tunnel segmentation already specified for MVPN and has been proposed for EVPN (https://tools.ietf.org/html/draft-zzhang-bess-evpn-bum-procedure-updates-03#section-6).

Here each segmentation region would be a BIER sub-domain, corresponding to a pod.

Jeffrey

From: BIER [mailto:[email-masked]] On Behalf Of Alia Atlas
Sent: Thursday, September 29, 2016 9:39 AM
To: Tony Przygienda <[email-masked]>
Cc: Toerless Eckert <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [Bier] draft-ietf-bier-use-cases: BUM use-case questions considerations

Regards,
Alia

IMO yes ...

-- tony

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 29 Sep 2016 19:38:21 +0200

On Wed, Sep 28, 2016 at 07:54:37PM -0700, Tony Przygienda wrote:
> > That efficiency is something i am mostly worried about:
> >

1) Addressing randomn small subsets of large number of BFER 
   (due to SI/subdomains)
   
   Large DC with 10,000 TORs, some VLAN (EVPN/VXLAN/whatever) with just 50 VMs.
   VMs spread aross 50 TORs.  Each TOR is BFER translating from VLANs L2/L3 multicast
   to BIER. Worst case, these 50 TORs have bits in 40 different SI/Subdomains
   (10,000 TORs / 256 BSL = 40). Sending TOR has to send 40 BIER packets instead of
   50 unicast packets without BIER/multicast. 

2) Large number of VMs behind small number of TORs

   Even if a VLAN has a large #VMs, but those are all behind few TORs,
   the savings via BIER or multicast is just that #TORs - a unicast solution
   would also only send #TOR copies of packets.

I have no DC stats. If i have VLANs with large #VMs behind large #TORs and
these an be 'regionalized' by eg: PODs, then BIER would do excellent. If either
of the two above scenarios are most common, it would not. So it would be great
to identify and write down known best cases in the use cases doc. Even if those
are just examples. But examples known to exist would IMHO be very helpfull.

Of course, any "M" example could likely result in orders of magnitude more
traffic than "BU" traffic,and even a factor of 2..10 replication efficiency
could justify BIER. And then it may not be relevant anymore that the total amount
of "BU" traffic efficiency optimization may on average not be high enough to 
justify BIER by itself.

Cheers
    Toerless

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Thu, 29 Sep 2016 18:03:25 +0000

Hi Toerless,

No there is no such assumption.

W/o segmentation, you may have to send 40 copies to reach just 40 VMs that are scattered sparsely.

W/ segmentation, you may need just one or a few copy to cover all those in the same ingress pod, and all those in different pods but are reachable via the border routers (segmentation points) in the same ingress pod.

Jeffrey

> -----Original Message-----
> From: Toerless Eckert [mailto:[email-masked]]
> Sent: Thursday, September 29, 2016 1:52 PM
> To: Jeffrey (Zhaohui) Zhang <[email-masked]>
> Cc: Alia Atlas <[email-masked]>; Tony Przygienda <[email-masked]>;
> Toerless Eckert <[email-masked]>; [email-masked]; draft-ietf-bier-use-
> [email-masked]
> Thanks, Jeff
> Are you implying that the mayority of EVPN/VXLAN VLANs would have VMs
> primarily only within a single POD ? Would be nice if the use-case draft
> could give some reasonable likely example.
> 
> Not sure if the following are realistic DC data, but if they wher,
> then this type of explanation could help to justify BIER for the readers
> that are like me now cued in that much about DC stats:
> 
>   - VMs in VLANs are most commonly spread inside a POD
>   - PODs likely have 100...1000 TORs
>   - VLANs can likely spread some 20..80% of TORs.
> 
> With numbers like these and BSL 256, BIER would give great replication
> efficiency - factor 50..200. Worst case 2..4 packets sent. Will never
> send a packet unnecessary to a TOR - with IP multicast one very likely
> would end up iwth setups where traffic would unnecessarily be sent to
> TORs.
> 
> etc. pp. Aka: try to figure out the best up-pitch we can describe.
> 
> Cheers
>     Toerless
> 
> On Thu, Sep 29, 2016 at 03:07:56PM +0000, Jeffrey (Zhaohui) Zhang wrote:
> > Provider tunnel segmentation already specified for MVPN and has been
> proposed for EVPN (https://tools.ietf.org/html/draft-zzhang-bess-evpn-bum-
> procedure-updates-03#section-6).
> >
> > A point of VXLAN and other NVO3 encapsulations is that there is a
> virtual network (L2 in the case of BUM being interesting) that is being
> emulated.  That means that the BUM traffic is sent to the other VMs in the
> same VNI - which should be significantly lower than all of the VMs in the
> data-center.
> >
> > On Wed, Sep 28, 2016 at 10:54 PM, Tony Przygienda
> <[email-masked]<mailto:[email-masked]>> wrote:
> >
> >
> > On Wed, Sep 28, 2016 at 6:47 PM, Toerless Eckert
> <[email-masked]<mailto:[email-masked]>> wrote:
> > it does for BU_, for multicast a solution is currently worked on.
> Overlay signaling for EVPN multicast with BIER has already been proposed.
> >
> >
> --
> ---
> [email-masked]

---

From: "Jeffrey (Zhaohui) Zhang" <[email-masked]>
Date: Thu, 29 Sep 2016 18:57:36 +0000

> BIER only inside
> PODs, Segmentation points become BUM level forwarders, receiving them as
> BFER, and replicating them out to all other necessary PODs BIER domains.

Yes that's the idea of segmentation.

Jeffrey

> Sent: Thursday, September 29, 2016 2:54 PM
> Ok, would this be a typial example then:
>   - Small-DC: my prior single POD example. 1..4 copies worst.
>   - Normal/large DC: Many pods with segmentation points between them.
>     #packets ~= 1..4 * #PODs covering the VLAN.
> 
> If the segmentation points would actively participate in the BUM solution,
> then IMHO we could get rid of those additional copies. Eg: Eg; BIER only
> inside
> PODs, Segmentation points become BUM level forwarders, receiving them as
> BFER, and replicating them out to all other necessary PODs BIER domains.
> 
> On Thu, Sep 29, 2016 at 06:03:25PM +0000, Jeffrey (Zhaohui) Zhang wrote:
> > Hi Toerless,
> >
> > W/ segmentation, you may need just one or a few copy to cover all those
> in the same ingress pod, and all those in different pods but are reachable
> via the border routers (segmentation points) in the same ingress pod.
> >
> > Jeffrey
> >
> > > > > which we have the architecture for ...  255 subdomain should give
> you
> > > pretty
> > > > Here each segmentation region would be a BIER sub-domain,
> corresponding
> > > to a pod.
> > > >
> > > > Jeffrey
> > > >
> > > >
> > > > Subject: Re: [Bier] draft-ietf-bier-use-cases: BUM use-case
> questions
> > > considerations
> > > >
> > > > which we have the architecture for ...  255 subdomain should give
> you
> > > pretty