Subject: [Bier] Datacenters and Use Cases

From: Caitlin Bestler <[email-masked]>
Date: Wed, 7 Oct 2015 09:36:12 -0700

An interesting side conversation started as the result of my prior email 
on the use-case document.
I'd like to capture some of the more pertinent thoughts about usage of 
BIER in datacenters.

First, I have a storage cluster application which will map to BIER 
almost perfectly. The sending nodes
pick a set of receivers for each chunk transfer. Being able to specify 
those targets with a bitmap fits
the application's needs almost perfectly.

I am also convinced that there are many HPC applications where this will 
be true. However we have
not yet gotten their attention to find out precisely what forms of 
addressing would be most useful
to those applications.

There is also a major difference between multicasting in a cluster and 
multicasting over the Internet.

Eliminating delivery to BFERs that do not need a given datagram is 
critical for long-haul networks.
Sending a datagram miles (or hundreds of miles) and then brillilantly 
figuring out that it did not
need to be delivered the remaining five feet isn't optimizing things much.

But for the clusters I'm designing a storage application for that is 
precisely the case (although the
distance between the racks is probably measured in yards rather than miles).

Basically it is fairly common to design a "non-blocking core", that is a 
set of switches which can
deliver all the traffic from all of the ingress ports to any set of 
egress ports as long as the aggregate
traffic to any egress port does not exceed that capacity. It is as 
though the core is a single 1024 port
non-blocking switch. Provisioning one is simply a matter of allocating 
sufficient internal links.

But in any distributed application there is always the potential that 6 
different senders will want to
send to the same destination concurrently. In order to manage that 
traffic it is important that the
egress port carry *no* excess traffic.

That means that excessive multicast deliveries are not just wasteful, 
they can undermine congestion
control efforts. Without congestion control a cluster can never achieve 
high network utilization.
In the storage cluster I am working on we rely on the target's being 
able to control who has permission
to send how much bulk data to them. Limiting the bulk data to be below 
some portion of the link
capacity means that deliveries can occur at nearly wire speed. But 
flooding of packets will kill that.
We need precisely tailored multicast groups without the overhead of 
IGMP/MLD joins being required
for each data transfer. BIER gives us a fully scalable solution that 
does that (we have other solutions
until BIER is widely deployed).

Anyway, this suggests that datacenter BIER applications get most of the 
benefit from elimination of
excess edge deliveries, while a long-haul BIER application benefits most 
from eliminating excess
core network deliveries.

---

From: Antoni Przygienda <[email-masked]>
Date: Wed, 7 Oct 2015 16:59:31 +0000

Ok, as cat-herder trying to find home for good ideas; best suggestion would be to write a use case in the draft you commented on, or not ? 

Then the hat trick is of course to find a molten-sand-square vendor willing to pour that into the next gen of their ware ...   ;-)

thanks for your comments, Caitlin

--- tony 

There are basically two types of people. People who accomplish things, and people who claim to have accomplished things. The first group is less crowded.
~~~ Mark Twain

> -----Original Message-----
> From: BIER [mailto:[email-masked]] On Behalf Of Caitlin Bestler
> Sent: Wednesday, October 07, 2015 9:36 AM
> To: [email-masked]
> Subject: [Bier] Datacenters and Use Cases
> 
> An interesting side conversation started as the result of my prior email on the
> use-case document.
> I'd like to capture some of the more pertinent thoughts about usage of BIER in
> datacenters.
> 
> First, I have a storage cluster application which will map to BIER almost
> perfectly. The sending nodes pick a set of receivers for each chunk transfer.
> Being able to specify those targets with a bitmap fits the application's needs
> almost perfectly.
> 
> I am also convinced that there are many HPC applications where this will be
> true. However we have not yet gotten their attention to find out precisely what
> forms of addressing would be most useful to those applications.
> 
> Eliminating delivery to BFERs that do not need a given datagram is critical for
> long-haul networks.
> Sending a datagram miles (or hundreds of miles) and then brillilantly figuring out
> that it did not need to be delivered the remaining five feet isn't optimizing things
> much.
> 
> But for the clusters I'm designing a storage application for that is precisely the
> case (although the distance between the racks is probably measured in yards
> rather than miles).
> 
> Basically it is fairly common to design a "non-blocking core", that is a set of
> switches which can deliver all the traffic from all of the ingress ports to any set
> of egress ports as long as the aggregate traffic to any egress port does not
> exceed that capacity. It is as though the core is a single 1024 port non-blocking
> switch. Provisioning one is simply a matter of allocating sufficient internal links.
> 
> But in any distributed application there is always the potential that 6 different
> senders will want to send to the same destination concurrently. In order to
> manage that traffic it is important that the egress port carry *no* excess traffic.
> 
> That means that excessive multicast deliveries are not just wasteful, they can
> undermine congestion control efforts. Without congestion control a cluster can
> never achieve high network utilization.
> In the storage cluster I am working on we rely on the target's being able to
> control who has permission to send how much bulk data to them. Limiting the
> bulk data to be below some portion of the link capacity means that deliveries
> can occur at nearly wire speed. But flooding of packets will kill that.
> We need precisely tailored multicast groups without the overhead of IGMP/MLD
> joins being required for each data transfer. BIER gives us a fully scalable solution
> that does that (we have other solutions until BIER is widely deployed).
> 
> Anyway, this suggests that datacenter BIER applications get most of the benefit
> from elimination of excess edge deliveries, while a long-haul BIER application
> benefits most from eliminating excess core network deliveries.
> 
> 
> _______________________________________________
> BIER mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bier