Subject: Re: [Bier] Questions on draft-eckert-bier-te-arch-01

From: Toerless Eckert <[email-masked]>
Date: Wed, 7 Oct 2015 15:10:35 -0700

Hmm... no good explanation why this mail slipped by me answering until
Greg kicked my back. Sorry for that (only lame explanations *sigh*).

I think Neal alreay sent some responses to you..., adding to those.

On Thu, Aug 27, 2015 at 02:49:17PM -0400, Eric C Rosen wrote:
> According to the draft:
> 
>    Currently, BIER-TE does not support BIER-sub-domains and it does not
>    not use BFR-id.  BIER-TE headers use the same format as BIER headers
>    (BFR-id is set to 0)
> 
> If the BFIR-id field in the BIER header is set to 0, it is impossible 
> for the BIER payload to be an MPLS packet whose label stack begins with 
> an upstream-assigned label.  That's because the BFIR-id provides the 
> context in which the upstream-assigned label is interpreted.
> 
> Is that the intention?  That would seem to mean that BIER-TE cannot be 
> used to support MVPN.

No, this was definitely not the intention. The author who wrote this
(me) wanted to explain that the BFIR-ID is not required for BIER-TE forwarding
itself (it is required for BIER forwarding if i am not mistaken),
and incorrectly wrote "does not support".

And i did not come to think of the benefit/need of BFIR-ID for the
context of upstream assigned inner labels. Thats actually quite nice.
A lot easier than non-BIER MPLS solutions if i remember those correctly.

Neal mentioned also that not assigning a BFIR-ID can help to save bitspace,
which is true in BIER, but if we only use BIER-TE forwarding, it would
not even cost bits.

So, how about the following text suggestion:

| BIER-TE forwarding does not require/use the BFIR-ID. The BFIR-ID can
| still be useful though for coordinated BFIR/BFER functions, such as 
| the context for upstream assigned labels for MPLS payloads in MVPN
| over BIER-TE.
| 
| If the BIER-TE domain is also running BIER, then the BFIR-ID in
| BIER-TE packets can be set to the same BFIR-ID as used with BIER
| packets.
|
| If the BIER-TE domain is not running full BIER or does not
| want to reduce the need to allocate bits in BIER bierstrings for
| BFIR-ID values, then the allocation of BFIR-ID values in BIER-TE packets can
| be done through other mechanisms outside the scope of this document,
| as long as this is appropraitely agreed upon between all BFIR/BFER.

If you don't think there is sufficient benefit to keep the door open
to NOT leverage normal BIER assignment procedures for BFIR-ID, then
i could just drop the third paragraph.. ?

> From section 3.2.4:
> 
> A "local_decap" adjacency passes a copy of the payload of the BIER-TE
> packet to the packets NextProto within the BFR (IPv4/IPv6,
> Ethernet,...).
> 
> These local_decap adjacencies sound an awful lot like BFER-ids. Don't 
> you need one for each BFER?  So the BitString has to have a bit for each 
> BFER and a bit for each link that is along the path to any of the BFERs? 
>  Wouldn't this require that the number of BFERs be much less than the 
> number of bits in the BitString?

I could not find BFER-ID term being used anyplace other than
draft-ietf-bier-isis-extensions-00.xml, so i would not venture to
guess what its semantics is - any pointer i overlooked ?

How many bits do you need for TE ? Thats the key question. Obviously,
you can't get traffic-engineering for free, so you will require more
bits for the same network as with BIER. Which is why the BIER-TE draft
has already some optimizations where TE bits are used more efficiently,
and there are more possible.

Section 4.3 explain how Leaf BFER effectively require no bit
(sorry typo: 4.3:s/BFIR/BFER/, done for -02). Eg: just one bit per link
to leafs, and one shared (local decap) bit for all leafs in the network.

Likewise rings require also only one bit per node (local decap), and
a single shared bit for the ring (Section 4.6) (or even for multiple
rings).

So, maybe as a simple rule of thumb for this first round of bit
optimizations: One Bit per BFER, and one bit for every link without
a BFER on it. And those last-hop links are commonly > 50% of links
(could probably calculate from stats like average ingres:egress #interfaces,
 path-length and the like).

Any suggestion if/what should be added to the draft about this ?

> From section 4.9 of the draft:
> 
>    In a large network, multiple BIFT may be necessary, each one
>    identified by a different SI value in the BIER header.
> 
> Well, the only BIER encapsulation referenced by the draft is the MPLS 
> encapsulation, which does not have an SI value in the BIER header. 
> Probably you need to point out in section 3.3 that the MPLS label needs 
> to be associated with an SI, and modify section 4.9 to say that the SI 
> is inferred from the label.

Ack:

| To support larger networks, multiple bitstrings may be required/beneficial.
| This specification utilizes multiple SI (Set Identifiers) to support this. 
| A BIER-TE packets Set Identifier can either be derived from the context of the BIER-TE 
| packet encapsulation, or it can be part of a particular encapsulations BIER-TE header.
| MPLS BIER encapsulation does not include an SI field in the BIER header,
| so the SI needs to be associated with an MPLS label. Eg: for 10 SI in BIER-TE,
| 10 MPLS labels must be assigned to BIER-TE.</t>

> Even so, I don't really understand how this would work.  In ordinary 
> BIER, the bits identify the BFERs.  If for a particular packet you have 
> more BFERs than bits, you can always make two packets, and send each one 
> to (say) half of the BFERs.  In BFR-TE, the bits identify the links over 
> which the packet must flow.  If a particular packet has to flow over 
> more links than can fit into the BitString, you can't just make two 
> packets, and have each one traverse half the links.   I think this needs 
> a bit more explanation.

As in: How do i minimize the duplication of bit assignments for intermediate
links in the bitstrings of different SI ?

I think thats where a good controllers logic comes into play, or a good
operators ID/SI assignments - and certainly ideas from folks looking more
at this.

The most simple logic is to segment your network into non-overlapping areas, assign
an SI to each area, and effectively unicast a copy from the BFIR to the desired entry
 points in each of the areas. Unicasting eg: via Routed Adjacencies. If the
segments are non-overlapping, you could even get away without routed-adjacencies
at all: The ingres BFR directly maps from the non-BIER side lookup result
into a list of adjacencies, each of which has an IGP label to the segments
BFIR, followed by that segments BIER-TE label (SI 0), followed by the
appropriate BIER-header for that network segment.

> Perhaps it is sub-domains rather than SIs that are needed here.  One 
> could in theory have a sub-domain for each flow, and in that sub-domain 
> you'd only have to assign bit positions needed for that flow.  It seems 
> like it might be challenging though to figure out the least number of 
> sub-domains you actually need for a given set of flows.

I guess i don't really confused about the difference between sub-domain
and SI from the arch document and under which circumstances one is
better than another. Any suggestions welcome.

> BTW, the references need updating.

Am i missing references, or where the numbers of the drafts references
too old, even for the time when i ran the xml2rfc ?

Thanks a lot for the review!

Toerless
> _______________________________________________
> BIER mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/bier

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 8 Oct 2015 19:26:02 -0700

Eric/*:

To add to the thread. Talking with Ice today, he thinks that
it would be a lot more logical to only use sub-domains as the logical
concept to enable multiple bitstrings for BIER-TE. And not SI.
I hope i am correctly restating his explanations and expand on it:

The SI in BIER is automatically assigned from BFR-ID which is
split into loewr-bits = index into bitstring, and SI = which
bitstring. In BIER-TE we do not use BFR-ID for forwarding, and we
will need to "engineer which bitstring to assign each BFER (or
link into it) to. That is also done in BIER with sub-domains.

Adding on to this: if we want to reuse in BIER-TE the BFR-ID
in the BIER header, eg: in support of the MPLS upstream label context
case, that too becomes easier if we do not need to tie the selection
of the BIER-TE bitstring set to the BFR-ID - but if we use
just sub-domain.

If that makes sense, i can accordingly come up with text for the draft.

Cheers
    Toerless

-- 
---
Toerless Eckert, [email-masked]

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 9 Oct 2015 09:24:55 -0700

From a usage perspective SIs and sub-domains can be fairly interchangeable.
For my storage application I shifted from mapping groups of storage servers
from being sub-domains to assigning their IDs so they had the same SI.

So I can easily see an argument for eliminating one of them.

But why the SI rather than the sub-domain?

Is there a summary of their difference from both the implementation and
usage perspectives?

---

From: Antoni Przygienda <[email-masked]>
Date: Fri, 9 Oct 2015 18:50:24 +0000

Study the architecture document for distinction between SI and subdomain. 

SIs are a concept to fit more than BSL of BFERs into a subdomain. 

A subdomain is something where you map a service on so you achieve separation or can separate clusters of receivers from each other. Each subdomain may follow a different topology, carry different type of service, use different assignment of BFR-ids and with that has a different distribution tree possibly. 

If you can play tricks with assignments to fit them into same SI, you are basically using SIs in a sense as sub-domain. This is maybe OK for you but won't work well if you need e.g. service speration in terms of scheduling or receivers joining/leaving subdomains and want to separate a subdomain from the impact of another subdomain. 

BIER is an architecture, not a 'multicast cookbook'. We kept it quite generic to allow for all kind of unexpected applicatons as you have it and give you simply a box of tools like SI, MT, subdomains to allow for mapping of your architectural entities into BIER abstractions to fit your needs  ... 

Thanks 

--- tony 

There are basically two types of people. People who accomplish things, and people who claim to have accomplished things. The first group is less crowded.
~~~ Mark Twain

> -----Original Message-----
> From: BIER [mailto:[email-masked]] On Behalf Of Caitlin Bestler
> Sent: Friday, October 09, 2015 9:25 AM
> To: [email-masked]
> Subject: Re: [Bier] Questions on draft-eckert-bier-te-arch-01
> 
> Is there a summary of their difference from both the implementation and usage
> perspectives?
>

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 9 Oct 2015 12:49:44 -0700

> BIER is an architecture, not a 'multicast cookbook'. We kept it quite generic to allow for all kind of unexpected applicatons as you have it and give you simply a box of tools like SI, MT, subdomai…
Studying the architecture document it is apparent that the authors 
assume that there is some magic sauce
associated with the routing underlay associated with each sub-domain. 
But there is almost nothing mandated
about a sub-domain that is different from an SI slice of a sub-domain.

The SI slices of one sub-domain must use the same routing underlay. But 
nothing says that multiple sub-domains
cannot use the same routing underlay.

The BFR-IDs are unique to the sub-domain, and an SI is a slice of those 
IDs. But since BFR-IDs are assigned by a
mysterious process outside the scope of the document this does not seem 
terribly relevant unless you need a
BIER domain to have more than 2**16 ids.

Is there some reason why you could not replace the SI concept with 
simply have more sub-domains that mapped
to the same number of routing underlays?

You end up with at most 2**24 BFR-IDs in either case, and the same 
options for sizing the Bitstring.

Were the drafters assuming that a sub-domain was "expensive"? There's 
nothing in the draft that requires every
sub-domain to be a full cost sub-domain with a unique routing underlay.

---

From: IJsbrand Wijnands <[email-masked]>
Date: Fri, 9 Oct 2015 14:48:22 -0700

Hi Caitlin,

> Studying the architecture document it is apparent that the authors assume that there is some magic sauce
> associated with the routing underlay associated with each sub-domain. But there is almost nothing mandated
> The SI slices of one sub-domain must use the same routing underlay. But nothing says that multiple sub-domains

The double ‘not' is a little confusing, but I think what you are saying is that multiple sub-domain’s are allowed to map to a single underlay, which is correct and the intention of the draft to allow that. You can also assigned a different underlay for each sub-domain. As Tony explained, SI is an automatic mechanism for when the BFR-id overspills, and we create a new SI for it. The sub-domain is something that the network operator can configure to implement specific services.

> 
> The BFR-IDs are unique to the sub-domain, and an SI is a slice of those IDs. But since BFR-IDs are assigned by a
> mysterious process outside the scope of the document this does not seem terribly relevant unless you need a
> Is there some reason why you could not replace the SI concept with simply have more sub-domains that mapped
> You end up with at most 2**24 BFR-IDs in either case, and the same options for sizing the Bitstring.
> 
> Were the drafters assuming that a sub-domain was "expensive"? There's nothing in the draft that requires every

A sub-domain is expensive in the sense that you need to allocate a new label(range) for it, advertise it through the network using the IGP and it will provision a new BIFT in all the BIER routers. 

Thx,

Ice.

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 9 Oct 2015 15:06:35 -0700

> A sub-domain is expensive in the sense that you need to allocate a new label(range) for it, advertise it through the network using the IGP and it will provision a new BIFT in all the BIER routers.
If there a N BFR-IDs total in the BIER domain, why is it more expensive 
to break them into 2x sub-domains as opposed
to X sub-domains that are each twice as large?

If there are 2x routing underlays then clearly things would be more 
expensive, but what expense is there in having
an additional sub-domain rather than an additional SI slice?

---

From: IJsbrand Wijnands <[email-masked]>
Date: Fri, 9 Oct 2015 15:26:10 -0700

Caitlin,

> If there a N BFR-IDs total in the BIER domain, why is it more expensive to break them into 2x sub-domains as opposed

Not sure what you mean here...

> 
> If there are 2x routing underlays then clearly things would be more expensive, but what expense is there in having

A SI is as expensive as a sub-domain, there is no difference.

Thx,

Ice.

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 9 Oct 2015 15:44:37 -0700

On 10/9/15 3:26 PM, IJsbrand Wijnands wrote:
> Not sure what you mean here..
If there are 5000 BFRs you need 5000 rules, no matter how many 
sub-domains you divide them over.

> A SI is as expensive as a sub-domain, there is no difference.
Exactly.

If there is no real difference then why have both concepts in the interface?

---

From: IJsbrand Wijnands <[email-masked]>
Date: Fri, 9 Oct 2015 15:55:09 -0700

Caitlin,

> If there is no real difference then why have both concepts in the interface?

There is a big difference in how the SI and Sub-domain are used, deployed and managed!

Thx,

Ice.

---

From: Antoni Przygienda <[email-masked]>
Date: Sat, 10 Oct 2015 04:45:52 +0000

> There is a big difference in how the SI and Sub-domain are used, deployed and

[Tony saiz:]  in very short, do not run subdomains until you start to realize the restrictions of the SI ;-)

To +1 Ice:

* SI is an _automatic_ mechanism that allows to deploy more than BSL  BFRs.  If you have a single service, are happy with possibly sparse allocation pattern with lots of packets due to SI replication and do not need fancy stuff like restricing links allowing to carry the multicast, it's all you need

* if you're a carrier deploying multiple overlays, services, needing to separate the way BFR-ids are assigned in SIs, need fairness across multiple multicast services, VPN SLAs, separation between data carried on same label for security purposes and other jazzy stuff customers shell $$$ for, you'll find subdomains intriguing and if you run gear by few serious vendors supporting MT IGPs, doubly so ... 

In short, if in DC, SIs is probably all you need and subdomain #0, MT #0 will serve your needs ... 

-- tony

---

From: Eric C Rosen <[email-masked]>
Date: Mon, 12 Oct 2015 14:34:20 -0400

Toerless, Neale,

Thanks for your interesting responses.  I am still inclined to say that 
this proposal needs more baking before it is ready for WG adoption.

> [Toerless] The author who wrote this (me) wanted to explain that the
>  BFIR-ID is not required for BIER-TE forwarding itself (it is
> required for BIER forwarding if i am not mistaken), and incorrectly
> wrote "does not support".

I don't think the BFIR-id is needed for BIER forwarding, it's only
needed if the multicast flow layer at the BFERs needs to do some sort of
RPF check. I don't think BIER and BIER-TE are different in this respect.

> [Neale] The intent was to reduce the number of bit-positions needed
> in the network by not requiring the BFIR to have one. The draft
> should therefore instead state that the BFIR-ID MAY not be set in
> the packet, and if not, then there are the consequences that you
> have identified.

> [Toerless] Neale mentioned also that not assigning a BFIR-ID can
> help to save bitspace, which is true in BIER, but if we only use
> BIER-TE forwarding, it would not even cost bits.

Right, in BIER-TE forwarding the BFIR-id doesn't require a bit in the
BitString, so eliminating the BFIR-id doesn't really seem to provide
much value.

> [Toerless] I could not find BFER-ID term being used anyplace other
> than draft-ietf-bier-isis-extensions-00.xml, so i would not venture
> to guess what its semantics is - any pointer i overlooked ?

A BFER-id is just the BFR-id of a BFER. Similarly, a BFIR-id is just
the BFR-id of a BFIR.

With regard to the issue of not having enough bits to represent the
entire multicast distribution tree ...

> [Neale] Since the data is not distributed throughout the domain, it
> does not need to be unique within the domain.

By "it", do you mean the bit position that represents a particular link?
Or do you mean the BFIR-id?  Note that BFR-ids are already unique only
relative to a sub-domain.

> [Neale] So, one might also consider applying the same BFIR-id to a

Same BFR-id?

> group of BFERs that belong to the topology. If that bit-position is
> then set in the packet, then it is received by all BFERs that the
> packet reaches, and that is determined by which other bits are set in
> the BitString. Since in TE BIER a packet cannot reach BFER if the
> BitString does not describe a path to it.

> [Toerless] Section 4.3 explain how Leaf BFER effectively require no
> bit (sorry typo: 4.3:s/BFIR/BFER/, done for -02). Eg: just one bit
> per link to leafs, and one shared (local decap) bit for all leafs in
> the network.

I didn't really understand this.  How does a node know whether it is a 
leaf (a BFER) for a particular packet?  Doesn't this mean that every BFR 
through which a packet passes is going to have to figure out whether it 
is supposed to be a BFER for that packet?  I'd like to see the details 
of how that would work.  BTW, however it works, it will probably require 
knowing the BFIR-id, unless that can be inferred from the sub-domain 
that is implicitly identified by the BIER-MPLS label.  (Consider, e.g., 
a case where a given flow is traveling on two different TE trees, 
perhaps from different BFIRs, but a given transit node is on both trees.)

> [Toerless] The most simple logic is to segment your network into
> non-overlapping areas, assign an SI to each area, and effectively
> unicast a copy from the BFIR to the desired entry points in each of
> the areas. Unicasting eg: via Routed Adjacencies.

Assuming we replace "SI" with "sub-domain", is this the same as Neale's
suggestion:

> [Neale] perhaps one could consider one or more uni-directional
> BIER-TE sub-domains rooted at each BFIR. Each individually signalled
> sub-domain comprises a sub-set of the BFERs (and the links to
> traverse to them) that the BFIR needs to address. Flows can then map
> onto one or more of these sub-domains. This scheme keeps the number
> of sub-domains scaling with network size (as with SPF-BIER sets),
> rather than number of flows.

So each sub-domain would be a sub-tree of the tree from a given BFIR.
You unicast-tunnel a packet to the root of a sub-domain, and then let 
BIER distribute the packet down the sub-tree. The number of sub-domains 
is then proportional to the number of BFIRs (which is still a lot of
sub-domains), and each transit node would have to determine for each
packet whether it needs to consume that packet or just pass it along.
Part of the BFER processing would be to figure out for each packet (a)
am I supposed to receive this packet and (b) did I receive it from the
sub-domain over which I'm expecting it?

It seems like there are a lot of details to be added, and some of them
might be dependent upon the multicast flow layer.

> [Toerless] Am i missing references, or where the numbers of the
> drafts references too old, even for the time when i ran the xml2rfc?

The references are to the old individual documents rather than to the WG
documents.

> [Toerless] rings require also only one bit per node (local decap),
> and a single shared bit for the ring (Section 4.6) (or even for
> multiple rings).

I don't quite understand from section 4.6 why a given packet will not
revolve around the ring forever, as the "ring bit" only seems to get
reset when the packet leaves the ring.  Also, if the "ring bit" gets 
reset when a packet leaves the ring, how do you use one bit for multiple 
rings?  Perhaps I've misunderstood something.

Eric

---

From: Eric C Rosen <[email-masked]>
Date: Mon, 12 Oct 2015 15:17:02 -0400

> to the same number of routing underlays?

In addition to what Ice and Tony have said, here's one way to think 
about it.

A particular application/service at a particular BFIR would be 
provisioned to send a certain class of flows in a certain sub-domain. 
The set of BFERs to which a given flow is to be sent would generally not 
be provisioned, but would be learned dynamically.

So the application/service tells BIER "here's a packet; send it on this 
sub-domain to this set of BFERs", where the set of BFERs is identified 
as a set of IP addresses (BFR-prefixes).  BIER maps the sub-domain and 
the set of BFR-prefixes to a set of BFR-ids.  If the set of BFR-ids 
cannot fit into a single BitString, the SI method is used to 
automatically create several packets.

This would all become rather more complicated if you had to use multiple 
sub-domains to reach a given set of BFERs, as the application/service 
would now have to know which sub-domain to use for each BFER.

Further, when a packet reaches a BFER, the BFER may need to figure out 
the BFR-prefix corresponding to the BFIR (this is necessary, e.g., for 
the MVPN application).  If the BFIR-id and the BFER-id were not from the 
same sub-domain, this would be difficult to do -- the packet would have 
to carry information allowing one to identify both the source sub-domain 
and the destination sub-domain.  But the architecture doesn't even have 
a notion of "source" and "destination" sub-domains.

Of course, it might be possible to rewrite the architecture to eliminate 
the SIs and replace them with sub-domains.  But since that would only 
make the architecture more complicated, it doesn't seem like a very good 
direction to pursue.

---

From: Antoni Przygienda <[email-masked]>
Date: Mon, 12 Oct 2015 20:38:42 +0000

> From: Eric C Rosen [mailto:[email-masked]]
> Sent: Monday, October 12, 2015 12:17 PM
> To: Caitlin Bestler; Antoni Przygienda; [email-masked]
> > simply have more sub-domains that mapped to the same number of routing
> > underlays?
> 
> In addition to what Ice and Tony have said, here's one way to think about it.
> 
> A particular application/service at a particular BFIR would be provisioned to send
> a certain class of flows in a certain sub-domain.
> The set of BFERs to which a given flow is to be sent would generally not be
> provisioned, but would be learned dynamically.
> 
> So the application/service tells BIER "here's a packet; send it on this sub-domain
> to this set of BFERs", where the set of BFERs is identified as a set of IP addresses
> (BFR-prefixes).  BIER maps the sub-domain and the set of BFR-prefixes to a set
> of BFR-ids.  If the set of BFR-ids cannot fit into a single BitString, the SI method
> is used to automatically create several packets.
> 
> This would all become rather more complicated if you had to use multiple sub-
> domains to reach a given set of BFERs, as the application/service would now
> have to know which sub-domain to use for each BFER.
> 
> Further, when a packet reaches a BFER, the BFER may need to figure out the
> BFR-prefix corresponding to the BFIR (this is necessary, e.g., for the MVPN
> application).  If the BFIR-id and the BFER-id were not from the same sub-
> domain, this would be difficult to do -- the packet would have to carry
> information allowing one to identify both the source sub-domain and the
> destination sub-domain.  But the architecture doesn't even have a notion of
> "source" and "destination" sub-domains.
> 
> Of course, it might be possible to rewrite the architecture to eliminate the SIs
> and replace them with sub-domains.  But since that would only make the
> architecture more complicated, it doesn't seem like a very good direction to
> pursue.

 [Tony saiz:]  I would whole-heartedly agree here. Sub-domains are really here to 
separate bigger-grain concepts like services or different router sets running BIER, 
think source/destination ports in a sense. 
Orthogonally, SI gives a transparent solution to the fact that we can't realistically 
carry 32KB receiver bitmasks in each packet, in a sense equivalent to IP 
fragmentation solving  the problem of limited media MTU. 

--- tony

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 12 Oct 2015 13:45:53 -0700

> fragmentation solving  the problem of limited media MTU.

The issue is that you are complicating the user interface to BIER for 
the benefit
of an implementation detail that you are not describing in the 
architecture, and
I believe is officially not-in-scope.

Would it be appropriate to add language to the effect that each Sub-Domain
SHOULD/MAY map to a different routing underlay?

---

From: Antoni Przygienda <[email-masked]>
Date: Mon, 12 Oct 2015 20:55:42 +0000

> >   [Tony saiz:]  I would whole-heartedly agree here. Sub-domains are
> > really here to separate bigger-grain concepts like services or
> > different router sets running BIER, think source/destination ports in a sense.
> > Orthogonally, SI gives a transparent solution to the fact that we
> > can't realistically carry 32KB receiver bitmasks in each packet, in a
> > sense equivalent to IP fragmentation solving  the problem of limited media
> MTU.
> >
> > --- tony
> 
> The issue is that you are complicating the user interface to BIER for the benefit
> of an implementation detail that you are not describing in the architecture, and I
> believe is officially not-in-scope.

 [Tony saiz:]  hmm, both SI and subdomain concepts are explained extensively in the architecture documents. 

> SHOULD/MAY map to a different routing underlay?

[Tony saiz:] sigh, no, in most generic sense the same routing underlay carries all sub-domains in the normal case. 

I am tad baffled by your seemingly unique interpretation of the architecture @ this point in time. We had tons of people looking @ it by now and no'one tried to replace SI with sub-domains or felt that the concepts are the same.

I utterly fail to see how the UI towards bier is being complicated given what you need, as I said, if your SDK towards BIER on your platform does not need  MT or sub-domains, just don't expose it and use 0,0 for everything. SI is hidden in the BIER layer itself. All you need really then is two (three) calls: 

Socketopt(my_BSL);   // set bitstring length if not default 
recv_on_bier(my_bfer_bit_nr);  // you can even skip the bit_nr given you're a leaf and frame arriving means it's for you as BFER 
send_to_bier(bfer_dst_bitmask);   // that will replicate/SI in the driver

the rest is lower layer/driver stuff. 

all done, forget all the other stuff like overlays/underlays/subdomains/MTs/BSL translations/unicast tunnels and anything else in the architecture that seems superfluous for your needs ... 

thanks 

--- tony

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 12 Oct 2015 14:21:05 -0700

> all done, forget all the other stuff like overlays/underlays/subdomains/MTs/BSL translations/unicast tunnels and anything else in the architecture that seems superfluous for your needs ...

This is more of an issue of document clarity than a request for an 
architecture change.

My areas of expertise are at the application layer, host stacks, NICs 
and L2 switching. Viewed through those perspectives
there is virtually nothing in the architecture document that explains 
why a sub-domain exists and what the implications of
using a sub-domain vs. the use of an SI-subset are. Section 6.1's 
discussion of adolescences alludes to a few specifics, but
does not present an explanation.

As it stands right now the reader is left with "sub-domains and Subnet 
Identifiers are different because we said so." You can
implement with that, because the differences are well explained. But the 
lack of an inline or even cited rationale is bothersome
from the perspective of someone who wants specs to be minimal and 
self-justifying.

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Oct 2015 17:02:57 -0700

> RPF check. I don't think BIER and BIER-TE are different in this respect.

Indeed.

> BitString, so eliminating the BFIR-id doesn't really seem to provide

16 bit header saved. 

> entire multicast distribution tree ...

Right...

> of how that would work.

Conroller determine if BFR is BFER:
  Is there a service edge using BIER-TE and BFR could be receiver
  on some of these services ?  Ok, you're a BFER.

Controller determines what bits are needed to support BFER:
  Is it even feasible and if so, desirable to send packets
  into this BFR and have them transit towards another BFR ?
  No ? Ok, you're a leaf BFER. 

If BFR is leaf BFER, then controller only assigns bit to the
links leading up to the leaf BFER, not the BFER itself.

For all non-leaf BFER, controller assigns an individual bit.

If there are one or more leaf BFER in network, controller assigns
one "shared leaf BFER" bit. Thats set to have local decap entry
in all the leaf BFER. It can be shared, because it would only be
reset when it passes through some BFR that has it set, and given
how all BFR that have it set are leaf BFR, this will not happen. QED.

> BTW, however it works, it will probably require 

I don't think BFIR-ID plays a role here. 

[...]
> >rather than number of flows.

Possibly the same, yes. Neal doesn't belabor the unicast/MPLS part
"how to get a BIER packet from any BFIR to some remote subdomain" as i do.
Not sure if he envisions something different than i or just thinks
that piece is obvious.

> BIER distribute the packet down the sub-tree.
^^^^^^ BIER-TE

> The number of sub-domains 

No, not really.

Lets run a simple example: 5 country network. 1000 BFER, each country
200 BFER, 3 routers in each country connecting to core. Call them 
them sub-domain BFIR if you want. Controller allocates
one sub-domain per country. Fills up 256 bitstring with in-country
BFER and in-country links Aka: lets say all relevant links between
sub-domain BFIR and BFER are < 56.

Lets look at BFIR service lookup for some IPTV (S,G) on ANY BFIR
across the whole network: OIF list of 5 replications, each replication
has unicast/MPLS label of one sub-domain BFIR follows by that BFIRs
label for the subdomain of interest, followed by the BIER bitstring for
the "in-country bits".

Scales with number of BFER + number of relevant replication
transit hops.  In above example, maybe 20% of BFER ???

If you'd try to run this example with BIER instead of BIER-TE and
did randomn BFR-ID == SI allocation, you may end up sending more copies
across a lot of links. Eg: when you end up with BFR-ID with 2 different
SI in a country. So the logic of allocating BFR-ID or sub-domains
is really necessary/beneficial almost as much in BIER as in BIER-TE
in these type of examples.

> sub-domain over which I'm expecting it?

Hmmm... no, can't parse that. knowing which sub-domain a packet is
for is like BIER via the label when MPLS encap is done (otherwise it
wouldhave to be in the BIER header). Punting to higher layer is
just based on having a bit with local decap and as explained above,
the controller can intelligently assign this shared for leaf BFERs.

> might be dependent upon the multicast flow layer.

Let me know specifically what piece of explanation you'd like to see.
If the above example, maybe with a picture would help, thats easy.

> The references are to the old individual documents rather than to the WG

Arghh... ok. stupid me. I was wondering what xml2rfc wouldn't
update automatically.

> rings?  Perhaps I've misunderstood something.

 "On BFR2, the adjacency also points to the
   clockwise neighbor BFR1, but without DNR set.  "

DNR bit is part of the adjacency in the BIFT, so only
the copies sent across the ring adjacencies will have it set, and then
we do not set DNR towards the end of the ring before it would loop. 

Multiple rings is just an example. DNR allows you to save bits
in bitstring whenever you are fine all BIER-TE traffic is flooded
across that sub-topology. Just set up in the BIFTs appropriate
adjacencies to flood (loop free) across the topology.

Cheers
    Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Oct 2015 18:16:33 -0700

If we want to maintain that "easy" way how flow overlay signaling
works, then IMHO we'd also need to rely on SI for BIER-TE,
or else we'd need to enhance the proposed signaling extensions
for eg: IGPs to better support sub-domains.

Even in BIER it seems in a larger networks that require BIER 
with multiple SI, the operator likely would want to allocate BFR-IDs
so that he doesn't create unnecessary duplicates

[ Eg: 10 edge areas with 200 BFR each. If you allocate a separate SI
  per area, each area will see one copy for each BIER packet. 
  Allocate BFR-ids randomnly -> up to 10 packets into an area]

When you come to think of this, you likely will start allocarting
BFR-id across larger networks keeping SI in mind, eg: in buckets of
256 or whatever the bitstring size is. And in each bucket you want
to keep some headroom for growths (additional BFER added to area
later).

The main difference for BIER-TE is that we'd need more unused headroom
in the BFR-id space thats not allocated to BFER because we need it
in TE for forwarding to intermediate links. Not sure percentage:
20%..30% of bits in bitstring ?

I for once could easily see how i could use the BIER BFR-id also for
BIER-te if my allocation scheme was keeping enough headroom,
eg: plan for initially 51%, longer term not more than 70% of
bits in SI for BFR-id, keep the rest for BIER-TE cases. 

If thats felt to not be appropriate by customer then he likely
would need to simply duplicate the BFR-id allocation: one
BFR-id in sub-domain 0 for BIER with the target of a very high
filling of bits for every SI, and another subdomain for BIER-TE
where the BFR-id in that subdomain are allocated with enough
headroom for the transit hops.

Makes sense ?

--
Toerless

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 13 Oct 2015 10:37:40 -0700

Caitlin, 

Q1: I think you are referring to the BIER arch, not just BIER-TE arch,
so i changed the mail thread subject. Maybe the BIER arch authorw will
comment on your call for more clarity. If you are asking for specific clarity
in BIER-TE, pls. let me know.

Q2: "adolescences" ? I do not understand.

To answer your content question, SI vs sub-domain.

-> An application using BIER does not want/need to be bothered with
   bits in a bitstring. It simply creates BIER sockets(), optionally
   specifies a sub-domain (other than 0), and the list of BFER
   BFR-prefixes (IP addresses of receivers) that this socket should send copies
   to. Sends packets. Andds/removes BFR-prefixes over time if receiver
   set is changing.  Done. (*)

   How does this magic work ? The most important piece is SIs. SIs
   allow to have a 16 bit space of BFR-ids that are through routing
   underlay (eg: IGP) mapped to BFR-prefixes. 

   Why then SI and sub-domains ?
   
   Well, network operator will start (IMHO) assigning SI to BFER based
   on physical network topology.  As explained in prior emails for large
   networks, i think this would not be randomn, but based on geography -
   split up network into regions, assign BFERs in one region BFR-ids with 
   same SI (as much as possible).

   Now some app sends out humunguous amount of data and always
   has eg: < 256 receivers, but  they're all in a different region.
   Bummer: 20 regions, 10 receivers each, but sender needs to send
   out 10 copies anyhow because of SI allocation. So network operator
   creates another virtual BIER network... oops: sub-domain. In this
   subdomain only the potential BFER for this app/user are mapped
   to BFR-id, and e voila, the app uses this sub-domain, and the
   traffic in the network is again as efficient as we'd like - only
   one  packet sent to reach all receivers.

-> Is this the full answer for your use-case, DC ? Not quite i think.
   Would be good to discuss more, but i don't see you coming to Yokohama ?

   Eg: AFAIK, DCs have huge (>> 2^16) VMs. Each user/application will
   most likely have much less than 2^16 receivers. Creating per-app
   sub-domains sounds like the obvious starting point. SIs would come
   into the process automatically, as soon as an applicatin requires more
   than Bitstringlength possible receivers. BUT:

   1. AFAIK, we have not defined for BIER any automated schemes to assign unique
   BFR-id to receivers. Doing it once for eg: the physcial topology - manually
   is a well known process for operators (like allocation of loopback addresses).
   Doing this repeatedly for multiple apps IMHO requires an automated
   scheme. This would be a good (potentially DC centric) signaling
   problem to look into.

   2. I do not know how many BIFT routers in the future would support.
   Its definitely important to find solutions that minimize the
   number of required BIFT. Each BIFT is in-network state, and one of
   BIERs claim to fame is to reduce state. Of course, even if you
   had a lot of state with BIER its still better than MPLS/PIM because
   its not dynamically changing with membership, but we should try
   to propose (SI,sub-domain) solutions that are as efficient as possible.

   So,... i don't know enough about your DC targets: Number of overall
   network-wide possible receivers, number of different app-instances
   (eg: with different subset of possible receivers),...

-> Hope this gives an idea why today i think the framework of (SI,subdomain)
   is necessary (can't get rid of either without making apps more difficult),
   but also sufficient (can't think of additionally useful concepts to scale BIER
   receivers AND create the option for optimized replication).

Cheers
    Toerless

---

From: Antoni Przygienda <[email-masked]>
Date: Tue, 13 Oct 2015 17:48:11 +0000

>    scheme. This would be a good (potentially DC centric) signaling

 [Tony saiz:]  look @  

https://tools.ietf.org/html/draft-prz-bier-bfrid-assignment-00

didn't have time to update, waiting for interest to spike once 
enough people see far enough to understand the problem.  Then I'll reissue. 

--- tony

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 13 Oct 2015 12:59:50 -0700

I meant to type "adjacencies". Apologies for not catching the 
auto-correction.

Anyway, from your explanation I think I have a fairly good understanding.

The key *intended* difference is that the way sub-domains are *intended* 
to be
used the end nodes would be automatically assigned to the sub-domains based
upon the routing underlay.

But the architecture document makes it clear that there *may* be 
multiple sub-domains
using the same routing underlay, in which case the assignment of end 
nodes to sub-domains
starts to look a lot like just assigning a BFR-ID.

Is there some succinct explanation as to why a deployment SHOULD NOT map 
multiple sub-domains
to a single routing underlay, but they MAY do so for the following 
reasons... ?

If sub-domains SHOULD map to a unique routing underlay while SI-subsets 
MUST map to the same
routing underlay then you have a justifiable distinction. But 
sub-domains that MAY map to the same
or different routing underlays while SI-Subsets MUST map to the same 
routing underlay as the SI just
seems like excessive detail in the API.

---

From: Caitlin Bestler <[email-masked]>
Date: Tue, 13 Oct 2015 13:48:01 -0700

On 10/13/15 10:37 AM, Toerless Eckert wrote:
> > Is this the full answer for your use-case, DC ? Not quite i think.

Yes, a typical intra-DC application will map a VLAN or VXLAN to a Bier 
Domain.
The membership in this virtual cluster may be for the duration of a job, or
indefinite. But the VLANs/VXLANs will probably be set up by cluster 
management
rather than by hand

Further, the desired forwarding path for a given virtual cluster is 
intended to
be very short. It might be longer if you are forced to allocate nodes in 
a large
number of racks, but that is not what the scheduler will be trying to do.

That also means you are unlikely to see BIER in datacenters until it is 
implemented
in commodity forwarding chips that are deployed in edge servers. 2-hops with
old fashioned multicast groups will beat 4 hops with BIER.

We rejected emulating what we wanted switches to do in edge switches for our
storage cluster because you can't make the latencies required when you 
add extra
physical hops to your path.

---

From: Eric C Rosen <[email-masked]>
Date: Fri, 16 Oct 2015 14:48:35 -0400

>> [Eric] in BIER-TE forwarding the BFIR-id doesn't require a bit in the

> [Toerless] 16 bit header saved.

I think that would count as "not much value" ;-)  FWIW, originally the 
MPLS encapsulation made the BFIR-id an optional field, but then you need 
a flag bit to say whether it is present, and it just seemed simpler to 
always include it.

> [Toerless] If BFR is leaf BFER, then controller only assigns bit to the

> For all non-leaf BFER, controller assigns an individual bit.

Got it, I didn't attend properly to the fact that non-leaf BFERs have 
their own bits.

Maybe I should have said "almost got it" ;-)  Under what conditions will 
a leaf node receive a packet that it doesn't need to consume?  That is, 
what is the significance of failing to set the local_decaps bit?

Doesn't this scheme make it difficult to add a new BFER for an existing 
flow, or to change a non-leaf node from being a non-BFER to being a BFER 
for an existing flow?

> [Toerless] Lets run a simple example: 5 country network. 1000 BFER, each country
> did random BFR-ID == SI allocation, you may end up sending more copies

In non-TE BIER, you could also set up each country as a separate BIER 
sub-domain (or even a completely separate BIER domain, if we had 
inter-domain signaling of some sort).  Something like that might be 
needed anyway for inter-AS BIER.

> the controller can intelligently assign this shared for leaf BFERs.

Think about an application like MVPN extranet, where one (S,G) flow is 
traveling through one sub-domain, another (S,G) flow is traveling 
through another sub-domain, and you have to make sure that each flow 
gets punted to the proper VRF.  If a given BFER for that flow is in both 
sub-domains, you'd need to know the sub-domain in order to dispatch to 
the proper VRF.  But I guess this could happen in non-TE BIER as well if 
different VPNs are using different sub-domains.

>> [Eric] I don't quite understand from section 4.6 why a given packet will not
> [Toerless]  "On BFR2, the adjacency also points to the

> we do not set DNR towards the end of the ring before it would loop.

Doesn't this assume that all the BIER-TE packets enter the ring at a 
single ingress?  If packets can enter the ring at multiple points, what 
is the meaning of "end of the ring"?

> [Toerless] Multiple rings is just an example. DNR allows you to save bits

I see how this would work for a sub-topology that is itself a tree 
(single ingress).  But if the non-leaf nodes in the sub-topology are 
also BFERs, you'd still need bits for them.

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Oct 2015 12:41:26 -0700

> a flag bit to say whether it is present, and it just seemed simpler to 

Agreed. I forgot the smiley in my text.

> what is the significance of failing to set the local_decaps bit?

With todays scope of BIER-TE only when the bitstring had some bit that
was assigned to an interface into the leaf. 

We could easily avoid the single "shared leaf bit" by having a forwarding
plane feature "you're a leaf BFER, every BIER-TE packet you receive
needs to be punted". But thats a new forwarding plane feature, the
"shared leaf bit" is just a bit allocation trick, so it keeps the forwarding
plane simpler/cleaner. 

> flow, or to change a non-leaf node from being a non-BFER to being a BFER 

Lets say you have a BFER with 2 upstream links into it. And the
bitstring already has enough bits to get the packet into both routers
north of these links.

If its a leaf BFER, you just add one of the two link bits to get
it to the leaf BFER. And the choice which one you use is your
traffic engineering aspect. And we assume that the shared-leaf bit
was already set for some other leaf BFER somewhere else in the network.

If its not a leaf BFER, you still need to set either of these two bits
for the two links, but now you also need to set one unique bit for
the BFER itself so the BFER can punt it. Because you may not want this
BFER to punt it, but instead just some other BFER further south of it.

> needed anyway for inter-AS BIER.

Sure. I hope BIER-TE can be agnostic and follow the same design principles:

- Use SI pretty much as in BIER to enable using same flow overly and
  routing underlay that relies on the BFR-ID (that implies SI).
- Use same subdomain as BIER or separate one based on the bit index
  allocation you use for BIER (do you have enough spare bit indices
  you can use in BIER-TE ? No ? -> use separate sub-domain).

> different VPNs are using different sub-domains.

The VRF to receive into is a parameter of the local_decap adjacency.
I would prefer i only need one bit per BFER per subdomain, which
for the default sub-domain would likely mean to receive only into
the global table and then the next encap, eg: MPLS label determines
which VRF to actually process the packet in. If you assume you have
multiple sub-domains, then you'd need to have a bit for the BFER in
each sub-domain anyhow, so each of these bits could have a local_decap
with a different VRF.

The shared leaf-BFER local_decap bit would likely still work  in
this context: you have some subdomain that likely is used for
a particular VPN context. All leaf-BFER with that sub-domain
would have this bit populated with local_decap(VRF1). And
the shared leaf-BFER bit in another subdomain in the same BFER
would have local_decap(VRF2).

If we tried to eliminate the shared leaf bit, then we would need
to have the forwarding logic: If you receive a BIER-TE packet
for a particular subdomain X, punt to VRF associated with X.

> is the meaning of "end of the ring"?

In the example, all packets are entering from one of two "upstream"
routers. That the typical MSO case. If you have ring traffic where
multicast can enter from any node, then the TE design is a bit
more complex: You effectively have to use two bits, one to pass
the packet clockwise around the ring, the other one to pass it
counterclockwise. You still choose one link on the ring where
you break the loop. Whenever you receive a packet fromm outside
the ring, the copy you send clockwise into the ring would already
have the counterclockwise bit reset because the DNR bit is only
keeping the bit set when the packet is replicated across the adjacency
on which the DNR bit is set. Liwise for the copy in the other
direction.

Maybe i should use that more complex example, i was just afraid it
would have readers eyes bleed too much ;-)

> also BFERs, you'd still need bits for them.

Yes. From the accounting side you always assume one bit per BFER,
then the question is how many more bits you need for the intervening
topology. Eg: Ring is one or two bits depending how traffic needs
to flow. Leaf BFER you only need additional bits for the second and
further links into the BFER, so non-redundant BFERs are especially
cheap on the bits (this is accounting. Forwarding reality is of course the
first link into the BFER requires a bit but the BFER itself not).

Another way to save bits i have not discussed is to even have 
transit BFER have a local_decap on the leaf-BFER punt bit - and
expect that the next layer can do a fast-drop for the payload if
the packet is not of interest.  But that of course depends highly
on the payload. Of course if you want to just traffic engineer
eg: for a particular sub-domain some stupid L2 broadcast service,
and you're only interested in the bits for the path engineering
in your core, then a single shared bit in that sub-domain for all
BFER is all you need.

Cheers
    Toerless

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 16 Oct 2015 14:47:33 -0700

I think I'm understanding BIER-TE finally, but I have a few questions to 
confirm my understanding.

How does an application use this?

    In particular, how is this done so the overhead of determining the
    end-to-end path is not
    imposed on the host or BFIR on a per packet basis?

    Presumably the application, or a utility library, translates each
    destination IP address into a
    bit position. This is safe because the mapping of IP address to bit
    position is stable.

    The host or the BFIR *extends* that bitmap to provide additional
    bits which are there for the
    benefit of intermediate forwarders which would not have build a map
    of all destinations for
    their own purposes. Edge BFRs would need to build such a map, but
    core BFRs with no clients
    directly attached would not have to. More importantly than not
    having to build it, they would
    not have to store it.

    What is the stability of these derived bits? Are we preventing
    transitory renumbering of internal
    links somehow? How often does the mapping of a destination IP
    address to intermediate paths
    to be followed hold? Is this something a user-mode BIER utility
    library, or a host stack, would
    have to be involved with?

What portion of the bitmap would be needed for the "derived" bits?

    What is a reasonable ratio between true destination bits and
    intermediate forwarding bits?

    How does this compare to using a TRILL-style encapsulation with the
    outer-BIER header containing
    *only* the forwarding bits and an inner BIER header containing only
    the ultimate destination bits?

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 16 Oct 2015 17:46:36 -0700

Inline.

>    imposed on the host or BFIR on a per packet basis?

I don't think we've done a lot of brainstorming how to do end-2-end BIER
where the two ends are actual applications instead of transport services
gateways, eg: BIER-PE or the like (encap/decap native IP multicast or
MPLS multicast).

Assuming we had end-to-end bier, for BIER proper you'd have a 1:1 mapping
between receiver and a bit in the bitstring, and its a matter of the
API we build whether or not we hide these bitstrings. In BIER-TE you'd have
to have like in other TE solutions some upfront magic that determines the
shape of the overall tree. Eg: PCE controller or so. This is not a BIER-TE
problem, this is a TE problem.

If you want to build something like Steiner trees than the overall tree
would want to change when you add/delete a receiver. If thats the
traffic optimization needed, then worst case the PCE controller would
need to give you for each possible combination of receivers a different
bitstring. Not really difficult, but potentially a bit of upfront
calculation depending on how many receivers you have and how
many subsets are of interest. Again, this alculation would need to be
done however you'd want to do TE, BIER or whatever else. Just the output
being a bitstring is BIER-TE specific.

If the path towards each receiver should not change whether or not
other receivers are included into the tree or not, then it gets
easier: The PCE controller would give you for each candidate reciver a
bitstring, and the actual bitstring you want to use in your BIER
packet is just the OR of the bitstrings for each receiver.

>    transitory renumbering of internal links somehow?

Assignment of bit-indices to links or BFRs or the like is all just
software in the BIER-TE controller software, For path selection it
may consult some external generic PCE controller or perform those
functions itself.

You can reuse a particular bit-index as soon as that bit-index is
not used for anything else in any running BIFT or BFIR adjacency.
This is really only a problem when the network growths. When you
have link-failure, you do not reassign bits, you just need to update
the bitstrings on the BFIR for packets to take alternative paths.

>    How often does the mapping of a destination IP
>    address to intermediate paths  to be followed hold?

IN BIER-TE you don't see the path in the bitstring, and it changes
automatically because of routing. In BIER-TE its fully or partially
visible. Its really just a matter of your policy when/how you want to
change it. Because of varying load ? Link/node failures, network
expansion... 

> Is this something a user-mode BIER utility

If a BIER packet is created by an application level entity then it
needs to insert the right bitstring, but as suggested above it might
consult an external entity like a PCE controller or BIER-TE controller
to get this information. Such interactions have not been defined yet.
But likewise in normal BIER, you'd today need manual config or be
able to retrieve IGP information to map from IP-address to BFR-id.

>    intermediate forwarding bits?

See one of my last emails. Depends on how much traffic engineering you
want/need. I think most deployment should be able to use less than 50%
of bits.

>    the ultimate destination bits?

Depends really on what you want to achieve. Easier to answer based
on an explicit example if you have one...

Cheers
    Toerless

---

From: Eric C Rosen <[email-masked]>
Date: Thu, 29 Oct 2015 13:52:06 -0400

> "shared leaf bit" is just a bit allocation trick, so it keeps the forwarding

I don't really follow this.  Presumably, if a BFR is a leaf BFER in a 
particular sub-domain, it must know that it is a leaf BFER in that 
sub-domain.  Otherwise it wouldn't punt the packet when it sees the 
"shared leaf bit".

If the BFR knows that it is a leaf BFER in a particular sub-domain, then 
when it gets a packet whose top label is the MPLS label it has assigned 
to that sub-domain, it can just punt the packet.  Recognizing that a 
particular MPLS label means "this packet is for me" doesn't seem like a 
new forwarding plane feature.

On the issue of adding new BFERs, suppose you have an existing flow, one 
of whose BFERs is BFR-1.  Now you need to add another BFER to that flow, 
say BFR-2.  Unfortunately, BFR-2 can only be reached using BFR-1 as a 
transit BFR.  So to add BFR-2 as a leaf, you have to turn BFR-1 into a 
non-leaf, which means assigning it a BFR-id, assigning a bit position to 
the link connecting BFR-1 and BFR-2, and modifying the flow's BitString 
appropriately.  I know that the magic central controller is going to 
figure all this out ;-), but can it disseminate the new bit position 
assignments without disrupting the flow that is already in progress? 
After all, you don't want to see a glitch in the TV show you're watching 
every someone else decides to tune in ;-)

One more issue.  The discussion of SI in the -02 revision of the draft 
still doesn't seem right to me.  In BIER-TE, the BitString in a packet 
has to represent the tree over which the packet will flow.  The only way 
I can think of using SI is to divide the tree into n sub=trees, where 
the links and BFERs for each sub-tree all have BFR-ids that fall into 
the same set.  Then the BFIR could make n copies of the packet, unicast 
each one to the root of the corresponding sub-tree, and allow BIER-TE to 
proceed from there.  Is that what you are actually thinking?