Subject: [Bier] comments for draft-zhang-bier-bierin6-04.txt

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Mar 2020 05:24:29 +0100

Comments on subject document inline. Thanks a lot for the work.

> 
> 
> 
> 
> BIER                                                            Z. Zhang
> Internet-Draft                                           ZTE Corporation
> Intended status: Standards Track                           A. Przygienda
> Expires: July 11, 2020                            Juniper Networks, Inc.
>                                                              I. Wijnands
>                                                            Cisco Systems
>                                                               H. Bidgoli
>                                                                    Nokia
>                                                               M. McBride
>                                                                Futurewei
>                                                          January 8, 2020
> 
> 
>                          BIER in IPv6 (BIERin6)
>                       draft-zhang-bier-bierin6-04
> 
> Abstract
> 
>    BIER is a new architecture for the forwarding of multicast data
>    packets.  This document defines native IPv6 encapsulation for BIER
>    hop-by-hop forwarding or BIERin6 for short.

Would suggest to change to something that does not say "native", because
the option described in this document of using ethernet-nonMPLS-BIER-IPv6 
encapsulation might not be seen as "native IPv6 encap" by IPv6
purists.

Propose:

"This document defines encapsulation hop-by-hop BIER forwarding for
IPv6, non-MPLS networks called BIERin6."

> Requirements Language
> 
>    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
>    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
>    document are to be interpreted as described in RFC2119.
> 
> Status of This Memo
> 
>    This Internet-Draft is submitted in full conformance with the
>    provisions of BCP 78 and BCP 79.
> 
>    Internet-Drafts are working documents of the Internet Engineering
>    Task Force (IETF).  Note that other groups may also distribute
>    working documents as Internet-Drafts.  The list of current Internet-
>    Drafts is at https://datatracker.ietf.org/drafts/current/.
> 
>    Internet-Drafts are draft documents valid for a maximum of six months
>    and may be updated, replaced, or obsoleted by other documents at any
>    time.  It is inappropriate to use Internet-Drafts as reference
>    material or to cite them other than as "work in progress."
> 
>    This Internet-Draft will expire on July 11, 2020.
> 
> 
> 
> 
> 
> 
> 
> Zhang, et al.             Expires July 11, 2020                 [Page 1]
> 
> Internet-Draft                   BIERv6                     January 2020
> 
> 
> Copyright Notice
> 
>    Copyright (c) 2020 IETF Trust and the persons identified as the
>    document authors.  All rights reserved.
> 
>    This document is subject to BCP 78 and the IETF Trust's Legal
>    Provisions Relating to IETF Documents
>    (https://trustee.ietf.org/license-info) in effect on the date of
>    publication of this document.  Please review these documents
>    carefully, as they describe your rights and restrictions with respect
>    to this document.  Code Components extracted from this document must
>    include Simplified BSD License text as described in Section 4.e of
>    the Trust Legal Provisions and are provided without warranty as
>    described in the Simplified BSD License.
> 
> Table of Contents
> 
>    1.  Introduction  . . . . . . . . . . . . . . . . . . . . . . . .   2
>    2.  IPv6 Header . . . . . . . . . . . . . . . . . . . . . . . . .   3
>      2.1.  IPv6 Options Considerations . . . . . . . . . . . . . . .   3
>    3.  BIER Header . . . . . . . . . . . . . . . . . . . . . . . . .   4
>    4.  IPv6 Encapsulation Advertisement  . . . . . . . . . . . . . .   4
>      4.1.  Format  . . . . . . . . . . . . . . . . . . . . . . . . .   4
>      4.2.  Inter-area prefix redistribution  . . . . . . . . . . . .   5
>    5.  IANA Considerations . . . . . . . . . . . . . . . . . . . . .   5
>    6.  Security Considerations . . . . . . . . . . . . . . . . . . .   5
>    7.  Acknowledgement . . . . . . . . . . . . . . . . . . . . . . .   5
>    8.  References  . . . . . . . . . . . . . . . . . . . . . . . . .   5
>      8.1.  Normative References  . . . . . . . . . . . . . . . . . .   6
>      8.2.  Informative References  . . . . . . . . . . . . . . . . .   6
>    Authors' Addresses  . . . . . . . . . . . . . . . . . . . . . . .   7
> 
> 1.  Introduction
> 
>    BIER [RFC8279] is a new architecture for the forwarding of multicast
>    data packets.  It provides optimal forwarding through a "multicast
>    domain" and it does not necessarily precondition construction of a
>    multicast distribution tree, nor does it require intermediate nodes
>    to maintain any per-flow state.
> 
>    This document specifies non-MPLS BIER forwarding in an IPv6 [RFC8200]
>    environment, refferred to as BIERin6, using non-MPLS BIER
>    encapsulation specified in [RFC8296].

This version of the sentence could also go into the abstract ;-)

>    MPLS BIER forwarding in IPv6 is outside the scope of this document.
> 
>    This document uses terminology defined in [RFC8279] and [RFC8296].
> 
> 
> 
> 
> Zhang, et al.             Expires July 11, 2020                 [Page 2]
> 
> Internet-Draft                   BIERv6                     January 2020
> 
> 
>    [RFC8296] defines the BIER encapsulation format in MPLS and non-MPLS
>    environment.  In case of non-MPLS environment, a BIER packet is the
>    payload of an "outer" encapsulation, which has a "next protocol"
>    codepoint that is set to a value that means "non-MPLS BIER".
> 
>    That can be used as is in a pure IPv6 non-mpls environment.  Between
>    two directly connected BFRs, a BIER header could directly follow link
>    layer header, e.g., an Ethernet header (with the Ethertype set to
>    0xAB37).  If a BFR needs to tunnel BIER packets to another BFR, e.g.
     ^^^^^^^^

Please add reference "according to RFC8296 section 2.2.3.

>    per [RFC8279] Section 6.9,

It would be worth IMHO to describe here that section RFC8279 6.9 describes
mechanisms to support partial BIER deployment by automatically calculating
non-adjacent downstream BFR and then uses tunneling to them to hop
over the non-BIER capable routers.

>    IPv6 encapsulation can be used, with the
>    destination address being the downstream BFR and the Next Header
>    field set to a to-be-assigned value for "non-MPLS BIER".

Would be clearer to state that RFC8279 does not prescribe an IPv6
next header type to encapsulate BIER into IPv6, but that this document
does this and uses the name "non-MPLS BIER" for it.

Also RFC8296 already uses the term "non-MPLS BIER" for Ethertype
0xAB37, so it would be an overload (*gasp* ;-) of that term to
also use it for a TBD IPv6 Next Header codepoint. Maybe come up with
another name for the TBD codepoint or use "non-MPLS BIER Ethertype"
vs. "non-MPLS BIER IPv6 NH".

>    The IPv6 encapsulation could be used even between two directly
>    connected BFRs in the following two cases:
> 
>    o  An operator mandates all traffic to be carried in IPv6.

An example (like on belows point) would help to motivate this.

IPv6-only forwarding plane features such as monitoring or filtering come to mind.

>    o  A BFR does not have BIER support in its "fast forwarding path" and
>       relies on "slow/software forwarding path", e.g. in environments
>       like [RFC7368] where high throughput multicast forwarding
>       performance is not critical.
> 
> 2.  IPv6 Header
> 
>    Whenever IPv6 encapsulation is used for BIER forwarding, The Next
>    Header field in the IPv6 Header (if there are no extension headers),
>    or the Next Header field in the last extension header is set to TBD,
>    indicating that the payload is a BIER packet.

Maybe use numbered TBD (TBD1) and introduce that TBD1 above where you first
talk about it.

>    If the neighbor is directly connected, The destination address in
>    IPv6 header SHOULD be the neighbor's link-local address on this
>    router's outgoing interface, the source destination address SHOULD be
>    this router's link-local address on the outgoing interface, and the
>    IPv6 TTL MUST be set to 1.

I've started a thread on 6man/6ops re. link-local-tunnel traffic as i
have the same also in my ANIMA draft, and in implementations i have
seen quite some issues tunneling traffic HW accelerated across IPv6 
link-local addresses and TTL=1.

Would welcome implementers in BIER to chime in if they would see this
as an issue.

If folks feel supporting this HW accelerated might be more complicated
than non-link-local tunnels, then it might be useful to change the
text to also allow/require support for using the BFR address even for
subnet adjacent BFR. In that case one would not use HL=1, and that
too i have seen to be a problem in HW accelerated forwarding plane:
HL/TTL=1 -> MUST punt (software processing).

>    Otherwise, the destination address SHOULD
>    be the BIER prefix of the BFR neighbor, the source address SHOULD be
>    this router's BIER prefix, and the TTL MUST be large enough to get
>    the packet to the BFR neighbor.

(Hop-limit, not TTL, MLD, not IGMP, everything gets better by new names ;-).

>    The Flow-ID in the IPv6 packet SHOULD be copied from the entropy
>    field in the BIER encapsulation.
> 
> 2.1.  IPv6 Options Considerations
> 
>    RFC 8200 section 4, defines the IPv6 extension headers.  Currently
>    there are two defined extension headers, Hop-by-Hop and Destination
>    options header, which can carry a variable number of options.  These

draft-ietf-6man-segment-routing-header is Routing Header, not Hop-by-Hop or Destination.

>    extension headers are inserted by the source node.
> 
> 
> 
> Zhang, et al.             Expires July 11, 2020                 [Page 3]
> 
> Internet-Draft                   BIERv6                     January 2020
> 
> 
>    For directly connected BIER routers, IPv6 Hop-by-Hop or Destination
>    options are irrelevant and SHOULD NOT be inserted by BFIR on the
>    BIERin6 packet.  In this case IPv6 header, Next Header field should
>    be set to TBD.  Any IPv6 packet arriving on BFRs and BFERs, with
>    multiple extension header where the last extension header has a Next
>    Header field set to TBD, SHOULD be discard and the node should
>    transmit an ICMP Parameter Problem message to the source of the
>    packet (BFIR) with an ICMP code value of TBD10 ('invalid options for
>    BIERin6').
> 
>    This also indicates that for disjoint BIER routers using IPv6

disjoint == Non subnet-adjacent ?

>    encapsulation, there SHOULD NOT be any IPv6 Hop-by-Hop or Destination
>    options be present in a BIERin6 packet.

This is a bit confusing, first you write what seems to only apply to
adjacent BFR, then in the next paragraph you seem to expand it to 
non-adjacent BFR. Maybe rewrite so it says you do not like hop-by-hop
options and destination options whether the BFR are adjacent or not ?

Don't have a strong opinion content wise. In general,
this is kinda safe and may make implementations easier, and can always
be superceeded by anothre RFC when there is an interesting option
in he future we would like to use with this encap. On the other hand, if/when
there is such an option header, there MUST be another RFC to make it
applicable to this RFC (by overriding it).

>    In this case, if additional
>    traffic engineering is required, IPv6 tunneling (i.e.  BIERin6 over
>    SRv6) can be implemented.

Given how the model described in this draft (source address is the sending BFR, not
the BFIR), is really end-to-end BIER (not end-to-end IPv6), i think 
each (adjacent, non-adjacent) BIER-hops IPv6 encapsulation is indeed IPv6
tunneling, so it is confusing/incorrect to use the term "IPv6 tunneling"
here only in the context of SRv6.

I would propose to improve the text to introduce the notion of "IPv6 tunneling"
earlier and explain at the point earlier where you start to dsicuss
option headers, that routing-headers are fine. 

If you want to use a much longer used form of IPv6 routing header than
SRv6 SRH, use the RPL Source Routing Header (RFC6554). Its also much more
widely deployed (millions of routers)

> 3.  BIER Header
> 
>    The BIER header MUST be encoded per Section 4.2 of [RFC8296].
> 
>    The BIFT-id is either encoded per
>    [I-D.ietf-bier-non-mpls-bift-encoding] or per advertised by BFRs, as
>    specificed in [I-D.dhanaraj-bier-lsr-ethernet-extensions].
> 
> 4.  IPv6 Encapsulation Advertisement

Seems like it would help to restructure 2,3 under forwarding plane and 4
a=under control/routing-plane sections ?

Also: Didn't i remember something from Alia saying that future control plane
work should be done in LSR ? I may be wrong. Just wondering if this section
can still be in this draft. Would make life easier, but we're still IETF.. ;-)

>    When IPv6 encapsulation is not required between directly connected
>    BFRs, no signaling in addition to that specified in
>    [I-D.dhanaraj-bier-lsr-ethernet-extensions] is needed.
> 
>    Otherwise, a node that requires IPv6 encapsualtion MUST advertise the
>    BIER IPv6 transportation sub-TLV/sub-sub-TLV according to local
>    configuration or policy in the BIER domain to request other BFRs to
>    always use IPv6 encapsulation.
> 
>    In presence of multiple encapsulation possibilities hop-by-hop it is
>    a matter of local policy which encapsulation is imposed and the
>    receiving router MUST accept all encapsulations that it advertised.
> 
> 4.1.  Format
> 
>    The BIER IPv6 transportation is a new sub-TLV of BIER defined in OSPF
>    [RFC8444], and a new sub-sub-TLV of BIER Info sub-TLV defined in ISIS
>    [RFC8401].
> 
> 
> 
> 
> 
> 
> 
> Zhang, et al.             Expires July 11, 2020                 [Page 4]
> 
> Internet-Draft                   BIERv6                     January 2020
> 
> 
>          0                   1                   2                   3
>          0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>         |    Type       |   Length      |
>         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
>    o  Type: For OSPF, value TBD1 (prefer 12) is used to indicate it is
>       the IPv6 transportation sub-TLV.  For ISIS, value TBD2 (prefer 3)
>       is used to indicate it is the IPv6 transportation sub-sub-TLV.
> 
>    o  Length: 0.

Not quite sure about the policies (others in the WG will know better):
Should this document be "Updates" to those two RFCs ?

If i where to implement ISIS/OSPF for BIER (or troubleshoot it on the wirese,
write wireshark modules, etc. pp), marking this document an
update would help to find it, but technically, this is just an extension,
not changing anything, so i am not sure if it even can be an update.

The other point is whether or not to distinguish if link-local or
BFR address are supported for IPv6 encapsulation. Given my past pain
with IPv6 link-local HW support, i would certainly like to see
the ability to announce what amounts to "this node can only encap/decap
for its BFR address, but not link-local - but please send me also
to that address if you're subnet adjacent".

> 4.2.  Inter-area prefix redistribution
> 
>    When BFR-prefixes are advertised across IGP areas per
>    [I-D.dhanaraj-bier-lsr-ethernet-extensions] or redistributed across
>    protocol boundaries per [I-D.zwzw-bier-prefix-redistribute], the BIER
>    IPv6 transportation sub-TLV or sub-sub-TLV MAY be re-advertised/re-
>    distributed as well.

Ok, what is missing up to here is an upfront explanation that this draft is 
end-to-end (BFIR to BFER) BIER with hop-by-hop L2 encap and/or
IPv6 tunnels. And discuss the implications. For example, you will not
get ICMP error messages all the way back to the BFIR as you would get
in a model where forwarding was hop-by-hop IPv6 (plus BIER). I have
no opinions how relevant this is, but today if an SP does end-to-end
IPv6, then he would get forwarding errors back to the encap router in
his network (assuiming that customer traffic is IPv6 encapsulated there).
Likewise, there is no end-to-end PMTUD from IPv6 supported in this model

(which i think is stupid anyhow for multicast, so i am not saying this
 as an argument against this method, just to make sure that users coming
 from IPv6 understand all the implications).

> 5.  IANA Considerations
> 
>    IANA is requested to assign a new "BIER" type for "Next Header" in
>    the "Assigned Internet Protocol Numbers" registry.

BIERin6 ?

>    IANA is requested to assign a new "BIERin6" type for "invalid
>    options" in the "ICMP code value" registry.
> 
>    IANA is requested to assign a new "BIER IPv6 transportation Sub-TLV"
>    type in the "OSPFv2 Extended Prefix TLV Sub-TLVs" Registry.

I would prefer "tunnel" instead of transport given how a new IPv6
header is added.

>    IANA is requested to set up a new "BIER IPv6 transportation Sub-sub-
>    TLV" type in the "IS-IS BIER Info sub-TLV" Registry.

same here.

> 6.  Security Considerations
> 
>    General IPv6 and BIER security considerations apply.

I am sure someone will have more security consideration ideas, but not me, not today . ;-)

> 7.  Acknowledgement
> 
>    The authors would like to thank Jeffrey Zhang for his review and
>    valuable contributions.
> 
> 8.  References
> 

Thanks a lot for the document

Toerless
> 
> 
> 
> 
> 
> 
> Zhang, et al.             Expires July 11, 2020                 [Page 5]
> 
> Internet-Draft                   BIERv6                     January 2020
> 
> 
> 8.1.  Normative References
> 
>    [RFC8200]  Deering, S. and R. Hinden, "Internet Protocol, Version 6
>               (IPv6) Specification", STD 86, RFC 8200,
>               DOI 10.17487/RFC8200, July 2017,
>               <https://www.rfc-editor.org/info/rfc8200>.
> 
>    [RFC8279]  Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
>               Przygienda, T., and S. Aldrin, "Multicast Using Bit Index
>               Explicit Replication (BIER)", RFC 8279,
>               DOI 10.17487/RFC8279, November 2017,
>               <https://www.rfc-editor.org/info/rfc8279>.
> 
>    [RFC8296]  Wijnands, IJ., Ed., Rosen, E., Ed., Dolganow, A.,
>               Tantsura, J., Aldrin, S., and I. Meilik, "Encapsulation
>               for Bit Index Explicit Replication (BIER) in MPLS and Non-
>               MPLS Networks", RFC 8296, DOI 10.17487/RFC8296, January
>               2018, <https://www.rfc-editor.org/info/rfc8296>.
> 
>    [RFC8401]  Ginsberg, L., Ed., Przygienda, T., Aldrin, S., and Z.
>               Zhang, "Bit Index Explicit Replication (BIER) Support via
>               IS-IS", RFC 8401, DOI 10.17487/RFC8401, June 2018,
>               <https://www.rfc-editor.org/info/rfc8401>.
> 
>    [RFC8444]  Psenak, P., Ed., Kumar, N., Wijnands, IJ., Dolganow, A.,
>               Przygienda, T., Zhang, J., and S. Aldrin, "OSPFv2
>               Extensions for Bit Index Explicit Replication (BIER)",
>               RFC 8444, DOI 10.17487/RFC8444, November 2018,
>               <https://www.rfc-editor.org/info/rfc8444>.
> 
> 8.2.  Informative References
> 
>    [I-D.dhanaraj-bier-lsr-ethernet-extensions]
>               Dhanaraj, S., Wijnands, I., Psenak, P., Zhang, Z., Yan,
>               G., and J. Xie, "LSR Extensions for BIER over Ethernet",
>               draft-dhanaraj-bier-lsr-ethernet-extensions-00 (work in
>               progress), January 2019.
> 
>    [I-D.ietf-bier-bar-ipa]
>               Zhang, Z., Przygienda, T., Dolganow, A., Bidgoli, H.,
>               Wijnands, I., and A. Gulko, "BIER Underlay Path
>               Calculation Algorithm and Constraints", draft-ietf-bier-
>               bar-ipa-06 (work in progress), November 2019.
> 
>    [I-D.ietf-bier-idr-extensions]
>               Xu, X., Chen, M., Patel, K., Wijnands, I., and T.
>               Przygienda, "BGP Extensions for BIER", draft-ietf-bier-
>               idr-extensions-07 (work in progress), September 2019.
> 
> 
> 
> Zhang, et al.             Expires July 11, 2020                 [Page 6]
> 
> Internet-Draft                   BIERv6                     January 2020
> 
> 
>    [I-D.ietf-bier-non-mpls-bift-encoding]
>               Wijnands, I., Xu, X., and H. Bidgoli, "An Optional
>               Encoding of the BIFT-id Field in the non-MPLS BIER
>               Encapsulation", draft-ietf-bier-non-mpls-bift-encoding-02
>               (work in progress), August 2019.
> 
>    [I-D.zhang-bier-babel-extensions]
>               Zhang, Z. and T. Przygienda, "BIER in BABEL", draft-zhang-
>               bier-babel-extensions-02 (work in progress), November
>               2019.
> 
>    [I-D.zwzw-bier-prefix-redistribute]
>               Zhang, Z., Bo, W., Zhang, Z., and I. Wijnands, "BIER
>               Prefix Redistribute", draft-zwzw-bier-prefix-
>               redistribute-03 (work in progress), September 2019.
> 
>    [RFC7368]  Chown, T., Ed., Arkko, J., Brandt, A., Troan, O., and J.
>               Weil, "IPv6 Home Networking Architecture Principles",
>               RFC 7368, DOI 10.17487/RFC7368, October 2014,
>               <https://www.rfc-editor.org/info/rfc7368>.
> 
> Authors' Addresses
> 
>    Zheng(Sandy) Zhang
>    ZTE Corporation
> 
>    EMail: [email-masked]
> 
> 
>    Tony Przygienda
>    Juniper Networks, Inc.
> 
>    EMail: [email-masked]
> 
> 
>    IJsbrand Wijnands
>    Cisco Systems
> 
>    EMail: [email-masked]
> 
> 
>    Hooman Bidgoli
>    Nokia
> 
>    EMail: [email-masked]
> 
> 
> 
> 
> 
> 
> Zhang, et al.             Expires July 11, 2020                 [Page 7]
> 
> Internet-Draft                   BIERv6                     January 2020
> 
> 
>    Mike McBride
>    Futurewei
> 
>    EMail: [email-masked]
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Zhang, et al.             Expires July 11, 2020                 [Page 8]

---

From: <[email-masked]>
Date: Thu, 12 Mar 2020 22:09:26 +0800 (CST)

Hi Toerless,

Thank you very much for your detailed comments! :-)

Very glad that more and more people are interested in BIERin6 work. 

We will discuss it and reply to you carefully.

Thank you again for your help! :-)

Best regards,

Sandy

原始邮件

发件人：ToerlessEckert <[email-masked]>
收件人：[email-masked] <[email-masked]>;
日 期 ：2020年03月12日 12:24
主 题 ：[Bier] comments for draft-zhang-bier-bierin6-04.txt

Propose:

     ^^^^^^^^

BIERin6 ?

same here.

Toerless
> Zhang, et al.             Expires July 11, 2020                 [Page 8]

_______________________________________________
BIER mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/bier

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 12 Mar 2020 08:55:47 -0700

> HL/TTL=1 -> MUST punt (software processing).

really? I'm surprised, actually, LL with TTL=1 for link local is an
absolutely common case since years for e.g. BFD and other thinks like BGP
sessions in many topologies so if a silicon does not support that it's a
strange IPv6 silicon. But yes, supporting LL properly in IPv6 is non
trivial (spec allows to have e.g. same LL on multiple interfaces in a
node). The spec here says SHOULD and not MUST so it's perfectly find using
any addresses (one could probably even use BFR prefix but that would lead
to ECMP non-deterministic behavior which may or may not be OK depending on
deployment).  If that needs to be spelled out (albeit it's redundant) I'm
not opposed to that.

Please cc: bier working group on any discussion you are starting with 6man
on the side ... Unless I missed my morning mail input ...

--- tony

---

From: "Nagendra Kumar Nainar (naikumar)" <[email-masked]>
Date: Thu, 12 Mar 2020 16:30:54 +0000

Hi Toerless,

Thank you for triggering the discussion with 6MAN. I was discussing with the authors offline on the same topic. I think it is better to have a clarity on this topic.

I personally think it is better to use non LL address even when the BFRs are directly connected for behavior consistency and to leverage some of the benefits such as IP-FRR.

Thanks,
Nagendra

From: BIER <[email-masked]> on behalf of Tony Przygienda <[email-masked]>
Date: Thursday, March 12, 2020 at 11:58 AM
To: Toerless Eckert <[email-masked]>
Cc: BIER WG <[email-masked]>
Subject: Re: [Bier] comments for draft-zhang-bier-bierin6-04.txt

really? I'm surprised, actually, LL with TTL=1 for link local is an absolutely common case since years for e.g. BFD and other thinks like BGP sessions in many topologies so if a silicon does not support that it's a strange IPv6 silicon. But yes, supporting LL properly in IPv6 is non trivial (spec allows to have e.g. same LL on multiple interfaces in a node). The spec here says SHOULD and not MUST so it's perfectly find using any addresses (one could probably even use BFR prefix but that would lead to ECMP non-deterministic behavior which may or may not be OK depending on deployment).  If that needs to be spelled out (albeit it's redundant) I'm not opposed to that.

Please cc: bier working group on any discussion you are starting with 6man on the side ... Unless I missed my morning mail input ...

--- tony

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 12 Mar 2020 11:22:08 -0700

Nagendra, IP-FRR is an interesting observation. There is no reason why LL
cannot be protected by another non-LL nexthop but it may not be as natural
as otherwise.

As I said, no problem in softenting the draft from SHOULD to something else
as long we allow both LL and non-LL addressing. The driver to LL & NHOP=1
was mainly to have people misunderstand that BIER can be forwarded all over
the place by default which will lead to a valley of tears long term albeit
it looks like a great idea first. BIER is a switching fabric technology @
L2.5 and not an overlay (and yes, it can be used that way but this is IMO
jsut as far from BIER as BESS is from idr ;-)

--- tony

On Thu, Mar 12, 2020 at 9:30 AM Nagendra Kumar Nainar (naikumar) <
[email-masked]> wrote:

> Hi Toerless,
>
>
>
> Thank you for triggering the discussion with 6MAN. I was discussing with
> the authors offline on the same topic. I think it is better to have a
> clarity on this topic.
>
>
>
> I personally think it is better to use non LL address even when the BFRs
> are directly connected for behavior consistency and to leverage some of the
> benefits such as IP-FRR.
>
>
>
> Thanks,
>
> Nagendra
>
>
>
> *From: *BIER <[email-masked]> on behalf of Tony Przygienda <
> [email-masked]>
> *Date: *Thursday, March 12, 2020 at 11:58 AM
> *To: *Toerless Eckert <[email-masked]>
> *Cc: *BIER WG <[email-masked]>
> *Subject: *Re: [Bier] comments for draft-zhang-bier-bierin6-04.txt
>
>
>
>
>
>

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Mar 2020 19:49:04 +0100

On Thu, Mar 12, 2020 at 08:55:47AM -0700, Tony Przygienda wrote:
> strange IPv6 silicon.

Even BFD is control plane, albeit a special case.

I was talking about forwarding plane, HW accelerated.

Check out if you can configure any type of tunnel (IPinIP, GRE, IPsec VTI,...)
on your router of choice with local and remote tunnel endpoint addresses being
link-local and let me know if it not only works but actually does not
get punted.

And then assign the same LL addresses on multiple interfaces
and run a tunnel across each of them. I would bet the CLI
might not even allow to enter the addresses correctly as
<ipv6addr>%<interface-name>

Last year i was struggling to figure out how to make those
link-local tunnels (ANIMA) work in *Swan on linux for IPsec tunnels.
Nightmare... (thats not even hardware).

This is of course all just lame industry stuff, in a perfect
world this shouldn't be a problem, but the real point is that
there is (IMHO) really no functional benefit using the link-local
addresses. 

> But yes, supporting LL properly in IPv6 is non

Tell me why we would not always want to use the global addresses.
(Some IPv6 archicture made us think we should use loopback for link-local communications ?)

In ANIMA we need to build link-local tunnels in the absence of
pre-existing routing for global node addresses. Hence we learned
about all the implementation issues.  I can't see how BIER is
working without the routes to the BIER prefixes, so why not use
them as the tunnel addresses to avoid unnecessary real-world problems.

Now, if someone comes back and says that the route-lookups for
global addresses will result in a less performant implementation
than link-locals which do not require another lookup, or anything
else of real technical value in support of the link-local addresses,
then i would be all in for also including them as an option,
but i would maintain my caution, that the MUST support should be
on th e safer bet, the global addresses.

> on the side ... Unless I missed my morning mail input ...

Isn't there a policy that one should be beaten up if doing too
much cross-posting ?

Toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Mar 2020 20:02:30 +0100

On Thu, Mar 12, 2020 at 11:22:08AM -0700, Tony Przygienda wrote:
> as long we allow both LL and non-LL addressing.

> The driver to LL & NHOP=1
> it looks like a great idea first.

parsing failed. Can you plsease rephrase ?

> BIER is a switching fabric technology @

See my other discuss point about the IMHO most 'useful' modes of
deployment/forwarding. Those would be good to discuss in the drafts.

Partial or full deployment of BIER with IPv6 encap:
  BIFR-to-BFER IPv6 encapsulated, per-BIER-hop rewrite of IPv6 header
  No differnce between partial or full deployment, 
     should work automatically calculating remote next-hop in partial.

Partial L2 BIER encap support
  Recommend not to use, but use IPv6 encap, as IPv6 encap/decap might more likely
  be impacting performance.

Full L2 BIER encap support
  determined by BFIR from IGP info, then ue L2 encap.

Cheers
  Toerless

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 12 Mar 2020 12:23:13 -0700

> parsing failed. Can you plsease rephrase ?

OK, if you use LL and TTL=1 you basically won't manage to send to wrong
link or more than one hop so it forces a link-by-link L2.5 fabric which is
BIER architecture day one on standard RFC (+ exception allowed to _bypass
non-BIER supporting nodes_). As I said, the spec is NOT MUST and should not
be. We can discuss out whether we should bput "SHOULD" with BIER prefix
(I'm kind of leery of v6 global addressing since that's not always given)
but then we need to talk how we deal with parallel links (or maybe we don't
have to given BIER only addresses whole BFR with a bit, I kind of didn't do
enough careful thinking here yet).

>      should work automatically calculating remote next-hop in partial.

There is somehow a huge misconception that as intermediate node you can
"rewrite headers and options on v6 while keeping the source address"in
flight and "adresses can encode my uncle's birthday" because "some people
said so now" but the ultimately governing body are v6 working groups not
us. In fact, AFAIR v6 specs you can't, almost without exceptions (see the
raging discussions on SRv6 PHP, SRv6 basically breaking the "no v6 address
has semantics behind subnetting", mucking around with options they have no
business to muck with and so on). BIER LL, TTL=1 is one hop forwarding
where v6 is acting as L2 basically and not some end-2-end forwarding
transport with semantics encoding.  If you start to shove BIER frames
around tunnels to show up in arbitrary places it's like shoving MPLS frames
to weird places and expect "labels to work". A BIER bitmask is basically a
very weird label with really only local significance on the link and not
using it correctly will make you expose BIER mask in places where they do
the wrong stuff (just like MPLS labels). Yes, bypassing non-supporting
nodes is ok if we control the semantics of that tightly but everything else
is basically brfeaking the architecture with a long tail fallout. BIER is
carefully thought out and reviewed and lots of stuff has security/OAM/etc
implications and that's what the WG is chartered to govern and not come up
with some "fashionable hacks" with the acronyms du jour.

And yes, you can use BIER "in overlay" but that is not the current charter
of BIER WG as many other things are not. Or that would be a "BIER-BESS"
group I guess ...

And yes, "it's lame industry stuff" you're quoting, we have v6 standards
track documents signed of and 6man to govern all that stuff and arguments
along the lines "but my silicon doesn't do that" or "this implementation is
lousy" I'm sympathetic to but to quote Yakov here "I cannot prevent you
doing whatever you want to do in your privacy with the spec I'm publishing
but do not ask me to standardize your stuff if it does not follow the
governing architecture and needs interoperability".

And yes, please cross-post, I don't want 6man or spring to be led down some
garden path without being plugged in into the BIER working group where the
BIER technology is governed.

--- tony

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 12 Mar 2020 12:26:41 -0700

> governing architecture and needs interoperability".

and not to be missed we're playing the card to certain extent towards
HOMENET and the irony of that doesn't bypass me but BIER support on 50c
home silicon in wi-fi routers will never happen and if it happens it will
happen over proper L2 encaps ...

--- tony

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 12 Mar 2020 21:29:09 +0100

On Thu, Mar 12, 2020 at 12:23:13PM -0700, Tony Przygienda wrote:
> non-BIER supporting nodes_).

Sure, and we have L2 encap for that and should promote it and make sure
it will happen in IPv6 deployments.

If LL IPv6 tunnels in forwarding plane would be used widely in othre IETF
technologies i would be less concerned. Right not it is not, and
so i am worried to have an IPv6 encap profile in our draft that makes
adoption most easy.

> flight

Well, thats what SRH does. BIER also rewrites bitstring.
rewriting IPv6 dest and optional source should in general
be easier than actual decap+encap. Tell me if i am wrong.

> and "adresses can encode my uncle's birthday" because "some people
> said so now"

Where where you when embedded RP was standardized so that you can throw stones ? ;-)

My concern about  SRH is that it is waste on midpoint SIDs not that 
abuse of IPv6 addresses is bad.

I liked Pierre's bitstring in IPv6 addr, i just felt it was too short to be useful.

I liked terrastreams QoS encoding in IPv6 addresses.

If you want people to stop (ab)using IPv6 addresses, you need
to give operators other header elements in which to carry policy.
Flexible extensible not only by vendors.

I tried to bring frameworks like this to IETF and was immediatly
confronted by the IETF perpass/privacy lobby: Routers should not know
anything.

Ok, now i've run miles away from subject matter to my favourite soap box.

> but the ultimately governing body are v6 working groups not
> business to muck with and so on).where v6 is acting as L2 basically and not
> some end-2-end forwarding transport with semantics encoding.

RFC8200 was written for uncontrolled internet paths, SRH/SRv6 was designed for
controlled networks. Huge overhead as a result. If not for
(predecessors) of RFC8200 would have from the beginning been something
like CRH (which still has to argue representing IPv6 adddresses with
mapping to suite RFC8200).

RFC8200 is a plane to travel the planet. No, its not ideal to drive on 
highways, but nobody in the IETF wants to build a car. IPng process was
30 years ago, and we still think its results is sufficient for all
futures.

> If you start to shove BIER frames
> the wrong stuff (just like MPLS labels).

I don't buy that argument. All VPNs are overlay networks (PE to PE), and
operators know how to build them. Most L3 router networks are overlay
networks on top of an optical switched underlay providing p2p colored
connections, and those paths are often manually configured, similar to
what i did 30 years ago with X.25 PVCs. All 3G/4G/5G "core" networks
are overlays. Building overlays happens everywhere. Some are better,
some are worth, often in the eye of the beholder.

> Yes, bypassing non-supporting

I think the best thing BIER can do is to promote the option of
automatic mapping from a self-intelligent underlay, aka: the
fully automatic IGP option described in rfc8297. 

I don't know how important partial upgrades/adoption of BIER within an
IGP domain is, but i have seen a lot of asks for this for prior IP
multicast technologies (especially in SPs), and it is so much easier now
with BIER given how we are relying on autoconfiguration via IGP
information. That makes it excitingly more attractive than the
problematic solutions we tried to build earlier where we where not
given the option to exten IGPs.

> of BIER WG as many other things are not. Or that would be a "BIER-BESS"

Ethernet is also a useless overlay for p2p links. Can we please
only support HDLC headers ? SRv6 whether RH or CRH is always
an overlay. 

If we come up with a solution that allows smooth incremental adoption
of BIER in an IGP domain by using IPv6 encap as long as there are
no full ethernet encap paths, and then automatically switch to  ethernet
encap paths unless there is a policy override ("operator loves IPv6
encap or reuse IPv6 FRR), i think we are on a good path for the first
round document.

The main issue i see is tht most multicst use is in VPNs, and i am
not sure we have a good VPN solution without MPLS without some
way to encapsulate destination SIDs like we have in MPLS.

> governing architecture and needs interoperability".

Its also bad to publish supposedly well-meaning architectures when
you're not the one having to suffer on its issues. Should i bring on
my rant about IPv6 scoped address architecture ? None of the folks who
wrote that ever thought about IPv6 multicast.

At best, architectures are good intentions. If they are assumed to be
cast in stome they become dogmas.

> BIER technology is governed.

Input from othre WGs is only that: Input. As i said, we do need to be
worried about them claiming they can prohibit options we as BIER like
to do - because of dogmas.

Toerless

-- 
---
[email-masked]

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 12 Mar 2020 13:59:54 -0700

Hi Tony,
I think that the example of LL BFD, i.e. single-hop BFD, is in fact
opposite, not in support of TTL/HC = 1. RFC 5881, following on the
extensive discussion of the GTSM in RFC 5082, requires that TTL/HC for the
single-hop BFD be set to 255 and BFD packet discarded by a receiver if
TTL/HC != 255. I'll note that we had an extensive discussion with IESG on
the TTL/HC value in the case of BFD over VXLAN. We've agreed that a VXLAN
tunnel is as single-hop IP and thus TTL/HC MUST be set to 255. Not sure if
all this information was helpful. My $.02

Regards,
Greg

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 12 Mar 2020 14:26:09 -0700

I'm all for TTL=1. It's one thing having a BFD frame run away, another a
multicast frame ;-)  But yeah, worth discussion on the list ...

--- tony

On Thu, Mar 12, 2020 at 2:00 PM Greg Mirsky <[email-masked]> wrote:

>> Please cc: bier working group on any discussion you are starting with
>> 6man on the side ... Unless I missed my morning mail input ...
>>
>> --- tony
>>
>>
>>
>>

---

From: Tony Przygienda <[email-masked]>
Date: Thu, 12 Mar 2020 14:31:52 -0700

Good rant but reread my Yakov quote. Yes, people will always love shortcuts
and have ideas for their golden use case, this has little to do with the
fact whether IETF should standardize it.

As side input: _All_ silicon peopleI I talked to (well, maybe I just talked
to sane silicon people ;-) considered decaps'ing far more desirable than
option processing, no matter whether it's v4 or v6. They like to end up
with a BIER frame they can throw @ BIER MPLS/non-MPLS block (and the
silicon for tunnel encaps/decaps is very good & generic on all sane
silicons AFAIU). Option processing is a mild nightmare, especially if
you're not sure where they are and how big they get and once they blow you
scratchpad size you end up re'circ the packet and pay 50% bandwidht penalty
(or you double your scratchpad size paying a completley non-linear cost in
real estate and power consumption) ... just my 2c but those guys are tied
by electrons and mostly have strong opinions based on atoms they shuffle
around ...

--- tony

---

From: Toerless Eckert <[email-masked]>
Date: Fri, 13 Mar 2020 01:12:04 +0100

Inline

> to sane silicon people ;-) considered decaps'ing far more desirable than

Who is sane, the silicon or the people ? ;-)

> silicons AFAIU).

That's fine. Thats independent of the possible optimization that you
probably can do re-encap easier when you already had the same header
before decap (just moving pointer forward to BIER header and the
backward later).

> Option processing is a mild nightmare, especially if

I'l compress that into "silicon hates variable offsets it need to calculate
through lookups".

Yes, i hate that limitation and want to get it changed, but agree that
for todays reality its good to live with it. Actually i do like to
see this as part of explanation in our document.

The idea from xie's draft that we could actually consider forcing
compliance of our defined header from that semantic being attached
to the desstination address, handling it like a SID would IMHO
actually be the best way to make life for this silicon easier:

The FIB lookup for this destination address immediately implies that
this the next processing is BIER header at offset <static>. Maybe
only as an optimization option.

This would still allow us to use either NH (sandy) or DO (xie) for
the BIER header.

The bigger issue is how we support the VPN solutions IMHO. Ideally
we would allow for an SRH/CRH header that would be processed
after BFER processing, to provide the additional context / processing
SIDs to the application on the BFER. But how could we place it
so that it doesn't impact the efficient BIER forwarding. Without
creating a third IPv6 header...

Cheers
    Toerless

-- 
---
[email-masked]

---

From: "IJsbrand Wijnands (iwijnand)" <[email-masked]>
Date: Tue, 17 Mar 2020 14:00:15 +0000

Hi Toerless,

Picking up on this thread, little late, sorry..

> I liked Pierre's bitstring in IPv6 addr, i just felt it was too short to be useful.

It would disagree, it’s about half the size of what we consider to be the standard interoperable mask of 256. We have defined the Set approach to overcome BM length limitations, at the expense of more copies sent by the ingress. Pun intended, BIER-TE is much more restricted to be useful 😉

Thx,

Ice.

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 17 Mar 2020 17:38:00 +0100

On Tue, Mar 17, 2020 at 02:00:15PM +0000, IJsbrand Wijnands (iwijnand) wrote:
> It would disagree, it???s about half the size of what we consider to be the standard interoperable mask of 256. We have defined the Set approach to overcome BM length limitations, at the expense of more copies sent by the ingress. Pun intended, BIER-TE is much more restricted to be useful ????

No pun needed. Valid point:

With BIER-TE, the goal is to get tree engineering with bitstrings, 
so we can get per-packet trees, and we think there will be good use-cases
too in more constrained environments, such as broadband access (see
our http draft) as well as in 'DetNet' type manufacturing and the like,
where hopefully the forwarders (due to lower speeds than 100Gbps++)
would be easier to support longer bitstrings as needed. And the length
of bitstrings is not nailed in stone forever like in IPv6, so
different markets could evolve to support different (longer) length.
The 256 is really todays lowest common denominator that we can
support in all SP edge/core-routers.

With IPv6 via bitstrings in addresses (and correct me if you disagree),
the main attraction was the ability of letting applications do BIER
without having change APIs/host-stacks. That is cool short-term, but
the long-term benefit deteriorates as soon as someone does write
a BIER SDK for linux and windows. And with that context in mind i
found the long-term limitation of not being able to grow bitstrings
in various markets to be too limiting.

Cheers
    Toerless

---

From: IJsbrand Wijnands <[email-masked]>
Date: Tue, 17 Mar 2020 20:37:36 +0100

Toerless,

> different markets could evolve to support different (longer) length.

That is true, by overloading the IPv6 address the BM is fixed, but, it is way in front of the packet, so easy accessible for HW, so there will be a trade-off between more bits vs. performance penalty for longer masks. Longer masks will not come for free.

> without having change APIs/host-stacks. That is cool short-term, but

Yes.

> in various markets to be too limiting.

We probably need to define what you consider short-term vs. long-term. Getting IGMPv3 supported in the host stack only took 10 years, is that long or short term?

Thx,

Ice.

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 18 Mar 2020 17:33:25 +0100

Ice,

> That is true, by overloading the IPv6 address the BM is fixed, but, it is way in front of the packet, so easy accessible for HW, so there will be a trade-off between more bits vs. performance penal…

It is quite annoying, that we have no good agreed upon "standards" for
expected acceptable packet header processing expectations. Such as
depth into the packet that lookups are e.g.:

"cheap"... "acceptable" ..."impossible"

Quite jealous of the IoT space, where e.g.: rfc7228 exists.
In terms of packet processing, high-speed devices are even more
"constrained", yet we have never attempted to classify those
constraints better publically.

I foronce am going on the assumption, that within 10 year time frame, 256 byte
lookahead for header processing is standard if we push for it. And
128 is standard now.

> We probably need to define what you consider short-term vs. long-term. Getting IGMPv3 supported in the host stack only took 10 years, is that long or short term?

IGMPv3 is yet another soap-box for me. We should have from day 1 
put that signaling on top of UDP to eliminate the slowest-link
in the dependency chain, the operating system vendors. 

I would call 10 years mid-term, aka: we will see 2nd generation
of implementations in dynamic parts of the market, where there
is growth, and thats where i would hope to see longer masks
becoming adopted, but in segments where there is no big market pickup,
there would be just proliferation of unchanged minimum-to-implement
functionality from the first generation.

Cheers
    Toerless