Subject: Limited Domains: (was: I-D Action: draft-filsfils-6man-structured-flow-label-00.txt)

From: Ron Bonica <[email-masked]>
Date: Mon, 12 Apr 2021 16:20:22 +0000

Folks,

If we redefine the flow label so that it has one semantic on the global Internet and another in Limited Domain A, do we still have a single IPv6 protocol? Or do we have IPv6 and IPv6.1, which are not compatible with one another?

If we go down that path, how many incompatible versions of IPv6 should we allow. One? Two? Forty-two? 

                                                Ron

Juniper Business Use Only

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Ahmed Abdelsalam (ahabdels)
Sent: Monday, April 12, 2021 12:02 PM
To: Brian E Carpenter <[email-masked]>; [email-masked]
Cc: [email-masked]
Subject: Re: I-D Action: draft-filsfils-6man-structured-flow-label-00.txt

[External Email. Be cautious of content]

Hi Brian,

Many thanks for your time.

Indeed, the scope of this draft is within a limited domain. The deployment scenario for this draft is documented in Section 3. In short: service provider network, whereas all IPv6 nodes are under the administrative control of the operator. The ingress PE receives traffic from customers (eth, ipv4, ipv6) and encapsulates it in a new IPv6 header. The source node of the IPv6 packet traversing the SP network is the ingress PE. We can clarify this further in the next revision of the draft.

I believe this would address your point.

We will also fix the nit in the illustration.

Many thanks,
Ahmed.

-----Original Message-----
From: Brian E Carpenter <[email-masked]>
Date: Thursday, 8 April 2021 at 04:19
To: "[email-masked]" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Subject: Re: I-D Action: draft-filsfils-6man-structured-flow-label-00.txt
Resent from: <[email-masked]>
Resent to: <[email-masked]>, ahabdels <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]>, <[email-masked]> Resent date: Thursday, 8 April 2021 at 04:18

    Hi,

    A few comments on this draft.

    As background, there have been numerous past proposals for semantics in the flow label; all the ones we could find in 2011 are discussed in https://urldefense.com/v3/__https://www.rfc-editor.org/rfc/rfc6294.html__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOgTZo7iN$ . The IETF has consistently declined to adopt any of them. There's also some rationale for the current standard in https://urldefense.com/v3/__https://www.rfc-editor.org/rfc/rfc6436.html__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOnCrSKIZ$ .

    My first comment on the present draft is that it doesn't state its target scenario (which might be LAG, because LAG is mentioned a few times). It also ignores the fact that most current operating systems follow RFC6437 by setting a 20-bit pseudorandom label for all TCP sessions. Of course this must not be changed en route across the Internet. One usage scenario is described in RFC7098, but it's clear that the draft isn't compatible with any scenario in which sources somewhere on the Internet do what RFC6437 tells them to do and downstream routers or load balancers assume that is the case.

    So is it correct that the draft is aimed only at sources (and routers and destinations) within some sort of limited domain? If so, that needs to be clearly stated at the beginning.

    There is a spec for using the flow label for ECMP/LAG tunnels in RFC6438. I'd be inclined to the view that 16 pseudorandom bits would be sufficient in that case. In any case, in that case the end-to-end flow label is not affected, just the tunnel, so the fact that four bits don't contribute to the hash is tolerable.

    However, just to be clear, you *cannot* declare that in a packet that goes out on the Internet, where the downstream routers support RFC6437, that 4 bits in the flow label are not part of the flow label. Such a thing would in no way be "seamless migration from RFC6437".

    Relying on specific statements by a couple of router vendors about what their current products do or don't do is invalid. Other vendors might be different, and as technology evolves those two vendors might change what they do. The argument in section 4 might work for an ECMP/LAG scenario but it *certainly* doesn't work for the server farm scenario (RFC7098), which it would simply break. So rather than "seamless migration" you get "broken user sessions".

    Again, you might be able to fix this by positioning the proposal for an ECMP/LAG scenario within a limited domain or a provider tunnel. But as a generic update to RFC6437, absolutely positively not.

    Nit:

    I'm not sure why in figs 1 and 2 you use little-endian bit numbering. It's confusing. I thought the issue was settled by RFC 791.

    Regards
       Brian Carpenter

    On 17-Mar-21 05:49, [email-masked] wrote:
    >
    > A New Internet-Draft is available from the on-line Internet-Drafts directories.
    >
    >
    >         Title           : Structured Flow Label
    >         Authors         : Clarence Filsfils
    >                           Ahmed Abdelsalam
    >                           Shay Zadok
    >                           Xiaohu Xu
    >                           Weiqiang Cheng
    >                           Daniel Voyer
    >                           Pablo Camarillo Garvia
    >   Filename        : draft-filsfils-6man-structured-flow-label-00.txt
    >   Pages           : 12
    >   Date            : 2021-03-16
    >
    > Abstract:
    >    This document defines the IPv6 Structured Flow Label.  The seamless
    >    nature of the change to [RFC6437] is demonstrated.  Benefits of the
    >    solution are explained.  Use-cases are illustrated.
    >
    >
    > The IETF datatracker status page for this draft is:
    > https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-filsfils-6man-structured-flow-label/__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOnB_SDAh$
    >
    > There are also htmlized versions available at:
    > https://urldefense.com/v3/__https://tools.ietf.org/html/draft-filsfils-6man-structured-flow-label-00__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOv2CiWS9$
    > https://urldefense.com/v3/__https://datatracker.ietf.org/doc/html/draft-filsfils-6man-structured-flow-label-00__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOrk-MX27$
    >
    >
    > Please note that it may take a couple of minutes from the time of submission
    > until the htmlized version and diff are available at tools.ietf.org.
    >
    > Internet-Drafts are also available by anonymous FTP at:
    > https://urldefense.com/v3/__ftp://ftp.ietf.org/internet-drafts/__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOhgc6EkY$
    >
    >
    > _______________________________________________
    > I-D-Announce mailing list
    > [email-masked]
    > https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/i-d-announce__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOvvLmZUo$
    > Internet-Draft directories: https://urldefense.com/v3/__http://www.ietf.org/shadow.html__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOmQbYE53$
    > or https://urldefense.com/v3/__ftp://ftp.ietf.org/ietf/1shadow-sites.txt__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOskgTb1M$
    >

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipv6__;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9ARUaOuTV9LL3$
--------------------------------------------------------------------

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Apr 2021 19:09:38 +0200

> If we go down that path, how many incompatible versions of IPv6 should we allow. One? Two? Forty-two? 

AFAIK, this is not a new issue for this draft, this has been going on forever.

https://bapk-videos.mamadrum.net/old/IETF-099-Prague-videos/03-joel-jaeggli-8.mp4

IMHO, attempting to harvest more bits from the IPv6 base header for improved functionality
is just another wave of this problem space. At some point in time you have to give up
on fields/functions that have been burned by bad specs and/or bad implementions and unforeseeable
interop issues in uncontrolled network paths. I had the very same experience with router
alert, but in that case one could solve the problem by just using a new code point. With
flow label being in the base header, this is not possible. Even more so, the IPv6 base header
has no notion to support limited domain scoping of functionality.

I can just repeat myself and say that we simply should look into new, more flexible,
extensible base header, backward compatible to IPv6/Internet (but with code points assigned
to functionality, as we do with extension headers for example). Would be great if more people
would be willing to even start collecting issues with the base header so that we could even
prepare longer term for something like that. We also overlooked for example this flow-label
 issue in draft-bryant-arch-fwd-layer-ps-02

Cheers
    Toerless

> ???-----Original Message-----
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6

-- 
---
[email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Mon, 12 Apr 2021 10:36:18 -0700

>  issue in draft-bryant-arch-fwd-layer-ps-02
Or just use extension headers which are the explicit method of
extensibility in IPv6.

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Apr 2021 19:46:10 +0200

> Or just use extension headers which are the explicit method of

Except that that we also burned a lot of that functionality through bad spec/implementations/scoping,
so adding more to it would IMHO also create more and more interop issues. 

I also do not like this "ad-hoc" negotiation about what header encoding is or is not appropriate
by mean of anectodal evidence, as also done in this draft from somee vendors. This just makes
every extension so much more painfull. We really need to have a future proof standardized data+encoding
 model for forwarding plane headers as we have with e.g. Yang and (multiple) transports for Mgmt and e.g.:
CBOR for control plane, but alas in forwardng plane only (not very complete) for MPLS.

Cheers
    toerless

-- 
---
[email-masked]

---

From: Ron Bonica <[email-masked]>
Date: Mon, 12 Apr 2021 17:49:26 +0000

Toerless,

You say that "we simply should look into new, more flexible, extensible base header, backward compatible to IPv6/Internet (but with code points assigned to functionality, as we do with extension headers for example)."

The idea is intriguing, but it leads to the following questions:

- How do you make this new base header backwards compatible with IPv6-classic?
- Is draft-filsfils-6man-structured-flow-label an example of this new base header?

                                                                      Ron

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Toerless Eckert
Sent: Monday, April 12, 2021 1:10 PM
To: Ron Bonica <rbonica=[email-masked]>
Cc: Ahmed Abdelsalam (ahabdels) <ahabdels=[email-masked]>; [email-masked]; [email-masked]
Subject: Re: Limited Domains: (was: I-D Action: draft-filsfils-6man-structured-flow-label-00.txt)

https://urldefense.com/v3/__https://bapk-videos.mamadrum.net/old/IETF-099-Prague-videos/03-joel-jaeggli-8.mp4__;!!NEt6yMaO-gk!R8KfrY_-UU1OFDv1r7zr1dkRVmyApILeyhDlijeT8hQjBvn8FFSlbJ6d77MKy9Os$

IMHO, attempting to harvest more bits from the IPv6 base header for improved functionality is just another wave of this problem space. At some point in time you have to give up on fields/functions that have been burned by bad specs and/or bad implementions and unforeseeable interop issues in uncontrolled network paths. I had the very same experience with router alert, but in that case one could solve the problem by just using a new code point. With flow label being in the base header, this is not possible. Even more so, the IPv6 base header has no notion to support limited domain scoping of functionality.

I can just repeat myself and say that we simply should look into new, more flexible, extensible base header, backward compatible to IPv6/Internet (but with code points assigned to functionality, as we do with extension headers for example). Would be great if more people would be willing to even start collecting issues with the base header so that we could even prepare longer term for something like that. We also overlooked for example this flow-label  issue in draft-bryant-arch-fwd-layer-ps-02

Cheers
    Toerless

> Subject: Re: I-D Action: 
> draft-filsfils-6man-structured-flow-label-00.txt
>
> Cc: "[email-masked]" 
> <[email-masked]>
> Subject: Re: I-D Action: 
> draft-filsfils-6man-structured-flow-label-00.txt
> Resent to: <[email-masked]>, ahabdels <[email-masked]>, 
> <[email-masked]>, <[email-masked]>, 
> <[email-masked]>, <[email-masked]>, 
> <[email-masked]> Resent date: Thursday, 8 April 2021 at 04:18
>
>     Hi,
>
> Administrative Requests: 
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipv6
> __;!!NEt6yMaO-gk!UKtsY4bMjBYvdfHQZ3fbIfkDpi1BjJTpG15KUpg06Bgvp-n29Pk9A
> RUaOuTV9LL3$
> Administrative Requests: 
> https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipv6
> __;!!NEt6yMaO-gk!R8KfrY_-UU1OFDv1r7zr1dkRVmyApILeyhDlijeT8hQjBvn8FFSlb
> J6d7wJj-Si3$

--
---
[email-masked]

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]
Administrative Requests: https://urldefense.com/v3/__https://www.ietf.org/mailman/listinfo/ipv6__;!!NEt6yMaO-gk!R8KfrY_-UU1OFDv1r7zr1dkRVmyApILeyhDlijeT8hQjBvn8FFSlbJ6d7wJj-Si3$
--------------------------------------------------------------------

---

From: Tom Herbert <[email-masked]>
Date: Mon, 12 Apr 2021 11:13:56 -0700

> - Is draft-filsfils-6man-structured-flow-label an example of this new base header?
Right, it's hard to imagine that a new base header could be used
without creating a new IP version which is at least twenty years just
to get off the ground. I suppose the idea might be to have special
value in the next protocol that indicates an extension to the base IP
header, but then doesn't that just degenerate to to be another
extension header?

Tom

---

From: Fred Baker <[email-masked]>
Date: Mon, 12 Apr 2021 11:26:05 -0700

> On Apr 12, 2021, at 10:36 AM, Tom Herbert <[email-masked]> wrote:
> 
>> we simply should look into new, more flexible,
>> extensible base header, backward compatible to IPv6/Internet

That can be difficult, as it is by definition a different protocol.

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Apr 2021 20:31:43 +0200

> - How do you make this new base header backwards compatible with IPv6-classic?

IMHO we only need semantic backward compatibility, not encoding backward compatbility.
New functionality only (IMHO) needs to work across transitive segments that support the
new encoding and the edge-nodes of such a domain need to to do header transcoding (but
no semantic adaptation).

Just as a simple mind game to explain (not a proposal, do not do this at home kids, there are better
ways!) replace the IPv6 base header with a TLV-only header and assign a bunch of TL for the existing
IPv6 base header fields.

Allows you to leave crud out
  - source address in limited domains - MPLS-SR doesn't have it either
  - QoS - mostly 0
Allows you to easily add variants / extensions
  - Different T or TL variants for different semantics we may want over time and just
    couldn't deploy because of above mentiond overloading issue.
  - Limited domains like shorter addresses, BIER an ICN need longer

yada yada..

> - Is draft-filsfils-6man-structured-flow-label an example of this new base header?

No because of overloading / change from prior semantics for the field.

Cheers
    toerless

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Apr 2021 20:40:30 +0200

On Mon, Apr 12, 2021 at 11:13:56AM -0700, Tom Herbert wrote:
> to get off the ground.

A new encoding could get off the ground very quickly for limited domain/controlled networks
and grow from there. Just think of all the cases where MPLS/MPLS-SR/SRH are being used.

And note that those have been arguably slow because they where/are tied very much to a
very HW-centric model and limited flexibility in those forwarders. But this is constantly
evolving and improving. The timing and adoption stories are quite different for other
technologies such as IP Multicast. But we have all these stories and should have learned from
them.

But yes, it would be IMHO a lot more prudent to start looking into evolutions that are
not solely ad-hoc one-off extensions. If we had a clearer strategy how to evolve long term,
the pain of short term workaround would also become more manageable IMHO (and from experience
in such past stories).

Cheers
    Toerless

> I suppose the idea might be to have special

-- 
---
[email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Mon, 12 Apr 2021 12:07:49 -0700

Toerless,

Right, SRH is an extension header that works with the existing
extensibility constructs for IPv6. Fundamentally, I don't understand
what problem a new encoding would address, perhaps you could
articulate that (personally, I wouldn't consider a use case where only
one bit of extensible information is needed as justification to create
a new version of IP).

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Mon, 12 Apr 2021 21:33:28 +0200

> one bit of extensible information is needed as justification to create

See my last email reply with the TLV header encoding example. Of course
this is not justified by a single featurette. See for example also
draft-bryant-arch-fwd-layer-ps. In that one we actually missed out to
discuss the flow ID issues, have to add.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 13 Apr 2021 08:51:11 +1200

It could be a bit more subtle, as the limited domain RFC tries to indicate in its "Functional Requirements" (https://www.rfc-editor.org/rfc/rfc8799.html#name-functional-requirements-of- ):

>>> 4. Node Eligibility. It must be possible for a node to determine which domain(s) it can potentially join and on which interface(s).
...
>>> 9. Peer Verification. A node must be able to verify whether another node is a member of the domain.

In other words, a redefined flow label could be used pairwise between hosts that have determined that they are both members of a particular limited domain.

In that worldview, there are as many variants of IPv6 as there are limited domains, plus one (which is the Internet-wide standard version).

   Brian

---

From: Fred Baker <[email-masked]>
Date: Mon, 12 Apr 2021 13:52:48 -0700

> On Apr 12, 2021, at 1:51 PM, Brian E Carpenter <[email-masked]> wrote:
> 

Which would eliminate the use of the flow label by systems that are not part of that domain.

---

From: "Manfredi (US), Albert E" <[email-masked]>
Date: Mon, 12 Apr 2021 20:52:53 +0000

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Tom Herbert

This topic seems to be similar to the one about giving semantic meaning to IP addresses. If the standards say no, and that's what they say, then anything like that would be restricted to "behind closed doors, among consenting adults."

Not sure how anything else can work, with the standards as written, and as widely implemented, for many years.

And if the implementation is non-standard, in limited domains, it could even be seen as a "feature," in that limited domain.

Bert

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 13 Apr 2021 09:01:08 +1200

> header, but then doesn't that just degenerate to to be another

Yes. And while we're at it, let's use Huffman coding for IP addresses.

Whatever we do, we won't get away from actual line-speed implementations
being constrained by current integrated circuit technology. So every few
years, with a new generation of technology, the constraints change, just
as the Strowger switch revolutionized telephony in 1891.

   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 13 Apr 2021 10:09:35 +1200

> Which would eliminate the use of the flow label by systems that are not part of that domain.

No, it would create two sets of nodes: nodes using RFC6437 between themselves, and nodes using some other interpretation of the flow label between themselves. Ships in the night, really just like running IPX and CLNP on the same network. We used to do that a lot.

I'm not advocating this, just pointing out that it's possible.

   Brian

---

From: Tom Herbert <[email-masked]>
Date: Mon, 12 Apr 2021 15:20:49 -0700

> No, it would create two sets of nodes: nodes using RFC6437 between themselves, and nodes using some other interpretation of the flow label between themselves. Ships in the night, really just like r…

Brian,

The problem is that there's no code point that  discriminates between
the different interpretations of the flow label. So in a limited
domain the deployment of the new implementation must be perfect, lest
there is at least one router that is inadvertently run with the legacy
interpretation, then that router may process the flow label and ECMP
per the standard leading to situations where packets of some flow are
unexpectedly going over different paths. Since there's nothing in the
packet that states what's happening, such problems can be really
difficult to debug.

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 13 Apr 2021 11:55:20 +1200

> packet that states what's happening, such problems can be really

I completely agree. I was thinking about how to code this safely, and
it's a mess, especially if you want routers to detect it at line speed.

    Brian

---

From: Gyan Mishra <[email-masked]>
Date: Mon, 12 Apr 2021 20:21:18 -0400

So I think we all agree that backwards compatibility is not possible and
things would break with the restructuring of the flow label.

On Mon, Apr 12, 2021 at 7:55 PM Brian E Carpenter <
[email-masked]> wrote:

> >>>> In other words, a redefined flow label could be used pairwise between
> hosts that have determined that they are both members of a particular
> limited domain.
> >>>
> >> No, it would create two sets of nodes: nodes using RFC6437 between
> themselves, and nodes using some other interpretation of the flow label
> between themselves. Ships in the night, really just like running IPX and
> CLNP on the same network. We used to do that a lot.
> >
> > Brian,
> >
-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: Mark Smith <[email-masked]>
Date: Tue, 13 Apr 2021 13:15:56 +1000

Hi Ron,

On Tue, 13 Apr 2021 at 02:22, Ron Bonica
<rbonica=[email-masked]> wrote:
>
> Folks,
>
> If we redefine the flow label so that it has one semantic on the global Internet and another in Limited Domain A, do we still have a single IPv6 protocol?

No, no single IPv6 protocol anymore. There would now be multiple
variants or dialects, similar to spoken and written human languages
like English.

I fear that the worst case use of "limited domains" will be for a
vendor to define their own proprietary version of standard protocols,
and then trick customers into buying them by obscuring that, and then
the customer is locked into continuing to have to buy the proprietary
"limited domain" version only available from that vendor.

> Or do we have IPv6 and IPv6.1, which are not compatible with one another?
>

If multiple, semantically different versions, variants, or field
structures of IPv6 are going to be created, the version number or
sub-version needs to be encoded in the packet somehow so that
receivers can distinguish the version or variant it to know how it is
to be processed.

What draft-filsfils-6man-structured-flow-label is really doing is
taking 4 bits from the flow label field and defining a completely new
field in the 26 year old, now full Internet Standard, IPv6 fixed
header, without changing the protocol version to indicate a new header
structure.

You can tell it is a new and unrelated field because it is not
required to be set or processed as flow related information:

"FLC semantics (hence, per-packet marking, potentially not per-flow constant)"

and

"FLC: 4-bit "[19, 16]": per-packet Control not included in ECMP hash".

Including the term "Flow Label" in the FLC field name is
misrepresenting what its purpose is. If those bits were available
because they were reserved, then I'd guess the authors wouldn't have
included the term "Flow Label" in the field name at all, instead would
have called it something like the "Generic Application Mark" field.

Regards,
Mark.

---

From: Tony Przygienda <[email-masked]>
Date: Tue, 13 Apr 2021 08:38:06 +0200

IME problem is deeper and I don't only see it on v6 either. There is a
class of folks that are protocol designers and there is a (growing in last
couple years?) class of folks saying, "oh, I see so many bytes, I can do so
many things in those bits and damn' the torpedoes and semantics" or even
worse "I can stick my stuff in somewhere in the packet header, worst case
let's add TLVs to packet header". But generally it's been going on since
time immemorial, DSCP, 8+8, BIER mask encoding in v6 address and so on.

Thankfully we still have strong WGs/chairs in many places that try to keep
the finger in the dyke (mostly) but it's frustrating for people with line
speed silicon to see lots folks pushing for those things who have no
understanding of limitations of header pipelines, impact of shifting
offsets, huge security filters etc on the ultimate power consumption, size
and finally price of the very electrons delivered to carry the bit that
writes those emails ;-)

my 2c

-- tony

On Mon, Apr 12, 2021 at 11:01 PM Brian E Carpenter <
[email-masked]> wrote:

> >> You say that "we simply should look into new, more flexible, extensible
> base header, backward compatible to IPv6/Internet (but with code points
> assigned to functionality, as we do with extension headers for example)."
> >>

---

From: "Ahmed Abdelsalam (ahabdels)" <[email-masked]>
Date: Tue, 13 Apr 2021 08:59:57 +0000

Inline [AA]

-----Original Message-----
From: Tom Herbert <[email-masked]>
Date: Tuesday, 13 April 2021 at 00:21
To: Brian E Carpenter <[email-masked]>
Cc: Fred Baker <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>, ahabdels <[email-masked]>
Subject: Re: Limited Domains:

    Brian,

[AA] I don't think we need a new codepoint.
The operator explicitly configures the routers under his administrative domain to support Structured Flow Label. Note that all the nodes involved in the IPv6 packet lifecycle (source node, transit nodes, destination nodes) ALL are under the same administrative domain of the operator. If there is a router that does not support the Structured Flow Label, then the 4FLC bits are NOT set/inspected by the source/transit node.
Keep in mind that packet reordering is part of the SLA in between the operator and their customers. The operator will not enable something that breaks their SLAs on purpose.

    Tom

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 13 Apr 2021 10:57:30 +0100

Ahmed Abdelsalam (ahabdels) wrote on 13/04/2021 09:59:
> The operator explicitly configures the routers under his
> administrative domain to support Structured Flow Label.

a codepoint is a marker to cause something to be interpreted in a 
specific and different way.  Moving this flag to the configuration 
domain is essentially admitting that the semantics of a codepoint are 
necessary, except that it removes the possibility of having these 
semantics interpreted automatically on a per-packet basis, and turns it 
into a per-router or possibly per-interface flag.  This is an unusual 
type of regression in terms of a protocol where so much work has been 
put into auto-configuration, and where the syntactic mechanism has been 
created for protocol extensions.

As a separate issue, if a regular ip packet from the outside 
accidentally enters into this administrative domain, how does a router 
on the inside of the domain know not to attempt to interpret the flow 
label as a SFL, as it could end up effectively parsing random junk?

Nick

---

From: "Ahmed Abdelsalam (ahabdels)" <[email-masked]>
Date: Tue, 13 Apr 2021 16:17:55 +0000

Hi Nick, 
Please find my answers inline [AA] 

-----Original Message-----
From: Nick Hilliard <[email-masked]>
Date: Tuesday, 13 April 2021 at 11:58
To: ahabdels <[email-masked]>
Cc: Tom Herbert <[email-masked]>, Brian E Carpenter <[email-masked]>, "[email-masked]" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: Limited Domains:

[AA] This is a domain wide config applied to all nodes of the domain. So either you decide to do on all routers or not. It doesn’t on per packet treatment.

[AA] All packet coming from external interfaces will be encapsulated in outer IPv6 header as explained the in section 3. The outer IPv6 is what matters for forwarding inside the domain. If a packet from outside accidentally enters into this administrative domain, then that is a bigger problem than SFL.

    Nick

---

From: Nick Hilliard <[email-masked]>
Date: Tue, 13 Apr 2021 17:34:16 +0100

Ahmed Abdelsalam (ahabdels) wrote on 13/04/2021 17:17:
> If a packet from outside accidentally enters into this administrative
> domain, then that is a bigger problem than SFL.
Ok.  It would be helpful to describe this in some detail in the security 
section of the draft, and that there would need to be a hard and heavily 
policed border in place due to the protocol incompatibility.

Nick

---

From: Tom Herbert <[email-masked]>
Date: Tue, 13 Apr 2021 09:46:46 -0700

> [AA] This is a domain wide config applied to all nodes of the domain. So either you decide to do on all routers or not. It doesn’t on per packet treatment.
Ahmed,

So this is all or nothing then, replete with a so-called "flag day".
What is the protocol that guarantees that every router in the domain
is properly configured, that no one ever inadvertently brings up a
router without proper configuration? And if configuration fails, which
even in a moderately sized domain will inevitably happen at some
point, how will the user detect the issue and debug it?

Tom

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 13 Apr 2021 19:32:07 +0200

Can i get a standard RFC for another idea, in which
one does not overload the flow-ID, but instead source and destination
addresss ? In isolated limited domains, i do not need all those
long 128 bit addresses, but i appreciate the benefit of having the
IPv6 socket API in every OS, so i can read/write very easily those
addresses. And the basic IPv6 forwarding code in every router i want
tominimally modify.

For example, i could just use 32 bit IPv4 or even shorter
(16 bit) addresses and then use the rest of the IPv6 header address
bits for my isolated networks requirements and hosts and routers hop-by-hop
can use those bits, every isolated network could come up with
different desired semantics. I for once would love to put exactly
all those ethernet header fields required for TSN QoS into them,
so i have a 1:1 mapping of TSN into (abused) "IPv6" (TSN in IPv6).

Now, if you think this is too hilarious and the IMHO correct answer
should be "Great, toerless, but you are too late for Aprils Fools RFC",
lets remember that its not too far off from other address overload proposals
we did have in the IETF.

IMHO, i don't think you can just keep the encoding and overload
the semantic of a packet header in semantically incompatible fashions and NOT
call it a new protocol. 

Of course all the existing infra especially host stacks but also
router code creates a strong pressure to do just minimal changes in
places that are expensive/slow to change, but instead of sliding down 
this dangerous overload lane, why not recognize this generic problem and take the
hit once more of doing a larger change (new extensible, semantically
backward compatible base protocol header) to have clean options in
the future to do minimal enhancements where needed (aka: add new
TLV in my past example).

This is barly the only place. Just look at ECN semantics for L4S. Hard
trying to avoid semantic overload because all they can do is to fudge
around existing IPv6 header bits.

Cheers
    oerless

On Tue, Apr 13, 2021 at 09:46:46AM -0700, Tom Herbert wrote:
> > [AA] This is a domain wide config applied to all nodes of the domain. So either you decide to do on all routers or not. It doesn???t on per packet treatment.
> >
> Ahmed,
> 

-- 
---
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 14 Apr 2021 11:39:17 +1200

>> [AA] This is a domain wide config applied to all nodes of the domain. So either you decide to do on all routers or not. It doesn’t on per packet treatment.

But what happens if a packet sourced inside the domain has a destination
outside the domain, where flow labels are 20 bits? What happens when a
packet from outside the domain with a 20 bit flow label arrives inside
the domain?

The only model I can imagine is that every router in the domain (not just
the border routers) would need to check every packet and, if either
the source or the destination prefix is outside the domain, then apply
the rules of RFC6437. That's mathematically possible, I suppose, but
it absolutely requires a per-packet treatment.

If you're going to steal codepoints for local use, steal them from the
DSCP. That is potentially rewritten at the domain boundary, and what's
more it's IP-version independent and belongs to another WG.

(Fair warning: this would be contentious too.)

Regards
     Brian

> .

---

From: Stewart Bryant <[email-masked]>
Date: Wed, 14 Apr 2021 11:36:10 +0100

As far as I can see the only safe limited domain protocol is one specifically designed for use in limited domains.

Any other approach leads to confusion, mistakes, security threats, complexity and cost.

Thus declaring that an “ordinary” IPv6 packet can simultaneously have both global and limited scope has the potential for creating significant issues for those wishing to use basic IPv6 in a limited domain.

We have an example of an IETF limited domain protocol: MPLS. This has a very simple lightweight data plane security model: it is a different protocol from IP and if it is presented with an IP packet at its edge, it simple wraps it in MPLS and sends it safely on its way across the network for export Into some other network. Operators have a lot of experience with this protocol and we know that the model that MPLS is not IP results in complete confidence that the network will not confuse the two.

Equally we know of cases where IP is vulnerable to attack because it is so difficult to exclude packets. This was at the heart of the reason that source routing, was deprecated some years ago.

Now I am not for a moment suggesting that the limited domain applications that the flow-label authors have in mind should be done in MPLS, but I am suggesting that if they want a limited domain protocol with properties different from IPv6, and there is no obvious way to unambiguously indicate the new functionality in basic IPv6, they ought to design a protocol with the properties that they require that is not IPv6.

I am reminded in this discussion of the a time when another SDO wanted to make a “small” incompatible change to MPLS and argued that as this was only deployed in a limited domain that was safe.The IETF position was that incompatible and unrecognisable modification to one of our network protocols was a bad thing. A protracted high profile argument ensued and in the end  the IETF view won the day.

This protracted discussion on flow labels seems to be in a similar mould, and I would argue that we should not accept a change to the forwarding actions on an IPv6 packet unless it is possible for the forwarder to know precisely and unambiguously  which action it is to take on the packet is is currently parsing.

- Stewart

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 15 Apr 2021 08:33:58 +1200

> As far as I can see the only safe limited domain protocol is one specifically designed for use in limited domains.

Absolutely. And also, there needs to be a secure and objective definition of the membership and boundary of the domain. That's why we wrote RFC8799.

   Brian

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 15 Apr 2021 00:01:41 +0200

> > As far as I can see the only safe limited domain protocol is one specifically designed for use in limited domains.

Which IMHO does not exclude the option to design such a protocol for
limited domain in a way that it also supports "non-limited domains".

> On Thu, Apr 15, 2021 at 08:33:58AM +1200, Brian E Carpenter wrote:

And especially the permissible/desirable in/out between the limited domain
and other networks, because limited domain >= isolated domain. That
particular issue is what makes semantic overloading problematic as
any enterprise operator can attest to who has gone through an M&A
of multiple 10 net domains, or who had to start multi-homing without
PI addresses just to name two common cases.

Cheers
    Toerless

> > Thus declaring that an ???ordinary??? IPv6 packet can simultaneously have both global and limited scope has the potential for creating significant issues for those wishing to use basic IPv6 in a limited domain.
> > 
> > I am reminded in this discussion of the a time when another SDO wanted to make a ???small??? incompatible change to MPLS and argued that as this was only deployed in a limited domain that was safe.The IETF position was that incompatible and unrecognisable modification to one of our network protocols was a bad thing. A protracted high profile argument ensued and in the end  the IETF view won the day.
> > 

-- 
---
[email-masked]

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 14 Apr 2021 23:10:54 +0100

> limited domain in a way that it also supports "non-limited domains".

we already have this with MPLS, but it was designed from the outset as a 
limited domain protocol and was only later extended to interoperate 
outside the domain boundary, and only then in specific and carefully 
controlled circumstances.

Re-scoping an existing inter-domain protocol like ipv6 into a limited 
domain situation is a fundamentally different proposition.  Unless the 
scoping can be managed on a mandatory per-packet basis, it's the sort of 
proposition that will end up causing a long wake of operational problems 
and consequences.

Nick

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 15 Apr 2021 00:23:34 +0200

On Wed, Apr 14, 2021 at 11:10:54PM +0100, Nick Hilliard wrote:
> limited domain protocol and was only later extended to interoperate outside
> the domain boundary, and only then in specific and carefully controlled
> circumstances.

Indeed. And this could probably improved.

> Re-scoping an existing inter-domain protocol like ipv6 into a limited domain
> situation is a fundamentally different proposition.  Unless the scoping can
> be managed on a mandatory per-packet basis, it's the sort of proposition
> that will end up causing a long wake of operational problems and
> consequences.

True. But i think we can collect the experiences we have, especially from
rfc1918 (multiple instances), ULA, IPv6 PD addressing /multiohoming and construct
from them the principles for a better protocol framework. Of course also
aspects such as the (now dead) site local addressing and (still alive)
4 bit of scope addressing in IPv6 multicast.

If we just stay where we are now, i am afraid that a lot
of industrial use of IP, which is IPv4 only will never move to IPv6
because the longer addresses and ULA are not really an improvement
over RFC1918 + NAT in the mindset of such operators. rfc1918 provides
"generic addressing", ULA not (for example). IPv4 addresses can be
memorized by operators, IPv6 addresses not (unless you use illegal
prefixes). Etc. pp.

-- 
---
[email-masked]

---

From: Robert Raszuk <[email-masked]>
Date: Fri, 16 Apr 2021 23:42:30 +0200

I think this this thread nicely demonstrates that we need to first define
what a "limited domain" is.

To some it seems to be 1980s definition of an IGP network boundary. More
modern folks would consider as "limited domain" a set of IGP ASNs areas
interconnected by p2p BGP still under the same administration.

For me "limited domain" is an arbitrary collection of sites anywhere in the
world using Internet for inter-connectivity.

So any protocol which claims to be defined for "limited domain" and which
claims that it is backwards compatible with nodes not supporting it is
equal to allow it to traverse Internet.

As simple as this.

Cheers,
Robert.

On Wed, Apr 14, 2021 at 12:36 PM Stewart Bryant <[email-masked]>
wrote:

> As far as I can see the only safe limited domain protocol is one
> specifically designed for use in limited domains.
>
> Thus declaring that an “ordinary” IPv6 packet can simultaneously have both
> global and limited scope has the potential for creating significant issues
> for those wishing to use basic IPv6 in a limited domain.
>
> We have an example of an IETF limited domain protocol: MPLS. This has a
> very simple lightweight data plane security model: it is a different
> protocol from IP and if it is presented with an IP packet at its edge, it
> simple wraps it in MPLS and sends it safely on its way across the network
> for export Into some other network. Operators have a lot of experience with
> this protocol and we know that the model that MPLS is not IP results in
> complete confidence that the network will not confuse the two.
>
> Equally we know of cases where IP is vulnerable to attack because it is so
> difficult to exclude packets. This was at the heart of the reason that
> source routing, was deprecated some years ago.
>
> Now I am not for a moment suggesting that the limited domain applications
> that the flow-label authors have in mind should be done in MPLS, but I am
> suggesting that if they want a limited domain protocol with properties
> different from IPv6, and there is no obvious way to unambiguously indicate
> the new functionality in basic IPv6, they ought to design a protocol with
> the properties that they require that is not IPv6.
>
> I am reminded in this discussion of the a time when another SDO wanted to
> make a “small” incompatible change to MPLS and argued that as this was only
> deployed in a limited domain that was safe.The IETF position was that
> incompatible and unrecognisable modification to one of our network
> protocols was a bad thing. A protracted high profile argument ensued and in
> the end  the IETF view won the day.
>
> This protracted discussion on flow labels seems to be in a similar mould,
> and I would argue that we should not accept a change to the forwarding
> actions on an IPv6 packet unless it is possible for the forwarder to know
> precisely and unambiguously  which action it is to take on the packet is is
> currently parsing.
>
> - Stewart
>
>

---

From: "Manfredi (US), Albert E" <[email-masked]>
Date: Fri, 16 Apr 2021 22:11:48 +0000

From: ipv6 <[email-masked]> On Behalf Of Robert Raszuk

> I think this this thread nicely demonstrates that we need to first define what a "limited domain" is. 
>
> To some it seems to be 1980s definition of an IGP network boundary. More modern folks would consider as "limited domain" a set of IGP ASNs areas interconnected by p2p BGP still under the same administration. 
>
> For me "limited domain" is an arbitrary collection of sites anywhere in the world using Internet for inter-connectivity.

Good point! Whereas to me, "limited domain" means, only inside this platform.

> So any protocol which claims to be defined for "limited domain" and which claims that it is backwards compatible with nodes not supporting it is equal to allow it to traverse Internet.

"Backwards compatible" may mean different things to different people, and it seem dubious in this case (because flow label is to be a random value, per IPv6). My main view is, if the domain is truly limited, firewalled or even air gapped, then what is the motivation to seek approval in a standards body?

Bert

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 17 Apr 2021 10:17:24 +1200

> To some it seems to be 1980s definition of an IGP network boundary. More modern folks would consider as "limited domain" a set of IGP ASNs areas interconnected by p2p BGP still under the same admin…

RFC8799 defines what it means by "limited domain" and it is not that.
 
> For me "limited domain" is an arbitrary collection of sites anywhere in the world using Internet for inter-connectivity. 

RFC8799 excludes that case. It specifically says "In other cases, it may refer to a defined set of users or nodes distributed over a much wider area, but drawn together by a single virtual network over the Internet, or a single physical network running in parallel with the Internet."

RFC8799 is not authoritative in any way, but it's quite clear on this point.

   Brian
 
> On Wed, Apr 14, 2021 at 12:36 PM Stewart Bryant <[email-masked] <mailto:[email-masked]>> wrote:
>

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 17 Apr 2021 00:25:11 +0200

> My main view is, if the domain is truly limited, firewalled or even air
gapped, then what is the motivation to seek approval in a standards body?

Spot on !

If someone (vendor or operator) is to seek to deploy some functionality
limited to his own network new ethertype should be allocated and then he is
free to do whatever needed.

Otherwise I do not subscribe to this limited domain vs Internet limitation
of any IETF spec. I have never seen for the Internet ethertype any way to
scope it not to be send over the limited domain boundary.

Best,
R.

On Sat, Apr 17, 2021 at 12:12 AM Manfredi (US), Albert E <
[email-masked]> wrote:

> > For me "limited domain" is an arbitrary collection of sites anywhere in
> the world using Internet for inter-connectivity.
>
> > So any protocol which claims to be defined for "limited domain" and
> which claims that it is backwards compatible with nodes not supporting it
> is equal to allow it to traverse Internet.
>
> "Backwards compatible" may mean different things to different people, and
> it seem dubious in this case (because flow label is to be a random value,
> per IPv6). My main view is, if the domain is truly limited, firewalled or
> even air gapped, then what is the motivation to seek approval in a
> standards body?
>
> Bert
>
>

---

From: Gyan Mishra <[email-masked]>
Date: Fri, 16 Apr 2021 18:35:25 -0400

+1 on the different meanings to different people on limited domains

The concept of limited domains has definitely evolved and grown but I think
the main underlying theme of limited or I think a better term is “closed”
domain is simply the “operators” network whatever that encompasses which
could be a single AS or 100s of ASs that maybe have many administrative sub
domains as containers within the overall operators domain.

MPLS was a means of confining the concept of a domain but within inter AS
options the domain sprawl can still be a collection of ASs.

There are many nuances with the term limited or closed domain so can also
be very different on a case by case basis.

Also the concept of domain varies drastically even from public operators
domain verses private internal domain.  Also even within a single operators
network their maybe multiple sub domains within public and private network
within an operators overall “domain” that may be parts of different
administrative boundaries but even so they still communicate and is not a
“limited or closed” domain.  There are even case where the public and
private have peering points to each other connect at the hip over MPLS.

At a high level you can think of a corporation and all of its many sub
domains which are part of many separate administrative domains that somehow
are all interconnected public and private are now part of one very large
“limited” or “closed” domain.

There are so many variables and variations and nuances the idea of a
“limited” or “closed” domain exists which could encompass all sub domains
and thus in some cases that limited domain may not see that limited as it
could be quite massive.

Thanks

Gyan

-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Tom Herbert <[email-masked]>
Date: Fri, 16 Apr 2021 16:22:33 -0700

On Fri, Apr 16, 2021 at 3:36 PM Gyan Mishra <[email-masked]> wrote:

> The concept of limited domains has definitely evolved and grown but I
> think the main underlying theme of limited or I think a better term is
> “closed” domain is simply the “operators” network whatever that encompasses
> which could be a single AS or 100s of ASs that maybe have many
> administrative sub domains as containers within the overall operators
> domain.
>
> Yes, and the scale of those large domains renders any proposal that relies
on a perfectly synchronized configuration across all the elements to
inevitably be problematic in deployment. Even in a limited domain, It is
better to use explicit code points for new features to avoid any ambiguity
and allow incremental deployment (i.e. no "flag days").

Tom

>> > To some it seems to be 1980s definition of an IGP network boundary.
>> More modern folks would consider as "limited domain" a set of IGP ASNs
>> areas interconnected by p2p BGP still under the same administration.
>> >

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 17 Apr 2021 11:43:14 +1200

On 17-Apr-21 10:25, Robert Raszuk wrote:
> 
>> My main view is, if the domain is truly limited, firewalled or even air gapped, then what is the motivation to seek approval in a standards body?
> 
> Spot on ! 

On that argument the SPRING WG should never have been chartered and the 6MAN WG should never have approved RFC8754. Also, we should never have defined diffserv in 1998. And NAT, of course, would be excluded by definition, and RFC1918 from 1996 would need to be obsoleted.
 
> If someone (vendor or operator) is to seek to deploy some functionality limited to his own network new ethertype should be allocated and then he is free to do whatever needed. 
> 
> Otherwise I do not subscribe to this limited domain vs Internet limitation of any IETF spec. I have never seen for the Internet ethertype any way to scope it not to be send over the limited domain boundary.

But we keep on defining limited domain protocols regardless of that problem. Please do read the RFC (https://www.rfc-editor.org/rfc/rfc8799.html). Don't imagine that this issue is ever going to disappear. I believe that the deeper problem is that IETF dogma is to pretend that this issue does not exist.

The problem with the current flow label proposal is of course exactly what you say: it forks the definition of the IPv6 packet, so could only coexist with the Internet-wide flow label if all the routers in the domain check whether the origin or destination of every packet is internal or external.

Regards,
   Brian

> Best,
> R.
> 
> 
> On Sat, Apr 17, 2021 at 12:12 AM Manfredi (US), Albert E <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     From: ipv6 <[email-masked] <mailto:[email-masked]>> On Behalf Of Robert Raszuk
>

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 17 Apr 2021 02:30:52 +0200

Hi Brian,

> On that argument the SPRING WG should never have been chartered and the
> 6MAN WG should never have approved RFC8754. Also, we should never have
> defined diffserv in 1998. And NAT, of course, would be excluded by
> definition, and RFC1918 from 1996 would need to be obsoleted.
>

I beg to differ in the perspective on this.

I would rather see 6man understanding the reality, understanding that
hardware engineering and innovation there have moved on and that protecting
original father's of IPv6 fixed size header wall garden does not make sense
any more.

Yet we still see orthodox protection over technically obsolete dogmas.

As we see in parallel there is equally interesting discussion on use of the
last 64 bits of IPv6 address happening. This is pretty interesting
considering that even most conservative folks agree that IPv6 address
should be assigned to the interfaces. So I define a logical interface which
is an anchor to internal switching vector and different processing
function.

Last RFC1918 just talks about address allocation - not sure how is this
related to any of this discussion ... you mean that the notion that private
addresses are not routable globally is the key point here ? I thought the
concern is with router's processing capabilities not with where such
processing is subject to happen.

Best,
R.

PS. Btw FWIW my interest and former activity in 6man list on SR topics
where not so much motivated in that I like SR. Much more I oppose hardcore
resistance to protect something which technology wise is all pretty much
obsoleted.

And I fully understand why this is going on like this - to make sure new
features do not break existing IPv6 world ... it is just that protecting
something which technically is already addressed and keeping innovation
gated is IMO not the best strategy for networking.

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 17 Apr 2021 02:40:41 +0200

> and allow incremental deployment (i.e. no "flag days").

I think its more subtle than that. What you say is IMHO very true for
a non-tagged field in a packet of (often incorrectly) assumed global scope
that often may be used in non-well bounded networks, if not the Internet. 
Aka: for IP/IPv6 base header fields. Those conditions are IMHO what makes
 overloading a non-starter.  (and i am leaving out a couple more issues)

In MPLS the base functionality of every label (except SPL) is: NONE, so you
MUST have a control plane to go along with the forwarding plane for anything
to happen. And the scope of consistency of course differs across the
different semantics of label, and most times the IETF defined such a semantic,
the problems of consistency where analyzed/understood, and sometimes/often?
even documented in RFCs. In this model of separating encoding from
semantics, the core problems of IP/IPv6 for untagged "overloading" of
fields does not really exist. But of course, the issue of e.g.:
"eventual consistency" becomes larger and larger the more the domain where
consistent interpretation is required grows. So IMHO a protocol that would
allow to pick and choose tagging/registration based semantic vs
untagged/control-plane based semantic would really be cool.

Toerless

> >> the world using Internet for inter-connectivity.
 s

-- 
---
[email-masked]

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 17 Apr 2021 03:06:19 +0200

> I beg to differ in the perspective on this.

I am sure Brian is dead serious undoing all the pragmatic workarounds
we did over the decades. NOT.

> Yet we still see orthodox protection over technically obsolete dogmas.

I think there is just a very justifyable desire not to break stuff
that works in a trillion dollar environment. But i also hope that most
folks by now will recognize that we have already been stretching the
applicability of IPv6 packet headers way beyond its original understood
goal of "IPv4 with longer addresses" and through the learning effort
burned many options (Flow ID) and created a lot of technology debt.

The desire by IPv6 evangelists to make IPv become the one thing that
fits all is well understandable, but when we did introduce an IPv6
version of SR, i think we recognized how much we are stretching what
 IPv6 with its basically 50 year old quite inflexible header and
address design can accommodate.

> is an anchor to internal switching vector and different processing

All these ongoing proposals to get more out of the IPv6 header remind
me of an old colleague who was moving out of his home after i think
20 years telling me "i have exhausted all possible renovations in this
place. Time to move on". (Hi Steve ! ;-)

And see my other posts wrt. how to maintain all functionality and 
"backward compatibility" (the way i define it).

> processing is subject to happen.

scoped addresses are a key element to building limited domains. Repeating
myself, RFC1918 for example spurred a lot of easily built, mostly industrial
networking with IP, just using "generic" 10.0.x.y addresses, same for
every instance of a product. IPv6 has taken that away.  Just one example.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 17 Apr 2021 13:59:13 +1200

> I beg to differ in the perspective on this. 

Not so much. I was simply trying to underline that the IETF has been actively defining limited domain solutions for many years, while claiming to define Internet-wide protocols.
 
> I would rather see 6man understanding the reality, understanding that hardware engineering and innovation there have moved on and that protecting original father's of IPv6 fixed size header wall garden does not make sense any more. 
> 

I also see people from the hardware side of the router and firewall world asking for less flexible header design. There's quite a tussle going on here.
 
> As we see in parallel there is equally interesting discussion on use of the last 64 bits of IPv6 address happening. This is pretty interesting considering that even most conservative folks agree that IPv6 address should be assigned to the interfaces. So I define a logical interface which is an anchor to internal switching vector and different processing function. 
> 
> Last RFC1918 just talks about address allocation - not sure how is this related to any of this discussion ... you mean that the notion that private addresses are not routable globally is the key point here ? 

I should probably have pointed the finger at RFC1597 (Address Allocation for Private Internets). But yes, that was an explicit enabler for limited domains within which you could do whatever private protocol you wanted, between a subset of hosts, in 1994.

> I thought the concern is with router's processing capabilities not with where such processing is subject to happen.

Routing is only one aspect. Limited domains exist for all kinds of reasons. Like it or not (and some people don't like it), the current architecture of the Internet includes thousands (probably millions) of limited domains.

> 
> Best,
> R.
> 
> PS. Btw FWIW my interest and former activity in 6man list on SR topics where not so much motivated in that I like SR. Much more I oppose hardcore resistance to protect something which technology wise is all pretty much obsoleted. 
> 
> And I fully understand why this is going on like this - to make sure new features do not break existing IPv6 world ... it is just that protecting something which technically is already addressed and keeping innovation gated is IMO not the best strategy for networking. 

Yes. That's exactly why I worked on RFC8799. 

    Brian

---

From: Toerless Eckert <[email-masked]>
Date: Wed, 28 Apr 2021 22:11:41 +0200

> time immemorial, DSCP, 8+8, BIER mask encoding in v6 address and so on.

Well. I think we had some good successes in actually making addresses more
useful by giving them more semantics than the IPv4 designers ever imagined.
Embedded-RP being one example. Not sure what the best example would be
for eg: SRH programming bits in addresses. 

Would i have choosen to do these functions he way we do them now in IPv6 ?
Probably not, but i think there was and still is no way to to go
through the learning experience. Similarily, we didn't know how to
interconnect two quite similar protocols IPv4/IPv6 in the 90th. SO now
we have a mess of 16++ 4<->6 'NAT' solutions, but we are a lot more knowledgable.

And we have certainly and still try to do things that as you rightfully say
have or should be rejected by strong contributors/WG-chairs. But in many
cases, that real or intended "abuse" is becaue our current protocol IPv6
is just not flexible enough for sane extensions. And given how our hardware
does improve not only in speed but also flexibility over the decades, it is
just sad to see that we do not try more to get the additional value ou of
our hardware by thinking about a more flexible new packet headers.

Yes, variable lengths is a big step. Trust me, i had to fight tooth and nails
to even get it into control plane ca. 15? years back in mLDP FEC addressing,
but it did open a lot of things we couldn't do before with PIM. And
we are now 50 years away from the point in time when we thought fix lengh
is reqired.

I think in hardware we did start to support more and more variable length
with things like MTRIE lookups or as bad as it is waste in IPv4 for IPv6
centric TCAM designs/setup. So i think we are rally at an inflecion
point that would allow us to do better variable length packet header
designs. Whether those would be TLV (like IPv6 extension headers) or
different is something i think we're going to discuss a lot in the MPLS DT.

> and finally price of the very electrons delivered to carry the bit that

I think the few vendors who make all the money with high-speed silicon
based routers not only deserve the pain of more and more users/researchers
wanting to get more easily built innovation out of those chips, i think
even more so, that supporting that direction is a core part of avoiding
to be commoditized by the white box chip designers. To me router land
still looks as if Intel tried to be successful without allowing any
not-in-house developed software to run on their CPU. 

Yes, opening up his proprietary HW stack will likely not work in the
same way as it would for general-purpose CPU, but more easily "programmable"
packet headers plus third-party programmable control plane would go
a long way in exploring and supporting new market opportunities and
business models. Especially for the wide variety of limited domain
networks.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 28 Apr 2021 21:15:27 +0100

Toerless Eckert wrote on 28/04/2021 21:11:
> So i think we are rally at an inflecion point that would allow us to
> do better variable length packet header designs.

Can you remind us if there's even been a point where the fast/good/cheap 
rule didn't apply.

Nick

---

From: Tony Przygienda <[email-masked]>
Date: Wed, 28 Apr 2021 22:51:03 +0200

yepp, researchers are having fun unencumbered largely by harsh realities
while meanwhile large scale IP forwarding is still governed by the
microcents/bit payload that customers can run their business profitably at
and this is governed in turn by the merciless trifecta of physical laws of
power/cooling/space that did not only not change in the last 50 years but
last 500 and probably much longer even before Newton laid his cranium under
the said apple tree ...

-- tony

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 28 Apr 2021 21:53:56 +0100

Tony Przygienda wrote on 28/04/2021 21:51:
> researchers are having fun unencumbered largely by harsh realities

to which: yay!  There is a place in the world for fast+good and we need 
to look at it and understand it.

Nick

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 29 Apr 2021 01:58:51 +0200

> yepp, researchers are having fun unencumbered largely by harsh realities

Well, the better researcher attempt to get more of an understanding
of the harsh realities by working with whats available to them, which is
quite limited - a few P4 platforms. This lack of openness to multi-party
innovation is what causes high-speed router platforms to become less and
less relevant and devolve into commodities, at least at the level where
folks in IETF are interested in them. Obviously the spped&feed hardware
level will stay an ongoing motor of innovation and maybe even sales
margin, but for the packet layer forwarding, we can close shop if we do
not have a strategy to allow new markets/innovation to be brought in.

Heck, take a look at the challengers in 5G/6G. They are already moving
purely into general purpose CPU because those challengers have no way to
put innovation into accelerated forwarding hardware due to lack of access,
complexity and also being stuck in a world of standards such as GTP that make
innovation difficult.

Its totally incomprehensible to me, why engineers who paycheck ultimately 
depends on hardware that already is way more flexible than what our current
protocols use do not try harder to think about better models for third parties
to get more value from this hardware. More flexible, programmable packet forwarding
options are one core option here. And obviously no sane customer
wants something thats not multi-vendor in this space which is where
IETF could come in. And no sane network operator wants another protocol silo.
Which is where IPv4/IPv6 compatibility comes in as a base level expectation.

> last 500 and probably much longer even before Newton laid his cranium under

When single-family home systems are built from 19" CPU server racks
providing compute across FTTH and powered by solar electricity, i think
we are also starting to see big shifts in the economics of those core
factors. But thats digression.

I think we agree that from the protocols we remember, IP/IPv6 has
done a pretty good job providing value out of the hardware
deployed. But we also know that other protocols like Ethernet and MPLS
have shown other sectors where they are doing better. So it seem to me
obvious that as developers we should look ahead and see what it is we
can best do to crearte more value. And certainly we know our hardware
can already do more than what is utilized by the protocols.

Cheers
    Toerless

---

From: Tom Herbert <[email-masked]>
Date: Wed, 28 Apr 2021 17:42:23 -0700

Toerless,

While I agree with your sentiment, this is really a lament about a
segment of the market not about anything IETF has done. For instance,
no IETF standard  requires high performance routers to be implemented
in fixed ASIC, in fact I believe the technology has evolved to the
extent the datapath could go through the CPU simultaneously achieving
both performance and flexibility (my company is working precisely on
that).

The hard problem is legacy and the Least Common Denominator effect. On
the Internet we are bound by the least common denominator of protocols
which is plain TCP/IPv4 and TCP/IPv6 that generally make it through
the Internet. UDP is looking a little better, at least because of QUIC
which took a behemoth to force support. The end result of all this is
the ossification of the Internet which as you point out stifles
innovation and creation of new markets. This is actually the appeal of
limited domains, it creates walled gardens where in which the least
common denominator can be raised and we can innovate in that sand box,
however limited domains don't address the underlying issue unless
their scope continually expands to cover more and more of the
Internet.

Tom

---

From: Dirk Trossen <[email-masked]>
Date: Thu, 29 Apr 2021 07:13:36 +0000

Tom,

I read your comment below as considering limited domains useful for innovation and we got to think about ways to facilitate their integration into the 'more and more of the Internet'? I think that this aspect is crucial when looking at the role of LDs. 

If so, however, couldn't a more flexible addressing be one of those ways, among others?

Best,

Dirk

-----Original Message-----
From: ipv6 [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: 29 April 2021 02:42
To: Toerless Eckert <[email-masked]>
Cc: 6man WG <[email-masked]>
Subject: Re: Limited Domains:

On Wed, Apr 28, 2021 at 4:59 PM Toerless Eckert <[email-masked]> wrote:
> Well, the better researcher attempt to get more of an understanding of 
> the harsh realities by working with whats available to them, which is 
> quite limited - a few P4 platforms. This lack of openness to 
> multi-party innovation is what causes high-speed router platforms to 
> become less and less relevant and devolve into commodities, at least 
> at the level where folks in IETF are interested in them. Obviously the 
> spped&feed hardware level will stay an ongoing motor of innovation and 
> maybe even sales margin, but for the packet layer forwarding, we can 
> close shop if we do not have a strategy to allow new markets/innovation to be brought in.
>
> purely into general purpose CPU because those challengers have no way 
> to put innovation into accelerated forwarding hardware due to lack of 
> access, complexity and also being stuck in a world of standards such 
> as GTP that make innovation difficult.
>
Toerless,

While I agree with your sentiment, this is really a lament about a segment of the market not about anything IETF has done. For instance, no IETF standard  requires high performance routers to be implemented in fixed ASIC, in fact I believe the technology has evolved to the extent the datapath could go through the CPU simultaneously achieving both performance and flexibility (my company is working precisely on that).

The hard problem is legacy and the Least Common Denominator effect. On the Internet we are bound by the least common denominator of protocols which is plain TCP/IPv4 and TCP/IPv6 that generally make it through the Internet. UDP is looking a little better, at least because of QUIC which took a behemoth to force support. The end result of all this is the ossification of the Internet which as you point out stifles innovation and creation of new markets. This is actually the appeal of limited domains, it creates walled gardens where in which the least common denominator can be raised and we can innovate in that sand box, however limited domains don't address the underlying issue unless their scope continually expands to cover more and more of the Internet.

Tom

> Its totally incomprehensible to me, why engineers who paycheck 
> ultimately depends on hardware that already is way more flexible than 
> what our current protocols use do not try harder to think about better 
> models for third parties to get more value from this hardware. More 
> flexible, programmable packet forwarding options are one core option 
> here. And obviously no sane customer wants something thats not 
> multi-vendor in this space which is where IETF could come in. And no sane network operator wants another protocol silo.
> > microcents/bit payload that customers can run their business 
> > profitably at and this is governed in turn by the merciless trifecta 
> > of physical laws of power/cooling/space that did not only not change 
> > in the last 50 years but last 500 and probably much longer even 
> > before Newton laid his cranium under the said apple tree ...
>
> providing compute across FTTH and powered by solar electricity, i 
> think we are also starting to see big shifts in the economics of those 
> core factors. But thats digression.
>
> I think we agree that from the protocols we remember, IP/IPv6 has done 
> a pretty good job providing value out of the hardware deployed. But we 
> also know that other protocols like Ethernet and MPLS have shown other 
> sectors where they are doing better. So it seem to me obvious that as 
> developers we should look ahead and see what it is we can best do to 
> crearte more value. And certainly we know our hardware can already do 
> more than what is utilized by the protocols.
>
> Cheers
>     Toerless
>

---

From: Tom Herbert <[email-masked]>
Date: Thu, 29 Apr 2021 07:04:29 -0700

On Thu, Apr 29, 2021, 12:13 AM Dirk Trossen <[email-masked]> wrote:

> Tom,
>
> I read your comment below as considering limited domains useful for
> innovation and we got to think about ways to facilitate their integration
> into the 'more and more of the Internet'? I think that this aspect is
> crucial when looking at the role of LDs.

Yes, to the extent that LDs are not used as a rationale to create regional
dialects of standard protocols that are non-conforming or not interoperable
with nodes outside the domain.

> If so, however, couldn't a more flexible addressing be one of those ways,

Sure, as long as conformance with RFC8200 and other standards is
maintained. For instance, I'd like to see every flow get assigned it's own
unique source address to prevent correlations being made by third parties
on the Internet that different flows are sourced from the same user (CGNAT
already does this somewhat).

Tom

> While I agree with your sentiment, this is really a lament about a segment
> of the market not about anything IETF has done. For instance, no IETF
> standard  requires high performance routers to be implemented in fixed
> ASIC, in fact I believe the technology has evolved to the extent the
> datapath could go through the CPU simultaneously achieving both performance
> and flexibility (my company is working precisely on that).
>
> The hard problem is legacy and the Least Common Denominator effect. On the
> Internet we are bound by the least common denominator of protocols which is
> plain TCP/IPv4 and TCP/IPv6 that generally make it through the Internet.
> UDP is looking a little better, at least because of QUIC which took a
> behemoth to force support. The end result of all this is the ossification
> of the Internet which as you point out stifles innovation and creation of
> new markets. This is actually the appeal of limited domains, it creates
> walled gardens where in which the least common denominator can be raised
> and we can innovate in that sand box, however limited domains don't address
> the underlying issue unless their scope continually expands to cover more
> and more of the Internet.
>
> Tom
>
> > multi-vendor in this space which is where IETF could come in. And no
> sane network operator wants another protocol silo.

---

From: Toerless Eckert <[email-masked]>
Date: Thu, 29 Apr 2021 21:17:51 +0200

> with nodes outside the domain.

I guess we would have to write down exactly what we mean with "conformance"
or "interoperability" to be sure we talk about the same thing, but 
i would say that "interoperability" is IMHO the most core
requirement.

E.g.: If i come up with a better header than the IPv6 header, for
example lets say simply allowing variable long addresses (and keep
everything else the same, just for the sake of the example), then that
to me would be a valid option. But how would we call it...
"backward compatible, conforming, interoperable" ?

Technically it could support 1:1 transformation between proper IPv6
packets and the subset of that new header packets that represent iPv6 packets
(lets say also for the sake of simplicity all 128 bit address packets).
Aka: This would not be different to some pre-existing header transformations
such as header compression (although this wouldn't be compression for the
128 bit addresses).

While there are other goals of interest, this is the discussion about
whether we could have new better headers that functionally are a superset
of what IPv6 does plsu stuff that the header does and that goes beond
IPv6 (RFC8200) would have to stay in the limited domain.

Which i think would be very easy to show because once such a packet hits a router
that on the outgoing interface only has an IPv6 neighbor, it would need
to translate the packet to IPv6, and it wouldn't translate, so drop/icmp.

This just gives us a lot more freeedom to build better packet encoding
than if we'd be constrained to just rfc8200. As i gave the example
in before: We could fit IP Multicast into IPv4/IPv6 because it could
fit the available addressing. We couldn't do the same for BIER. So
instead of repeating the one-off new headers (like we had to do for
BIER), lets try to figure out what more flexible header would suffice
to add new semantics.

> already does this somewhat).

What the most simple explained example fot this ? I am asking,
because i think you can not well technically defend against your
access service provider doing this correlation (at least down to
subscriber level), so to protect against a snooping access provider,
legal rquirements seem to be more appropriate. And the cloud services
themelves would perpass all data they want from higher layers
due to the wonderful world of browsers and untrustworthy apps. So,
who else along the path poses the biggest risk ? Specifically because
different attackers have totally different amount of money for the
attack and distribute it differently. Some naughty security agency
could easily redirect resources to individual target as opposed
to those attackers more interested in full spectrum marketing analysis.

Cheers
    Toerless

-- 
---
[email-masked]