Subject: RE: v6 option types for IOAM data fields

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 8 Nov 2018 08:36:55 +0000

Hi Mark, Mike,

thanks for summarizing the concerns around leaked packets – and outlining an approach to mitigate those. There are a couple of other challenges we face. Unfortunately we did not get to a discussion in the 6man WG meeting this time due to the limited time we had. The meeting slides https://datatracker.ietf.org/meeting/103/materials/slides-103-6man-in-situ-oam-ipv6-options-00 provide a summary of concerns.

In a nutshell:

1.       Potential HbH ext header and IOAM option insertion and removal by transit nodes (i.e. “en route”) in a restricted administrative domain.

a)       How do you deal with PMTU? – Packet size changes might exceed PMTU.

b)      Misleading ICMP errors confusing the source

c)       Possible leaks that affect the forwarding behavior and state of network elements outside the domain.

2.       Incremental Trace IOAM HbH Option – which is to support a hardware-friendly implementation: Changes Option Data Len en-route.

a)       Dealing with PMTU– Packet size changes can exceed PMTU; see above

3.       Clarify applicability statement and scope

While 3. is something that is in the works, there are no easy answers to 1. and 2. Considerations:

On 1. Supporting HbH ext header and option insertion/removal in transit. A couple of solution approaches…

1.       Fix PMTU and offset for packet size change in PMTU discovery - https://tools.ietf.org/html/draft-troan-6man-pmtu-solution-space-00
Hope that we can make progress towards a solution tomorrow…

2.       IP-in-IP with IOAM extension header in the *inner* packet.
New IPv6 packet is created with encapsulating node as source and the original destination as the destination
(this is slightly different than what you, Mark, suggest – because we’d like to keep the forwarding path the same. Tunneling with ULA would not get us there; the slides above have a diagram that depicts the potential encap):

a)       Payload of this packet is the original IPv6 packet along with an extension header inserted inside.

b)      The original packet is restored by removing the outer IPv6 header and the inner extension header by a node at the domain boundary.

c)       Caveats:

                                                               i.      Modified packet may still leak – but will only confuse the destination node. The dest node will likely send an ICMP back to the encap node, which gets the encap node an understanding that something is going wrong.

                                                             ii.      ECMP computation needs to be reworked.

                                                           iii.      Complex/costly implementation in HW & SW – IOAM-capable nodes need to hunt for the IOAM data fields in the inner packet.

3.       No support of IOAM in transit network. Support only source initiated IOAM tracing, proof of transit..
Caveat: Limits usage of IOAM significantly.

On 2. Incremental Trace IOAM HbH Option

1.       Use PMTU to determine max possible Incremental Trace IOAM Option length

2.       Do not support Incremental Trace IOAM Option in IPv6.

While there isn’t an easy answer, IMHO we should still try to find a workable and implementable solution. I sometimes compare the situation with the road network: What we have today is a network of gravel roads (Internet with often poor ext header processing) and cars are built to cope with gravel roads. Now faster and comfy cars become available, though they only work on tared/concrete roads (specific administrative domains). If you try to drive these new cars on gravel roads they break and the wrecks jam the roads…, still we probably want to allow folks to buy and use the fast & comfy cars, because they will also push for better roads  as well (do proper ext header processing).

Thoughts?

Thanks, Frank

From: Mark Smith <[email-masked]>
Sent: Dienstag, 30. Oktober 2018 05:26
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: Re: v6 option types for IOAM data fields

Hi Frank,

On Tue., 30 Oct. 2018, 05:36 Frank Brockners (fbrockne), <[email-masked]<mailto:[email-masked]>> wrote:
Thanks Mike. On the scope of an IOAM deployment:
draft-ietf-ippm-ioam-data-04 clarifies in section 3 that IOAM is a domain focused feature, i.e. not expected to be deployed on the open Internet.

That still violates RFC 8200, there are no exceptions for "closed" domains.

From section 3:

"Designers of
   carrier protocols for IOAM must specify mechanisms to ensure that
   IOAM data stays within an IOAM domain.  In addition, the operator of
   such a domain is expected to put provisions in place to ensure that
   IOAM data does not leak beyond the edge of an IOAM domain, e.g. using
   for example packet filtering methods."

Neither the designers or the operators can ensure this IOAM information won't leak.

Possible reasons it can leak:

- operator configuration error - e.g. forgetting to configure the domain boundary (e.g. router with multiple domain exit interfaces, forgetting to add that option on one of them), or misconfiguring it.

- partial device failure - the device still forwards packets, but ceases to remove this information due to a hardware fault that has appeared

- vendor implementation bugs that fails to remove the information.

Any or all of these can occur, and as it is on egress, the consequences may not be visible to the domain network operator, because network operators rarely inspect packets after they've left their network. Once a packet is sent onto somebody else, it is assumed to have been sent without faults.

There are many instances of leaks at the boundary of what are supposed to be closed domains, such as packets containing RFC1918 private addresses (in packet headers themselves, or in DNS packets - such a big problem that www.as112.net<http://as112.net> was created), and route leaks e.g. https://bgpmon.net/how-the-internet-in-australia-went-down-under/

As operators usually have only one device at the edge of the domain, that would be performing the domain boundary function, that device becomes a single point of failure. Enforcement is therefore quite fragile. A domain boundary enforced by two domain edge devices in-line would increase robustness, however I'd think that would be rarely deployed, because we haven't seen that commonly with IPv4 NAT.

The only way for an operator to ensure these leaks don't and never occur is for the domain to literally not be physically connected to the Internet i.e. the domain is air gapped from the Internet. If there is a link between the domain and the Internet that can carry IP packets, then there is always a possibility that information can leak from the domain.

So accepting that leaks can always occur, a far more robust option is to leave the original packet alone entirely, and encapsulate it in a domain local IPv6 header (i.e RFC2473, section 3.1) with domain local limited addresses (i.e. ULA addresses) and the additional EH.

That still doesn't ensure that packet won't leak outside the domain, because those packets will follow a default route, however as the packet's addressing is invalid outside of the domain, that packet will be dropped once it reaches either an invalid Internet address filter, or a default free Internet router. The packet with the failed-to-be-removed outer ULA IPv6 header will never reach the destination address of the inner packet, because the DA of the inner packet is in the ULA IPv6 packet's payload.

Failure of the mechanism would be much more obvious, localised to the domain implementing the mechanism (rather than being externalised to somebody else's domain/network), and absolute, because the failure mode is packets being entirely discarded.

Regards,
Mark.

Frank

-----Original Message-----
From: C. M. Heard <[email-masked]<mailto:[email-masked]>>
Sent: Montag, 29. Oktober 2018 16:03
To: 6man <[email-masked]<mailto:[email-masked]>>; IPPM <[email-masked]<mailto:[email-masked]>>
Cc: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Subject: Re: v6 option types for IOAM data fields

On Thu, 25 Oct 2018 15:06:57 +0000 Frank Brockners (fbrockne) wrote:
> Quick heads up: In the 6MAN meeting in BKK, we’ll review
> draft-ioametal-ippm-6man-ioam-ipv6-options-01 – which requests 2
> option types from the DO/HbyH options sub-registry.
>
> While the bulk of the IOAM work is progressed in the IPPM WG, we’d
> greatly appreciate your feedback on
> draft-ioametal-ippm-6man-ioam-ipv6-options-01,
> which defines how IOAM data fields are carried using v6 extension headers.
> Cc’ing the IPPM WG as well, to keep everyone on the same page.

I have two brief comments on this work.

First, I see that the Incremental Tracing Option changes length in transit.
It is not appropriate for it to be carried in an IPv6 option intended for use on the open Internet, for exactly the same reason that insertion of extension headers by intermediate nodes is not allowed on the open Internet.

Second, I see that two IPv6 option code points are requested, one with the "chg" flag set, the other with the "chg" flag clear. While there is no harm in this, it is not strictly necessary; the only real purpose of this flag is to determine whether the option data is or is not included in the Authentication Header Integrity Check Value computation.

Mike Heard
--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]<mailto:[email-masked]>
Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: Mark Smith <[email-masked]>
Date: Thu, 8 Nov 2018 22:40:29 +1100

Hi Frank,

> c)       Possible leaks that affect the forwarding behavior and state of network elements outside the domain.

Yes, all of the above, good summary.

An additional one is that if you were are receiver of this outside of
the domain because it as leaked, at the destination, after the packet
as traversed a number of ASes over the Internet, there is nothing to
identify the device/AS that inserted the EH. Very time consuming to
troubleshoot who inserted the EH but didn't remove it (you have to
walk the AS path, contacting each operator, asking them to verify
their config, get packet captures etc.), and all the while you may
have a number of angry customers/end-users.

>                                                                i.      Modified packet may still leak – but will only confuse the destination node. The dest node will likely send an ICMP back to th…

This is certainly better, because the source address identifies the
device that inserted the EH of the packet that leaked to where ever it
leaked to.

What the source address should be is an interesting question.

If it is a global/public source address, then anybody who receives the
packet leaked with the EH can identify who inserted it and therefore
who didn't remove it.

OTOH, if it is a private/local ULA Source Address, then BCP38 source
address filters would cause the invalid leaked packet to be dropped
sooner, rather than being forwarded onto the final destination. Total
packet loss of OAM encapsulated packets would make the failure obvious
to the OAM domain operator. Slight drawback is BCP38 filters haven't
been universally implemented across the Internet.

I think I like the latter better because commonly it would localise
the symptoms of the failure to remove the EH to the domain that
inserted it, rather than having to have an external party get involved
in troubleshooting.

>                                                            iii.      Complex/costly implementation in HW & SW – IOAM-capable nodes need to hunt for the IOAM data fields in the inner packet.

Yes, the implementation complexity of insertion is another concern -
encapsulation is the most common and simplest cross layer forwarding
function performed and there's decades of history in optimising it.

The root cause of both the possible leak and the risk that the
inserted EH will reach the final packet destination is that the Dest.
Addr. of the packet with the inserted EH is both valid outside of the
domain, and valid all the way across the Internet to the final and
actual original packet destination. That creates a "default permit" if
a packet leaks with the inserted EH rather than a far better "default
deny" in this failure situation.

In this OAM case, it seems to me the ideal and the requirements would be:

- have the packet with the added OAM information follow the same path
within the domain that the same packet without the OAM information
would follow within the domain

- have the Dest. Addr. of the packet with the added OAM information
only be valid within the OAM domain i.e. an "interior DA", so that if
through failure the packet leaks, it isn't a valid DA on the Internet
and will be dropped by, if nothing else earlier, an Internet router
with a default free route table.

- encapsulate rather than insert, because encapsulation would be
simpler, and is the traditional way of adding information to PDUs
across many decades and many protocols (I can only think of VLAN ID
insertion as the only existing counter example, and according to Radia
Perlman in her book, that is only because at the time they weren't
sure if all NICs commonly available could successfully encapsulate a
full sized ethernet frame inside another one).

In other words, preserve the original IPv6 packet, add an OAM header
in front of it, then encapsulate it in another header to forward
within and across the OAM domain.

Bog standard IGP/LDP MPLS could achieve those requirements, with the
OAM header inserted after the final MPLS tag and before the IPv6
header. The MPLS/OAM/IPv6 packet follows an LSP across the domain that
matches the path that the IPv6 packet without MPLS would follow across
the domain. In that scenario, the MPLS LSP across the domain is the
"interior DA" that isn't valid outside of the OAM domain, because MPLS
wouldn't (normally) be valid outside of the OAM domain (i.e. OAM
domain is MPLS domain 1:1). If MPLS fails to be removed at the OAM
domain egress edge, the packet is dropped immediately, so a fail safe,
localised to the OAM domain.

For an IPv6-in-IPv6 scenario:

- for each OAM domain interior IPv6 routes/prefixes used by hosts
(i.e. hosts have addresses from these prefixes), automatically create
a matching interior OAM route/prefix (maybe just a /128), 1:1, with
the interior OAM route/prefix originated by the same router(s) where
the interior IPv6 routes/prefixes are created/originated. The interior
OAM routes come from ULA space or perhaps a new designated and
reserved private/local OAM address space for this purpose.

When a end host originated packet ingresses the OAM domain edge, the
DA address of the packet is looked up to find the 1:1 matching
internal OAM route/prefix/address. This internal OAM
route/prefix/address is used as the DA for the outer IPv6 header,
outer header SA is the OAM address of the OAM ingress device, an OAM
header follows, and then the original, inner IPv6 packet follows
without modification.

As the OAM route/prefix/address is originated at the same place as the
OAM domain interior IPv6 route/prefix, the path the now IPv6-in-IPv6
packet follows through the OAM domain should be the same as that which
the original, now inner packet would have followed with its DA. In a
sense this OAM route/prefix/address is an internal routeable index or
proxy for the inner packet destination prefix.

- for exterior routes, IPv6-in-IPv6 tunnelling too between BGP border
routers per RFC1772, "A.2.3 Encapsulation", with the addition of the
OAM header between the new outer IPv6 header and the inner original
IPv6 packet, and OAM local addresses for the tunnel end points. This
avoids having to create 1:1 matching internal OAM route/prefix/address
for each exterior route.

(And perhaps a somewhat radical idea for reducing tunnelling overhead.
All of this tunnelling stuff already exists for IPv4; and it is sunk
cost. Would there be much harm in using IPv4 as the interior outer
tunnelling protocol for this IPv6 traffic with the added OAM
information? I'd much rather keep running an IPv4 OAM island to carry
IPv6 than any splitting apart of IPv6 packets to insert EHs in them
...)

> 2.       Do not support Incremental Trace IOAM Option in IPv6.

Could what I've suggested above solve those? I think the use of
tunnelling/encapsulation supports a transit OAM scenario.

Regards,
Mark.

> On Tue., 30 Oct. 2018, 05:36 Frank Brockners (fbrockne), <[email-masked]> wrote:
>
> There are many instances of leaks at the boundary of what are supposed to be closed domains, such as packets containing RFC1918 private addresses (in packet headers themselves, or in DNS packets - such a big problem that www.as112.net was created), and route leaks e.g. https://bgpmon.net/how-the-internet-in-australia-went-down-under/
>
>
>
> From: C. M. Heard <[email-masked]>
> To: 6man <[email-masked]>; IPPM <[email-masked]>
> Cc: Frank Brockners (fbrockne) <[email-masked]>
> [email-masked]

---

From: Tom Herbert <[email-masked]>
Date: Thu, 8 Nov 2018 13:52:40 -0800

On Thu, Nov 8, 2018, 12:37 AM Frank Brockners (fbrockne) <[email-masked]>
wrote:

> Hi Mark, Mike,
>
>
>
> thanks for summarizing the concerns around leaked packets – and outlining
> an approach to mitigate those. There are a couple of other challenges we
> face. Unfortunately we did not get to a discussion in the 6man WG meeting
> this time due to the limited time we had. The meeting slides
> https://datatracker.ietf.org/meeting/103/materials/slides-
> 103-6man-in-situ-oam-ipv6-options-00 provide a summary of concerns.
>
>
>
> In a nutshell:
>
> 1.       Potential HbH ext header and IOAM option insertion and removal
> by transit nodes (i.e. “en route”) in a restricted administrative domain.
>

There's already been significant discussion in 6man list about option
insertion/removal. I believe the conclusion is that RFC8200 prohibits it
exactly for the reasons listed above and maybe a few others. I don't think
that restricting the use to a limited domain is sufficient rationale to
create an exception. As mentioned previously, encapsulation can be used to
achieve the proper effect.

> 2.       Incremental Trace IOAM HbH Option – which is to support a
> hardware-friendly implementation: Changes Option Data Len en-route.
>

This has basically the same problems as option insertion. RFC8200 only
allow Option Data to be changeable en route, and that is only if the third
bit of the option type is set.

Here are two other issues for these:

- No accountability as to who changed the packet in flight
- They break authentication header

> 1.       Fix PMTU and offset for packet size change in PMTU discovery -
> https://tools.ietf.org/html/draft-troan-6man-pmtu-solution-space-00
> 2.       IP-in-IP with IOAM extension header in the **inner** packet.
> (this is slightly different than what you, Mark, suggest – because we’d
> like to keep the forwarding path the same. Tunneling with ULA would not get
> us there; the slides above have a diagram that depicts the potential encap):
>
> b)      The original packet is restored by removing the outer IPv6 header
> and the inner extension header by a node at the domain boundary.
>
> c)       Caveats:
>
>                                                                i.      Modified
> packet may still leak – but will only confuse the destination node. The
> dest node will likely send an ICMP back to the encap node, which gets the
> encap node an understanding that something is going wrong.
>
>                                                              ii.      ECMP
>                                                            iii.      Complex/costly
> implementation in HW & SW – IOAM-capable nodes need to hunt for the IOAM
> data fields in the inner packet.
>
> 3.       No support of IOAM in transit network. Support only source
> initiated IOAM tracing, proof of transit..
> While there isn’t an easy answer, IMHO we should still try to find a
> workable and implementable solution. I sometimes compare the situation with
> the road network: What we have today is a network of gravel roads (Internet
> with often poor ext header processing) and cars are built to cope with
> gravel roads. Now faster and comfy cars become available, though they only
> work on tared/concrete roads (specific administrative domains). If you try
> to drive these new cars on gravel roads they break and the wrecks jam the
> roads…, still we probably want to allow folks to buy and use the fast &
> comfy cars, because they will also push for better roads  as well (do
> proper ext header processing).
>
>
>
> Thoughts?
>
>
>
> Thanks, Frank
>
>
>
>
>
> *From:* Mark Smith <[email-masked]>
> *Sent:* Dienstag, 30. Oktober 2018 05:26
> *To:* Frank Brockners (fbrockne) <[email-masked]>
> *Cc:* C. M. Heard <[email-masked]>; 6man WG <[email-masked]>;
> [email-masked]
> *Subject:* Re: v6 option types for IOAM data fields
>
>
>
>
> Hi Frank,
>
>
>
> draft-ietf-ippm-ioam-data-04 clarifies in section 3 that IOAM is a domain
> focused feature, i.e. not expected to be deployed on the open Internet.
>
>
>
> - operator configuration error - e.g. forgetting to configure the domain
> boundary (e.g. router with multiple domain exit interfaces, forgetting to
> add that option on one of them), or misconfiguring it.
>
>
>
> - partial device failure - the device still forwards packets, but ceases
> to remove this information due to a hardware fault that has appeared
>
>
>
> Any or all of these can occur, and as it is on egress, the consequences
> may not be visible to the domain network operator, because network
> operators rarely inspect packets after they've left their network. Once a
> packet is sent onto somebody else, it is assumed to have been sent without
> faults.
>
>
>
> There are many instances of leaks at the boundary of what are supposed to
> be closed domains, such as packets containing RFC1918 private addresses (in
> packet headers themselves, or in DNS packets - such a big problem that www.a
> s112.net <http://as112.net> was created), and route leaks e.g.
> https://bgpmon.net/how-the-internet-in-australia-went-down-under/
>
>
>
> As operators usually have only one device at the edge of the domain, that
> would be performing the domain boundary function, that device becomes a
> single point of failure. Enforcement is therefore quite fragile. A domain
> boundary enforced by two domain edge devices in-line would increase
> robustness, however I'd think that would be rarely deployed, because we
> haven't seen that commonly with IPv4 NAT.
>
>
>
> The only way for an operator to ensure these leaks don't and never occur
> is for the domain to literally not be physically connected to the Internet
> i.e. the domain is air gapped from the Internet. If there is a link between
> the domain and the Internet that can carry IP packets, then there is always
> a possibility that information can leak from the domain.
>
>
>
> So accepting that leaks can always occur, a far more robust option is to
> leave the original packet alone entirely, and encapsulate it in a domain
> local IPv6 header (i.e RFC2473, section 3.1) with domain local limited
> addresses (i.e. ULA addresses) and the additional EH.
>
>
>
> That still doesn't ensure that packet won't leak outside the domain,
> because those packets will follow a default route, however as the packet's
> addressing is invalid outside of the domain, that packet will be dropped
> once it reaches either an invalid Internet address filter, or a default
> free Internet router. The packet with the failed-to-be-removed outer ULA
> IPv6 header will never reach the destination address of the inner packet,
> because the DA of the inner packet is in the ULA IPv6 packet's payload.
>
>
>
> Failure of the mechanism would be much more obvious, localised to the
> domain implementing the mechanism (rather than being externalised to
> somebody else's domain/network), and absolute, because the failure mode is
> packets being entirely discarded.
>
>
>
> Regards,
>
> Mark.
>
>
>
>
>
>
> Frank
>
> It is not appropriate for it to be carried in an IPv6 option intended for
> use on the open Internet, for exactly the same reason that insertion of
> extension headers by intermediate nodes is not allowed on the open Internet.
>
> Second, I see that two IPv6 option code points are requested, one with the
> "chg" flag set, the other with the "chg" flag clear. While there is no harm
> in this, it is not strictly necessary; the only real purpose of this flag
> is to determine whether the option data is or is not included in the
> Authentication Header Integrity Check Value computation.
>
> Mike Heard

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Mon, 12 Nov 2018 08:40:00 +0000

Hi Mark,

thanks for the details. On your suggested solution for v6-in-v6: Let's assume a setup with source and destination host addresses (SA, DA), as well as an IOAM domain which has domain Edge-routers E(i). If I understand your suggestion correctly, then the ingress edge router with source address E-SA would have /128 routes to all other domain Edge-routers E-DA(i). For v6-in-v6, the ingress edge router would use its own source address as the source address of the encapsulated packet, and it would use E-DA(j) as the destination address, where E-DA(j) is the address that points to the same outgoing link as the original (now inner) DA.
Now the question: Why would this choice of E-DA(j) ensure that the encapsulated packet leaves the IOAM domain at the same egress router, as would a packet without the extra v6 header? Or in other terms, why would the encapsulated packet be forwarded the same way as a non-encapsulated packet in the IOAM domain?

Many thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]> 
Sent: Donnerstag, 8. November 2018 12:40
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: Re: v6 option types for IOAM data fields

Hi Frank,

An additional one is that if you were are receiver of this outside of the domain because it as leaked, at the destination, after the packet as traversed a number of ASes over the Internet, there is nothing to identify the device/AS that inserted the EH. Very time consuming to troubleshoot who inserted the EH but didn't remove it (you have to walk the AS path, contacting each operator, asking them to verify their config, get packet captures etc.), and all the while you may have a number of angry customers/end-users.

> On 1. Supporting HbH ext header and option insertion/removal in 
> transit. A couple of solution approaches…
>
> New IPv6 packet is created with encapsulating node as source and the 
> original destination as the destination (this is slightly different than what you, Mark, suggest – because we’d like to keep the forwarding path the same. Tunneling with ULA would not get us there; the slides above have a diagram that depicts the potential encap):
>

This is certainly better, because the source address identifies the device that inserted the EH of the packet that leaked to where ever it leaked to.

If it is a global/public source address, then anybody who receives the packet leaked with the EH can identify who inserted it and therefore who didn't remove it.

OTOH, if it is a private/local ULA Source Address, then BCP38 source address filters would cause the invalid leaked packet to be dropped sooner, rather than being forwarded onto the final destination. Total packet loss of OAM encapsulated packets would make the failure obvious to the OAM domain operator. Slight drawback is BCP38 filters haven't been universally implemented across the Internet.

I think I like the latter better because commonly it would localise the symptoms of the failure to remove the EH to the domain that inserted it, rather than having to have an external party get involved in troubleshooting.

>                                                            iii.      Complex/costly implementation in HW & SW – IOAM-capable nodes need to hunt for the IOAM data fields in the inner packet.

Yes, the implementation complexity of insertion is another concern - encapsulation is the most common and simplest cross layer forwarding function performed and there's decades of history in optimising it.

The root cause of both the possible leak and the risk that the inserted EH will reach the final packet destination is that the Dest.
Addr. of the packet with the inserted EH is both valid outside of the domain, and valid all the way across the Internet to the final and actual original packet destination. That creates a "default permit" if a packet leaks with the inserted EH rather than a far better "default deny" in this failure situation.

- have the packet with the added OAM information follow the same path within the domain that the same packet without the OAM information would follow within the domain

- have the Dest. Addr. of the packet with the added OAM information only be valid within the OAM domain i.e. an "interior DA", so that if through failure the packet leaks, it isn't a valid DA on the Internet and will be dropped by, if nothing else earlier, an Internet router with a default free route table.

- encapsulate rather than insert, because encapsulation would be simpler, and is the traditional way of adding information to PDUs across many decades and many protocols (I can only think of VLAN ID insertion as the only existing counter example, and according to Radia Perlman in her book, that is only because at the time they weren't sure if all NICs commonly available could successfully encapsulate a full sized ethernet frame inside another one).

In other words, preserve the original IPv6 packet, add an OAM header in front of it, then encapsulate it in another header to forward within and across the OAM domain.

Bog standard IGP/LDP MPLS could achieve those requirements, with the OAM header inserted after the final MPLS tag and before the IPv6 header. The MPLS/OAM/IPv6 packet follows an LSP across the domain that matches the path that the IPv6 packet without MPLS would follow across the domain. In that scenario, the MPLS LSP across the domain is the "interior DA" that isn't valid outside of the OAM domain, because MPLS wouldn't (normally) be valid outside of the OAM domain (i.e. OAM domain is MPLS domain 1:1). If MPLS fails to be removed at the OAM domain egress edge, the packet is dropped immediately, so a fail safe, localised to the OAM domain.

- for each OAM domain interior IPv6 routes/prefixes used by hosts (i.e. hosts have addresses from these prefixes), automatically create a matching interior OAM route/prefix (maybe just a /128), 1:1, with the interior OAM route/prefix originated by the same router(s) where the interior IPv6 routes/prefixes are created/originated. The interior OAM routes come from ULA space or perhaps a new designated and reserved private/local OAM address space for this purpose.

When a end host originated packet ingresses the OAM domain edge, the DA address of the packet is looked up to find the 1:1 matching internal OAM route/prefix/address. This internal OAM route/prefix/address is used as the DA for the outer IPv6 header, outer header SA is the OAM address of the OAM ingress device, an OAM header follows, and then the original, inner IPv6 packet follows without modification.

As the OAM route/prefix/address is originated at the same place as the OAM domain interior IPv6 route/prefix, the path the now IPv6-in-IPv6 packet follows through the OAM domain should be the same as that which the original, now inner packet would have followed with its DA. In a sense this OAM route/prefix/address is an internal routeable index or proxy for the inner packet destination prefix.

- for exterior routes, IPv6-in-IPv6 tunnelling too between BGP border routers per RFC1772, "A.2.3 Encapsulation", with the addition of the OAM header between the new outer IPv6 header and the inner original
IPv6 packet, and OAM local addresses for the tunnel end points. This avoids having to create 1:1 matching internal OAM route/prefix/address for each exterior route.

(And perhaps a somewhat radical idea for reducing tunnelling overhead.
All of this tunnelling stuff already exists for IPv4; and it is sunk cost. Would there be much harm in using IPv4 as the interior outer tunnelling protocol for this IPv6 traffic with the added OAM information? I'd much rather keep running an IPv4 OAM island to carry
IPv6 than any splitting apart of IPv6 packets to insert EHs in them
...)

> 2.       Do not support Incremental Trace IOAM Option in IPv6.

Could what I've suggested above solve those? I think the use of tunnelling/encapsulation supports a transit OAM scenario.

Regards,
Mark.

> - partial device failure - the device still forwards packets, but 
> ceases to remove this information due to a hardware fault that has 
> appeared
>
>
>
> There are many instances of leaks at the boundary of what are supposed 
> to be closed domains, such as packets containing RFC1918 private 
> addresses (in packet headers themselves, or in DNS packets - such a 
> big problem that www.as112.net was created), and route leaks e.g.

---

From: Mark Smith <[email-masked]>
Date: Wed, 21 Nov 2018 12:28:04 +1100

Hi Frank,

Sorry not to get back to you sooner.

On Mon, 12 Nov 2018 at 19:40, Frank Brockners (fbrockne)
<[email-masked]> wrote:
>
> Hi Mark,
>
> thanks for the details. On your suggested solution for v6-in-v6: Let's assume a setup with source and destination host addresses (SA, DA), as well as an IOAM domain which has domain Edge-routers E(i). If I understand your suggestion correctly, then the ingress edge router with source address E-SA would have /128 routes to all other domain Edge-routers E-DA(i).

Close, but not quite.

Rather than having a /128 per Edge-router, you would have a domain
internal /128 per non-IOAM prefix announced by each Edge-edge router,
1:1. You do this so that you can have the same IGP metric for the
domain internal /128 for each of the corresponding non-IOAM prefixes.
That then means that the IOAM tunnelled packet should then follow the
same path through the IOAM domain as the packet if it wasn't IOAM
tunnelled.

e.g. IOAM Router A has the following global prefixes configured on its
interfaces, with the following IGP metrics:

intf A: 2001:db8:0:1234::/64, metric 10

intf B: 2001:db8:0:2345::/64, metric 40

intf C: 2001:db8:0:4567::/64, metric 100
intf C: 2001:db8:0:abcd::/64, metric 150

intf D: 2001:db8:0:4321::/64, metric 10

Note that intf C has been configured with two different prefixes with
different metrics.

When these prefixes are configured on the interfaces, IOAM Router A
would also automatically generate and announce into the IGP (or BGP
with No Export community) matching IOAM internal /128s for each of
them, using a ULA or other internal scope prefix (perhaps a special
IANA reserved one for this IOAM purpose so routers could be factory
configured with it) - I'll use fd04:3f29:de90::/48 for this example.

The corresponding example IOAM /128s  for IOAM Router A's interfaces
would (could) be:

intf A: fd04:3f29:de90::1234/128, metric 10

intf B: fd04:3f29:de90::2345/128, metric 40

intf C: fd04:3f29:de90::4567/128, metric 100
intf C: fd04:3f29:de90::abcd/128, metric 150

intf D: fd04:3f29:de90::4321/128, metric 10

(There may be better schemes to generate the matching global /64:IOAM
/128 prefixes, its probably useful to encode some unique part of the
global prefix in the corresponding /128.)

So when a packet enters the IOAM domain at IOAM Router Z, with a
destination address that falls within 2001:db8:0:1234::/64 (IOAM
Router A's intf A: prefix), the outer tunnel header's DA would be the
corresponding fd04:3f29:de90::1234/128 (IOAM Router A's intf A: OAM
/128 prefix). This post-tunnel packet should follow the same path
through the IOAM domain that a packet destined towards
2001:db8:0:1234::/64 should, from IOAM Router Z to IOAM Router A,
because 2001:db8:0:1234::/64 and fd04:3f29:de90::1234/128 prefixes
have both the same origin router and the same path metrics.

If another packet enters the IOAM domain at IOAM Router Z, with a
destination address that falls within 2001:db8:0:4567::/64 (IOAM
Router A intf C's second prefix), then the outer packet tunnel DA
would be the corresponding fd04:3f29:de90::4567/128 ((IOAM Router A
intf C's second and matching IOAM /128 prefix). As the metric for
2001:db8:0:4567::/64 / fd04:3f29:de90::4567/128 on IOAM Router A is
100, the path through the IOAM domain between IOAM Router Z and IOAM
Router A for this packet (IOAM tunnel/tagged or not) could be
different to the one taken in the previous example.

For completeness, for different IOAM Router Y, with a destination
address that falls within 2001:db8:0:abcd::/64 (IOAM Router A intf C's
second prefix), then the outer packet tunnel DA would be the
corresponding fd04:3f29:de90::abcd/128 ((IOAM Router A intf C's second
and matching IOAM /128 prefix).

> For v6-in-v6, the ingress edge router would use its own source address as the source address of the encapsulated packet, and it would use E-DA(j) as the destination address, where E-DA(j) is the address that points to the same outgoing link as the original (now inner) DA.

Hopefully the above example answers that. There are automatically
generated IOAM tunnel end-point /128s per individual non-IOAM prefix,
rather than per IOAM router, and as the IOAM /128s are per non-IOAM
prefix, they can have and carry per prefix specific metric information
that matches the corresponding non-IOAM prefix.

This would the size of the IGP's route table, as there is now a
matching IOAM /128 for each non-IOAM prefix. However, there is an
optimisation opportunity. If two different non-IOAM prefixes have the
same metric at the origin IOAM router (as intf A:
2001:db8:0:1234::/64, metric 10, and intf D: fd04:3f29:de90::4321/128,
metric 10 do above on IOAM Router A), then a per-router IOAM /128
could be used instead, suppressing the generation of 1:1 per-prefix
IOAM /128s when origin prefix metrics are the same.

Regards,
Mark.