Subject: Re: [ippm] v6 option types for IOAM data fields

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Wed, 13 Mar 2019 16:23:23 +0000

With the upcoming meeting in Prague, I wanted to come back to this discussion:

We've just recently posted a new draft (draft-ioametal-ippm-6man-ioam-ipv6-deployment-00) - which summarizes a set of deployment considerations for IOAM with IPv6, i.e. complementing 6man draft-ioametal-ippm-6man-ioam-ipv6-options-01. draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 also includes the deployment scenario that Mark detailed below. 

Appreciate your thoughts - especially from 6man to provide for proper guidance to IPPM.

Thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]> 
Sent: Mittwoch, 21. November 2018 02:28
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: Re: v6 option types for IOAM data fields

Hi Frank,

Sorry not to get back to you sooner.

On Mon, 12 Nov 2018 at 19:40, Frank Brockners (fbrockne) <[email-masked]> wrote:
>
> Hi Mark,
>
> thanks for the details. On your suggested solution for v6-in-v6: Let's assume a setup with source and destination host addresses (SA, DA), as well as an IOAM domain which has domain Edge-routers E(i). If I understand your suggestion correctly, then the ingress edge router with source address E-SA would have /128 routes to all other domain Edge-routers E-DA(i).

Close, but not quite.

Rather than having a /128 per Edge-router, you would have a domain internal /128 per non-IOAM prefix announced by each Edge-edge router, 1:1. You do this so that you can have the same IGP metric for the domain internal /128 for each of the corresponding non-IOAM prefixes.
That then means that the IOAM tunnelled packet should then follow the same path through the IOAM domain as the packet if it wasn't IOAM tunnelled.

e.g. IOAM Router A has the following global prefixes configured on its interfaces, with the following IGP metrics:

intf A: 2001:db8:0:1234::/64, metric 10

intf B: 2001:db8:0:2345::/64, metric 40

intf C: 2001:db8:0:4567::/64, metric 100 intf C: 2001:db8:0:abcd::/64, metric 150

intf D: 2001:db8:0:4321::/64, metric 10

Note that intf C has been configured with two different prefixes with different metrics.

When these prefixes are configured on the interfaces, IOAM Router A would also automatically generate and announce into the IGP (or BGP with No Export community) matching IOAM internal /128s for each of them, using a ULA or other internal scope prefix (perhaps a special IANA reserved one for this IOAM purpose so routers could be factory configured with it) - I'll use fd04:3f29:de90::/48 for this example.

The corresponding example IOAM /128s  for IOAM Router A's interfaces would (could) be:

intf A: fd04:3f29:de90::1234/128, metric 10

intf B: fd04:3f29:de90::2345/128, metric 40

intf C: fd04:3f29:de90::4567/128, metric 100 intf C: fd04:3f29:de90::abcd/128, metric 150

intf D: fd04:3f29:de90::4321/128, metric 10

(There may be better schemes to generate the matching global /64:IOAM
/128 prefixes, its probably useful to encode some unique part of the global prefix in the corresponding /128.)

So when a packet enters the IOAM domain at IOAM Router Z, with a destination address that falls within 2001:db8:0:1234::/64 (IOAM Router A's intf A: prefix), the outer tunnel header's DA would be the corresponding fd04:3f29:de90::1234/128 (IOAM Router A's intf A: OAM
/128 prefix). This post-tunnel packet should follow the same path through the IOAM domain that a packet destined towards
2001:db8:0:1234::/64 should, from IOAM Router Z to IOAM Router A, because 2001:db8:0:1234::/64 and fd04:3f29:de90::1234/128 prefixes have both the same origin router and the same path metrics.

If another packet enters the IOAM domain at IOAM Router Z, with a destination address that falls within 2001:db8:0:4567::/64 (IOAM Router A intf C's second prefix), then the outer packet tunnel DA would be the corresponding fd04:3f29:de90::4567/128 ((IOAM Router A intf C's second and matching IOAM /128 prefix). As the metric for
2001:db8:0:4567::/64 / fd04:3f29:de90::4567/128 on IOAM Router A is 100, the path through the IOAM domain between IOAM Router Z and IOAM Router A for this packet (IOAM tunnel/tagged or not) could be different to the one taken in the previous example.

For completeness, for different IOAM Router Y, with a destination address that falls within 2001:db8:0:abcd::/64 (IOAM Router A intf C's second prefix), then the outer packet tunnel DA would be the corresponding fd04:3f29:de90::abcd/128 ((IOAM Router A intf C's second and matching IOAM /128 prefix).

> For v6-in-v6, the ingress edge router would use its own source address as the source address of the encapsulated packet, and it would use E-DA(j) as the destination address, where E-DA(j) is the address that points to the same outgoing link as the original (now inner) DA.
> Now the question: Why would this choice of E-DA(j) ensure that the encapsulated packet leaves the IOAM domain at the same egress router, as would a packet without the extra v6 header? Or in other terms, why would the encapsulated packet be forwarded the same way as a non-encapsulated packet in the IOAM domain?
>

Hopefully the above example answers that. There are automatically generated IOAM tunnel end-point /128s per individual non-IOAM prefix, rather than per IOAM router, and as the IOAM /128s are per non-IOAM prefix, they can have and carry per prefix specific metric information that matches the corresponding non-IOAM prefix.

This would the size of the IGP's route table, as there is now a matching IOAM /128 for each non-IOAM prefix. However, there is an optimisation opportunity. If two different non-IOAM prefixes have the same metric at the origin IOAM router (as intf A:
2001:db8:0:1234::/64, metric 10, and intf D: fd04:3f29:de90::4321/128, metric 10 do above on IOAM Router A), then a per-router IOAM /128 could be used instead, suppressing the generation of 1:1 per-prefix IOAM /128s when origin prefix metrics are the same.

Regards,
Mark.

> Many thanks, Frank
>
> -----Original Message-----
> From: Mark Smith <[email-masked]>
> Sent: Donnerstag, 8. November 2018 12:40
> To: Frank Brockners (fbrockne) <[email-masked]>
> Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; 
> [email-masked]
> Subject: Re: v6 option types for IOAM data fields
>
> Hi Frank,
>
> On Thu, 8 Nov 2018 at 19:36, Frank Brockners (fbrockne) <[email-masked]> wrote:
> >
> > Hi Mark, Mike,
> >
> >
> >
> > thanks for summarizing the concerns around leaked packets – and outlining an approach to mitigate those. There are a couple of other challenges we face. Unfortunately we did not get to a discussion in the 6man WG meeting this time due to the limited time we had. The meeting slides https://datatracker.ietf.org/meeting/103/materials/slides-103-6man-in-situ-oam-ipv6-options-00 provide a summary of concerns.
> >
> >
> >
> > In a nutshell:
> >
> > 1.       Potential HbH ext header and IOAM option insertion and removal by transit nodes (i.e. “en route”) in a restricted administrative domain.
> >
> > a)       How do you deal with PMTU? – Packet size changes might exceed PMTU.
> >
> > b)      Misleading ICMP errors confusing the source
> >
> > c)       Possible leaks that affect the forwarding behavior and state of network elements outside the domain.
> >
>
> Yes, all of the above, good summary.
>
> An additional one is that if you were are receiver of this outside of the domain because it as leaked, at the destination, after the packet as traversed a number of ASes over the Internet, there is nothing to identify the device/AS that inserted the EH. Very time consuming to troubleshoot who inserted the EH but didn't remove it (you have to walk the AS path, contacting each operator, asking them to verify their config, get packet captures etc.), and all the while you may have a number of angry customers/end-users.
>
> > 2.       Incremental Trace IOAM HbH Option – which is to support a hardware-friendly implementation: Changes Option Data Len en-route.
> >
> > a)       Dealing with PMTU– Packet size changes can exceed PMTU; see above
> >
> > 3.       Clarify applicability statement and scope
> >
> >
> >
> > While 3. is something that is in the works, there are no easy answers to 1. and 2. Considerations:
> >
> >
> >
> > On 1. Supporting HbH ext header and option insertion/removal in 
> > transit. A couple of solution approaches…
> >
> > 1.       Fix PMTU and offset for packet size change in PMTU discovery - https://tools.ietf.org/html/draft-troan-6man-pmtu-solution-space-00
> > Hope that we can make progress towards a solution tomorrow…
> >
> > 2.       IP-in-IP with IOAM extension header in the *inner* packet.
> > New IPv6 packet is created with encapsulating node as source and the 
> > original destination as the destination (this is slightly different than what you, Mark, suggest – because we’d like to keep the forwarding path the same. Tunneling with ULA would not get us there; the slides above have a diagram that depicts the potential encap):
> >
> > a)       Payload of this packet is the original IPv6 packet along with an extension header inserted inside.
> >
> > b)      The original packet is restored by removing the outer IPv6 header and the inner extension header by a node at the domain boundary.
> >
> > c)       Caveats:
> >
> >                                                                i.      Modified packet may still leak – but will only confuse the destination node. The dest node will likely send an ICMP back to the encap node, which gets the encap node an understanding that something is going wrong.
> >
>
> This is certainly better, because the source address identifies the device that inserted the EH of the packet that leaked to where ever it leaked to.
>
> What the source address should be is an interesting question.
>
> If it is a global/public source address, then anybody who receives the packet leaked with the EH can identify who inserted it and therefore who didn't remove it.
>
> OTOH, if it is a private/local ULA Source Address, then BCP38 source address filters would cause the invalid leaked packet to be dropped sooner, rather than being forwarded onto the final destination. Total packet loss of OAM encapsulated packets would make the failure obvious to the OAM domain operator. Slight drawback is BCP38 filters haven't been universally implemented across the Internet.
>
> I think I like the latter better because commonly it would localise the symptoms of the failure to remove the EH to the domain that inserted it, rather than having to have an external party get involved in troubleshooting.
>
>
>
> >                                                              ii.      ECMP computation needs to be reworked.
> >
> >                                                            iii.      Complex/costly implementation in HW & SW – IOAM-capable nodes need to hunt for the IOAM data fields in the inner packet.
> >
>
> Yes, the implementation complexity of insertion is another concern - encapsulation is the most common and simplest cross layer forwarding function performed and there's decades of history in optimising it.
>
> The root cause of both the possible leak and the risk that the inserted EH will reach the final packet destination is that the Dest.
> Addr. of the packet with the inserted EH is both valid outside of the domain, and valid all the way across the Internet to the final and actual original packet destination. That creates a "default permit" if a packet leaks with the inserted EH rather than a far better "default deny" in this failure situation.
>
> In this OAM case, it seems to me the ideal and the requirements would be:
>
> - have the packet with the added OAM information follow the same path 
> within the domain that the same packet without the OAM information 
> would follow within the domain
>
> - have the Dest. Addr. of the packet with the added OAM information only be valid within the OAM domain i.e. an "interior DA", so that if through failure the packet leaks, it isn't a valid DA on the Internet and will be dropped by, if nothing else earlier, an Internet router with a default free route table.
>
> - encapsulate rather than insert, because encapsulation would be simpler, and is the traditional way of adding information to PDUs across many decades and many protocols (I can only think of VLAN ID insertion as the only existing counter example, and according to Radia Perlman in her book, that is only because at the time they weren't sure if all NICs commonly available could successfully encapsulate a full sized ethernet frame inside another one).
>
> In other words, preserve the original IPv6 packet, add an OAM header in front of it, then encapsulate it in another header to forward within and across the OAM domain.
>
> Bog standard IGP/LDP MPLS could achieve those requirements, with the OAM header inserted after the final MPLS tag and before the IPv6 header. The MPLS/OAM/IPv6 packet follows an LSP across the domain that matches the path that the IPv6 packet without MPLS would follow across the domain. In that scenario, the MPLS LSP across the domain is the "interior DA" that isn't valid outside of the OAM domain, because MPLS wouldn't (normally) be valid outside of the OAM domain (i.e. OAM domain is MPLS domain 1:1). If MPLS fails to be removed at the OAM domain egress edge, the packet is dropped immediately, so a fail safe, localised to the OAM domain.
>
> For an IPv6-in-IPv6 scenario:
>
> - for each OAM domain interior IPv6 routes/prefixes used by hosts (i.e. hosts have addresses from these prefixes), automatically create a matching interior OAM route/prefix (maybe just a /128), 1:1, with the interior OAM route/prefix originated by the same router(s) where the interior IPv6 routes/prefixes are created/originated. The interior OAM routes come from ULA space or perhaps a new designated and reserved private/local OAM address space for this purpose.
>
> When a end host originated packet ingresses the OAM domain edge, the DA address of the packet is looked up to find the 1:1 matching internal OAM route/prefix/address. This internal OAM route/prefix/address is used as the DA for the outer IPv6 header, outer header SA is the OAM address of the OAM ingress device, an OAM header follows, and then the original, inner IPv6 packet follows without modification.
>
> As the OAM route/prefix/address is originated at the same place as the OAM domain interior IPv6 route/prefix, the path the now IPv6-in-IPv6 packet follows through the OAM domain should be the same as that which the original, now inner packet would have followed with its DA. In a sense this OAM route/prefix/address is an internal routeable index or proxy for the inner packet destination prefix.
>
> - for exterior routes, IPv6-in-IPv6 tunnelling too between BGP border 
> routers per RFC1772, "A.2.3 Encapsulation", with the addition of the 
> OAM header between the new outer IPv6 header and the inner original
> IPv6 packet, and OAM local addresses for the tunnel end points. This avoids having to create 1:1 matching internal OAM route/prefix/address for each exterior route.
>
> (And perhaps a somewhat radical idea for reducing tunnelling overhead.
> All of this tunnelling stuff already exists for IPv4; and it is sunk 
> cost. Would there be much harm in using IPv4 as the interior outer 
> tunnelling protocol for this IPv6 traffic with the added OAM 
> information? I'd much rather keep running an IPv4 OAM island to carry
> IPv6 than any splitting apart of IPv6 packets to insert EHs in them
> ...)
>
> > 3.       No support of IOAM in transit network. Support only source initiated IOAM tracing, proof of transit..
> > Caveat: Limits usage of IOAM significantly.
> >
> >
> > On 2. Incremental Trace IOAM HbH Option
> >
> > 1.       Use PMTU to determine max possible Incremental Trace IOAM Option length
> >
> > 2.       Do not support Incremental Trace IOAM Option in IPv6.
> >
> >
>
> Could what I've suggested above solve those? I think the use of tunnelling/encapsulation supports a transit OAM scenario.
>
> Regards,
> Mark.
>
>
> >
> > While there isn’t an easy answer, IMHO we should still try to find a workable and implementable solution. I sometimes compare the situation with the road network: What we have today is a network of gravel roads (Internet with often poor ext header processing) and cars are built to cope with gravel roads. Now faster and comfy cars become available, though they only work on tared/concrete roads (specific administrative domains). If you try to drive these new cars on gravel roads they break and the wrecks jam the roads…, still we probably want to allow folks to buy and use the fast & comfy cars, because they will also push for better roads  as well (do proper ext header processing).
> >
> >
> >
> > Thoughts?
> >
> >
> >
> > Thanks, Frank
> >
> >
> >
> >
> >
> > From: Mark Smith <[email-masked]>
> > Sent: Dienstag, 30. Oktober 2018 05:26
> > To: Frank Brockners (fbrockne) <[email-masked]>
> > Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; 
> > [email-masked]
> > Subject: Re: v6 option types for IOAM data fields
> >
> >
> >
> >
> > Hi Frank,
> >
> >
> >
> > On Tue., 30 Oct. 2018, 05:36 Frank Brockners (fbrockne), <[email-masked]> wrote:
> >
> > Thanks Mike. On the scope of an IOAM deployment:
> > draft-ietf-ippm-ioam-data-04 clarifies in section 3 that IOAM is a domain focused feature, i.e. not expected to be deployed on the open Internet.
> >
> >
> >
> > That still violates RFC 8200, there are no exceptions for "closed" domains.
> >
> >
> >
> > From section 3:
> >
> >
> >
> > "Designers of
> >
> >    carrier protocols for IOAM must specify mechanisms to ensure that
> >
> >    IOAM data stays within an IOAM domain.  In addition, the operator 
> > of
> >
> >    such a domain is expected to put provisions in place to ensure 
> > that
> >
> >    IOAM data does not leak beyond the edge of an IOAM domain, e.g.
> > using
> >
> >    for example packet filtering methods."
> >
> >
> >
> > Neither the designers or the operators can ensure this IOAM information won't leak.
> >
> >
> >
> > Possible reasons it can leak:
> >
> >
> >
> > - operator configuration error - e.g. forgetting to configure the domain boundary (e.g. router with multiple domain exit interfaces, forgetting to add that option on one of them), or misconfiguring it.
> >
> >
> >
> > - partial device failure - the device still forwards packets, but 
> > ceases to remove this information due to a hardware fault that has 
> > appeared
> >
> >
> >
> > - vendor implementation bugs that fails to remove the information.
> >
> >
> >
> > Any or all of these can occur, and as it is on egress, the consequences may not be visible to the domain network operator, because network operators rarely inspect packets after they've left their network. Once a packet is sent onto somebody else, it is assumed to have been sent without faults.
> >
> >
> >
> > There are many instances of leaks at the boundary of what are 
> > supposed to be closed domains, such as packets containing RFC1918 
> > private addresses (in packet headers themselves, or in DNS packets - 
> > such a big problem that www.as112.net was created), and route leaks e.g.
> > https://bgpmon.net/how-the-internet-in-australia-went-down-under/
> >
> >
> >
> > As operators usually have only one device at the edge of the domain, that would be performing the domain boundary function, that device becomes a single point of failure. Enforcement is therefore quite fragile. A domain boundary enforced by two domain edge devices in-line would increase robustness, however I'd think that would be rarely deployed, because we haven't seen that commonly with IPv4 NAT.
> >
> >
> >
> > The only way for an operator to ensure these leaks don't and never occur is for the domain to literally not be physically connected to the Internet i.e. the domain is air gapped from the Internet. If there is a link between the domain and the Internet that can carry IP packets, then there is always a possibility that information can leak from the domain.
> >
> >
> >
> > So accepting that leaks can always occur, a far more robust option is to leave the original packet alone entirely, and encapsulate it in a domain local IPv6 header (i.e RFC2473, section 3.1) with domain local limited addresses (i.e. ULA addresses) and the additional EH.
> >
> >
> >
> > That still doesn't ensure that packet won't leak outside the domain, because those packets will follow a default route, however as the packet's addressing is invalid outside of the domain, that packet will be dropped once it reaches either an invalid Internet address filter, or a default free Internet router. The packet with the failed-to-be-removed outer ULA IPv6 header will never reach the destination address of the inner packet, because the DA of the inner packet is in the ULA IPv6 packet's payload.
> >
> >
> >
> > Failure of the mechanism would be much more obvious, localised to the domain implementing the mechanism (rather than being externalised to somebody else's domain/network), and absolute, because the failure mode is packets being entirely discarded.
> >
> >
> >
> > Regards,
> >
> > Mark.
> >
> >
> >
> >
> >
> >
> > Frank
> >
> > -----Original Message-----
> > From: C. M. Heard <[email-masked]>
> > Sent: Montag, 29. Oktober 2018 16:03
> > To: 6man <[email-masked]>; IPPM <[email-masked]>
> > Cc: Frank Brockners (fbrockne) <[email-masked]>
> > Subject: Re: v6 option types for IOAM data fields
> >
> > On Thu, 25 Oct 2018 15:06:57 +0000 Frank Brockners (fbrockne) wrote:
> > > Quick heads up: In the 6MAN meeting in BKK, we’ll review
> > > draft-ioametal-ippm-6man-ioam-ipv6-options-01 – which requests 2 
> > > option types from the DO/HbyH options sub-registry.
> > >
> > > While the bulk of the IOAM work is progressed in the IPPM WG, we’d 
> > > greatly appreciate your feedback on 
> > > draft-ioametal-ippm-6man-ioam-ipv6-options-01,
> > > which defines how IOAM data fields are carried using v6 extension headers.
> > > Cc’ing the IPPM WG as well, to keep everyone on the same page.
> >
> > I have two brief comments on this work.
> >
> > First, I see that the Incremental Tracing Option changes length in transit.
> > It is not appropriate for it to be carried in an IPv6 option intended for use on the open Internet, for exactly the same reason that insertion of extension headers by intermediate nodes is not allowed on the open Internet.
> >
> > Second, I see that two IPv6 option code points are requested, one with the "chg" flag set, the other with the "chg" flag clear. While there is no harm in this, it is not strictly necessary; the only real purpose of this flag is to determine whether the option data is or is not included in the Authentication Header Integrity Check Value computation.
> >
> > Mike Heard
> > --------------------------------------------------------------------
> > IETF IPv6 working group mailing list [email-masked] Administrative 
> > Requests: https://www.ietf.org/mailman/listinfo/ipv6
> > --------------------------------------------------------------------

---

From: Tom Herbert <[email-masked]>
Date: Wed, 13 Mar 2019 09:49:45 -0700

Hi Frank,

You might want to consider the interaction of IOAM with segment
routing. For instance, and IOAM destination option could appear before
the routing header to be processed at each segment. In this case, the
option can be modified by intermediate destinations.

Tom

> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 14 Mar 2019 08:54:57 +0000

-----Original Message-----
From: Tom Herbert <[email-masked]> 
Sent: Mittwoch, 13. März 2019 17:50
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Mark Smith <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: Re: v6 option types for IOAM data fields

Hi Frank,

You might want to consider the interaction of IOAM with segment routing. For instance, and IOAM destination option could appear before the routing header to be processed at each segment. In this case, the option can be modified by intermediate destinations.

Tom

Hi Tom,

this is an interesting aspect - potentially - pending the decision whether SR will have TLVs in the SRH or whether destination options are going to be used. Would it make sense to wait for a conclusion from the WG on this topic first? From an IOAM perspective, a destination option header carrying IOAM E2E options should only be interpreted by the IOAM encapsulating and IOAM decapsulating nodes and be left untouched by intermediate devices.

Thanks, Frank

> /128 prefixes, its probably useful to encode some unique part of the 
> global prefix in the corresponding /128.)
>
>
> So when a packet enters the IOAM domain at IOAM Router Z, with a 
> destination address that falls within 2001:db8:0:1234::/64 (IOAM 
> Router A's intf A: prefix), the outer tunnel header's DA would be the 
> corresponding fd04:3f29:de90::1234/128 (IOAM Router A's intf A: OAM
> /128 prefix). This post-tunnel packet should follow the same path 
> through the IOAM domain that a packet destined towards
> If another packet enters the IOAM domain at IOAM Router Z, with a 
> destination address that falls within 2001:db8:0:4567::/64 (IOAM 
> Router A intf C's second prefix), then the outer packet tunnel DA 
> would be the corresponding fd04:3f29:de90::4567/128 ((IOAM Router A 
> intf C's second and matching IOAM /128 prefix). As the metric for
> > > New IPv6 packet is created with encapsulating node as source and 
> > > the original destination as the destination (this is slightly different than what you, Mark, suggest – because we’d like to keep the forwarding path the same. Tunneling with ULA would not get us there; the slides above have a diagram that depicts the potential encap):
> > >
> > - have the packet with the added OAM information follow the same 
> > path within the domain that the same packet without the OAM 
> > information would follow within the domain
> >
> > - for exterior routes, IPv6-in-IPv6 tunnelling too between BGP 
> > border routers per RFC1772, "A.2.3 Encapsulation", with the addition 
> > of the OAM header between the new outer IPv6 header and the inner 
> > original
> > >    IOAM data stays within an IOAM domain.  In addition, the 
> > > operator of
> > >
> > > private addresses (in packet headers themselves, or in DNS packets 
> > > - such a big problem that www.as112.net was created), and route leaks e.g.
> > > > While the bulk of the IOAM work is progressed in the IPPM WG, 
> > > > we’d greatly appreciate your feedback on 
> > > ------------------------------------------------------------------
> > > -- IETF IPv6 working group mailing list [email-masked] 
> > > Administrative

---

From: Tom Herbert <[email-masked]>
Date: Thu, 14 Mar 2019 07:27:57 -0700

Hi Frank,

I was more thinking someone might want to use the Pre-allocated Trace
Option in conjunction with the routing header. This allows the user to
get IOAM from precisely the intermediate destinations of interest.

I don't think there's any need to wait on Segement Routing. IOAM needs
to be general to work with SR, other routing headers, or without
routing header. Putting it in DO or HBH options is sufficient for all
those cases.

Tom

> > > > --

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 14 Mar 2019 18:49:05 +0000

Hi Tom,

Thanks. Got it. We can add a paragraph with considerations in case IOAM is deployed with a routing header present. 
Will send out some proposed text once I'm back in the office.

Thanks, Frank

-----Original Message-----
From: Tom Herbert <[email-masked]> 
Sent: Donnerstag, 14. März 2019 15:28
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Mark Smith <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: Re: v6 option types for IOAM data fields

On Thu, Mar 14, 2019 at 1:55 AM Frank Brockners (fbrockne) <[email-masked]> wrote:
> Cc: Mark Smith <[email-masked]>; C. M. Heard 
> <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Hi Frank,

I was more thinking someone might want to use the Pre-allocated Trace Option in conjunction with the routing header. This allows the user to get IOAM from precisely the intermediate destinations of interest.

I don't think there's any need to wait on Segement Routing. IOAM needs to be general to work with SR, other routing headers, or without routing header. Putting it in DO or HBH options is sufficient for all those cases.

Tom

> > Router A's intf A: prefix), the outer tunnel header's DA would be 
> > the corresponding fd04:3f29:de90::1234/128 (IOAM Router A's intf A: 
> > OAM
> > > border routers per RFC1772, "A.2.3 Encapsulation", with the 
> > > addition of the OAM header between the new outer IPv6 header and 
> > > the inner original
> > > All of this tunnelling stuff already exists for IPv4; and it is 
> > > sunk cost. Would there be much harm in using IPv4 as the interior 
> > > outer tunnelling protocol for this IPv6 traffic with the added OAM 
> > > >    such a domain is expected to put provisions in place to 
> > > > ensure that
> > > >
> > > > - partial device failure - the device still forwards packets, 
> > > > but ceases to remove this information due to a hardware fault 
> > > > that has appeared
> > > >
> > > >
> > > >
> > > > supposed to be closed domains, such as packets containing 
> > > > RFC1918 private addresses (in packet headers themselves, or in 
> > > > DNS packets
> > > > > draft-ioametal-ippm-6man-ioam-ipv6-options-01 – which requests 
> > > > > 2 option types from the DO/HbyH options sub-registry.
> > > > >

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Mon, 18 Mar 2019 09:01:21 +0000

Hi Tom,

When I started the write up, it made me think about the possible options again.
Assumption: We have a routing header present in the v6 packet.
Question: What v6 headers should I use for carrying IOAM options, and where should they be located?

IOAM E2E options are by definition only processed by the IOAM encapsulating/decapsulating node. Thus IOAM E2E options should be carried using Destination-Option "note 2" style (RFC 8200, section 4.1), i.e. the destination option header for IOAM E2E should always appear after the routing header, so that only the ultimate destination processes it.

For IOAM Trace Option headers we have 2 options:
* Option 1: The operator desires that the trace options are *only* processed/updated by hops specified in the routing header. In that case, we’d use Destination Option “note 2” style, i.e. the DestOption header with IOAM trace options would appear before the Routing Option header.
* Option 2: There are no constraints for IOAM tracing, i.e. nodes configured to update IOAM tracing info should be able to do so. In that case, we’d carry IOAM trace options in a HbH header.

Which leads to the question: Given that Option1 is really a subset of Option2, we could simply go with Option 2 and avoid having 2 different uses of the destination option header in the packet. It would make things simpler from an implementation perspective, at the expense that an operator who desires to only trace the hops spelled out in the routing header would need to explicitly configure all nodes in the network correspondingly. That said, Option 1 might be attractive for networks that use the routing header, desire to trace the hops the routing header describes, but try to avoid the use of the HbyH header and thus automatically ignore hops in the "underlay".

What do you - and others - think of this trade-off between simplicity and potential benefits? Should we document both options (and also consider 2 different IOAM code points for destination options)?

Thanks, Frank

-----Original Message-----
From: ippm <[email-masked]> On Behalf Of Frank Brockners (fbrockne)
Sent: Donnerstag, 14. März 2019 19:49
To: Tom Herbert <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; Mark Smith <[email-masked]>; [email-masked]
Subject: Re: [ippm] v6 option types for IOAM data fields

Hi Tom,

Thanks, Frank

-----Original Message-----
From: Tom Herbert <[email-masked]>
Sent: Donnerstag, 14. März 2019 15:28
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Mark Smith <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: Re: v6 option types for IOAM data fields

Hi Frank,

Tom

> > > > Second, I see that two IPv6 option code points are requested, one with the "chg" flag set, the other with the "chg" flag clear. While there is no harm in this, it is not strictly necessary; t…
_______________________________________________
ippm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ippm

---

From: Mark Smith <[email-masked]>
Date: Sun, 24 Mar 2019 12:59:56 +1100

Hi Frank,

Apologies not to get back to you sooner, I missed this email (and now
have a filter to highlight emails directly sent to me). Also apologies
for sending feedback so close to IETF104.

> Appreciate your thoughts - especially from 6man to provide for proper guidance to IPPM.

<snip>

Thanks very much to you and the other authors for writing up that this
Internet Draft.

Regarding

"3.  Considerations for IOAM deployment in IPv6 networks"

that's an excellent list and set of descriptions of what I think are
the requirements for when information is added to an existing IPv6
packet as it transits a network domain that wishes to add and then
remove local domain supplementary information.

I think those requirements also apply to the addition of Segment
Routing information to existing packets as they transit an SR domain,
so one thought could be to extract them out and publish them as a more
general Internet Draft, titled something like "Considerations For
Adding Supplemental Information to IPv6 Packets Transiting a Network
Sub-Domain". (Actually, reading more, I think most of the ID is not
really that specific to IOAM, so most of the ID could probably be
published as a more generic document on considerations for adding
supplemental information to transit IP packets).

Regarding,

"5.1.1.  IPv6-in-IPv6 encapsulation"

where the outer encapsulation header destination IPv6 address is set
to the same as the inner, original IPv6 destination address, for
packets transiting the IOAM domain, I think that method would be quite
vulnerable to C3:

"C3 Packets with IOAM data or associated ICMP errors, should not
      arrive at destinations which have no knowledge of IOAM.  Consider
      using IOAM in transit devices; misleading ICMP errors due to
      addition and/or presence of OAM data in the packet can confuse a
      source of the packet that did not insert the OAM information."

As the outer header IPv6 address is exterior to the IOAM domain as it
is also the inner packet's DA, it means that standard and default IPv6
destination address based forwarding would forward the packet as is
(i.e. encapsulated with IOAM information) external to the domain,
without removing the outer IPv6 encapsulation and the IOAM
information. In other words, IOAM information would leak if standard
IPv6 forwarding was used, which of course is enabled by default for
IPv6 routers.

To prevent that, the egress IOAM device would have to use
non-standard, non-default IPv6 forwarding, which means deeper
inspection of each and every egress packet regardless of its
destination address, to see if it is an encapsulated IOAM packet, and
to then remove the outer IPv6 header and IOAM information if it
exists, before then standard IPv6 forwarding the original inner
packet.

The advantage of using a separate and IOAM domain local and interior
address space as in "5.1.2.  IP-in-IPv6 encapsulation with ULA", is
that the outer packet ULA destination address is distinguishing which
packets need IOAM encapsulation processing from those that don't (and
of course, as the ULA address space is local to and unique to the IOAM
domain, if they do leak externally, they have no valid destination to
reach).

As those encapsulated IOAM packets are distinguished using the (outer)
Destination Address, the selector for standard IPv6 forwarding, there
is now no need for any special, new and more complex packet handling
by the IOAM domain egress device. Using standard IPv6 forwarding
avoids the complexity and implementation that would then be imposed on
every router implementation that could be used as an egress IOAM
domain device.

"2. To simplify debugging in case of leaked IOAM data fields in

       packets, consider a new IOAM E2E destination option to identify
       the Source IOAM domain (AS, v6 prefix).  Insert this option into
       the IOAM destination options EH attached to the outer IPv6
       header.  This additional information would allow for easy
       identification of an AS operator that is the source of packets
       with leaked IOAM information."

is suggesting a solution to the leaked IOAM issue. However I think it
is always better to avoid a problem using an alternative method if
possible, rather than to continue to let it the problem exist and then
mitigate or eliminate its effects with even further mechanisms. It's
more complexity to mitigate or eliminate, and that's more things that
can break.

Regarding,

"5.1.2.  IP-in-IPv6 encapsulation with ULA"

Some suggested text -

"Addressing and routing in the ION are to be configured so
   that the IP-in-IPv6 encapsulated packets follow the same path as the
   original, non-encapsulated packet would have taken."

add something like

"This would create an internal IPv6 forwarding topology using the IOAM
domain's interior ULA address space that is parallel with the
forwarding topology that exists with the non-IOAM address space (the
topology and address space that would be followed by packets that do
not have supplemental IOAM information) . Establishment and
maintenance of the parallel IOAM ULA forwarding topology could be
automated in the future, similar to how LDP [RFC5036] is used in MPLS
to establish and maintain an LSP forwarding topology that is parallel
to the network's IGP forwarding topology."

"Assuming that the operational guidelines specified in Section 4 of
   [RFC4193] are properly followed, the probability of leaks in this
   approach will be almost close to zero."

add something like

"If the packets do leak through IOAM egress device misconfiguration or
partial IOAM egress device failure, the packets' ULA destination
address is invalid outside of the IOAM domain. There is no exterior
destination to be reached, and the packets will be dropped when they
encounter either a router external to the IOAM domain that has a
packet filter that drops packets with ULA destinations, or a router
that does not have a default route."

Regards,
Mark.

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Sun, 24 Mar 2019 08:18:25 +0000

Frank,

Thanks for describing the issue and solution options so clearly. I would vote for option 2.

Please see inline,

Correction - “Note 3” style, not “note 2”. Right ?

Correction - “note 1” style, not note 2. Right?

This option could give us the flexibility to define IOAM encapsulating/decapsulating nodes as either network domain ingress/egress nodes (just 2 nodes processing IOAM), or ingress/egress plus 3 transit nodes, or all of the nodes dictated by the list (of nodes) included in routing header.

The downside of this approach is that it requires nodes to be spelled out explicitly I.e. path engineering at the ingress. In many cases, this could be unnecessary if the intent is to only get IOAM information, and let the dynamic routing dictate the path. This downside could lead to operational headaches.

Ditto.

While you mention Tracing, other IOAM options (POT, E2E, TS) should work here as well. No?

Here, the advantage is that no path engineering is mandated to get the IOAM data from fewer or all nodes.

The slight downside is that HbH header would obviously be to enable the network (nodes) for IOAM processing. Given that, it would be (more or less) one time thing, it should be deemed very much acceptable and manageable.

Hence, I would vote for option 2.
Keeps both implementation and deployment simpler.

Cheers,
Rajiv

On Mar 18, 2019, at 10:02 AM, Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Tom,

Thanks, Frank

-----Original Message-----
From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Frank Brockners (fbrockne)
Sent: Donnerstag, 14. März 2019 19:49
To: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>; Mark Smith <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] v6 option types for IOAM data fields

Hi Tom,

Thanks, Frank

-----Original Message-----
From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Donnerstag, 14. März 2019 15:28
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: Mark Smith <[email-masked]<mailto:[email-masked]>>; C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: v6 option types for IOAM data fields

On Thu, Mar 14, 2019 at 1:55 AM Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:

-----Original Message-----
From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 13. März 2019 17:50
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: Mark Smith <[email-masked]<mailto:[email-masked]>>; C. M. Heard
<[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: v6 option types for IOAM data fields

On Wed, Mar 13, 2019 at 9:24 AM Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Frank,

Tom

Hi Tom,

Hi Frank,

Tom

Thanks, Frank

Thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 21. November 2018 02:28
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;
[email-masked]<mailto:[email-masked]>
Subject: Re: v6 option types for IOAM data fields

Hi Frank,

On Mon, 12 Nov 2018 at 19:40, Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Mark,

Close, but not quite.

intf C: 2001:db8:0:4567::/64, metric 100 intf C:
2001:db8:0:abcd::/64, metric 150

intf C: fd04:3f29:de90::4567/128, metric 100 intf C:
fd04:3f29:de90::abcd/128, metric 150

(There may be better schemes to generate the matching global
/64:IOAM
/128 prefixes, its probably useful to encode some unique part of the
global prefix in the corresponding /128.)

Regards,
Mark.

Many thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Donnerstag, 8. November 2018 12:40
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;
[email-masked]<mailto:[email-masked]>
Subject: Re: v6 option types for IOAM data fields

Hi Frank,

On Thu, 8 Nov 2018 at 19:36, Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Mark, Mike,

In a nutshell:

c)       Caveats:

(And perhaps a somewhat radical idea for reducing tunnelling overhead.
All of this tunnelling stuff already exists for IPv4; and it is
sunk cost. Would there be much harm in using IPv4 as the interior
outer tunnelling protocol for this IPv6 traffic with the added OAM
information? I'd much rather keep running an IPv4 OAM island to
carry
IPv6 than any splitting apart of IPv6 packets to insert EHs in
them
...)

Regards,
Mark.

Thoughts?

Thanks, Frank

From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Dienstag, 30. Oktober 2018 05:26
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;
[email-masked]<mailto:[email-masked]>
Subject: Re: v6 option types for IOAM data fields

Hi Frank,

On Tue., 30 Oct. 2018, 05:36 Frank Brockners (fbrockne), <[email-masked]<mailto:[email-masked]>> wrote:

From section 3:

"Designers of

  carrier protocols for IOAM must specify mechanisms to ensure
that

There are many instances of leaks at the boundary of what are
supposed to be closed domains, such as packets containing
RFC1918 private addresses (in packet headers themselves, or in
DNS packets
- such a big problem that www.as112.net<http://www.as112.net> was created), and route leaks e.g.
https://bgpmon.net/how-the-internet-in-australia-went-down-under
/

Regards,

Mark.

Frank

-----Original Message-----
From: C. M. Heard <[email-masked]<mailto:[email-masked]>>
Sent: Montag, 29. Oktober 2018 16:03
To: 6man <[email-masked]<mailto:[email-masked]>>; IPPM <[email-masked]<mailto:[email-masked]>>
Cc: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Subject: Re: v6 option types for IOAM data fields

Mike Heard
----------------------------------------------------------------
--
-- IETF IPv6 working group mailing list [email-masked]<mailto:[email-masked]>
Administrative
Requests: https://www.ietf.org/mailman/listinfo/ipv6
----------------------------------------------------------------
--
--
--------------------------------------------------------------------
IETF IPv6 working group mailing list [email-masked]<mailto:[email-masked]> Administrative
Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------
_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm
--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]<mailto:[email-masked]>
Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Mon, 25 Mar 2019 05:06:28 +0000

I also think that there is a value in dividing this draft in 2 separate drafts – one generic, and another focused on IOAM specifics.

Also, it might be useful to distinguish the deployment considerations that relate to Inter-AS and/or Inter-Provider IOAM usage. For ex, C3, C4. An IOAM domain An AS can have more than one IOAM domains after all. In such usages, IOAM information would need to be exchanged outside the domain & AS.

BTW, wrt C6, which section of rfc8200 prescribes using encapsulation? IMO, using encap breaks C1. Of course, using option insertion into the original packets breaks C6. In light of that, it would be useful to add the 5.1.4 that lists non-encapsulation option as well.

--
Cheers,
Rajiv

From: ipv6 <[email-masked]> on behalf of Mark Smith <[email-masked]>
Date: Saturday, March 23, 2019 at 10:01 PM
To: "Frank Brockners (fbrockne)" <[email-masked]>
Cc: "C. M. Heard" <[email-masked]>, 6man WG <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Hi Frank,

On Thu, 14 Mar 2019 at 03:23, Frank Brockners (fbrockne)
<[email-masked]<mailto:[email-masked]>> wrote:

<snip>

Regarding

Regarding,

Regarding,

Some suggested text -

add something like

add something like

Regards,
Mark.

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Mon, 25 Mar 2019 12:14:01 +0000

Rajiv,

Thanks – and you’re 100% correct on me using the wrong “note” references. The one you state below are the correct ones.

Frank

From: Rajiv Asati (rajiva)
Sent: Sonntag, 24. März 2019 09:18
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: Tom Herbert <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: Re: v6 option types for IOAM data fields

Frank,

Please see inline,

Ditto.

Cheers,
Rajiv

Thanks, Frank

-----Original Message-----
From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Frank Brockners (fbrockne)
Sent: Donnerstag, 14. März 2019 19:49
To: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>; Mark Smith <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] v6 option types for IOAM data fields

Hi Tom,

Thanks, Frank

-----Original Message-----
From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 13. März 2019 17:50
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: Mark Smith <[email-masked]<mailto:[email-masked]>>; C. M. Heard

Hi Frank,

Tom

Hi Tom,

Hi Frank,

Tom

Thanks, Frank

Thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 21. November 2018 02:28
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;

Hi Frank,

Hi Mark,

Close, but not quite.

Regards,
Mark.

Many thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Donnerstag, 8. November 2018 12:40
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;

Hi Frank,

Hi Mark, Mike,

In a nutshell:

c)       Caveats:

Regards,
Mark.

Thoughts?

Thanks, Frank

Hi Frank,

From section 3:

"Designers of

Regards,

Mark.

Frank

-----Original Message-----
From: C. M. Heard <[email-masked]<mailto:[email-masked]>>
Sent: Montag, 29. Oktober 2018 16:03
To: 6man <[email-masked]<mailto:[email-masked]>>; IPPM <[email-masked]<mailto:[email-masked]>>
Cc: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Subject: Re: v6 option types for IOAM data fields

Mike Heard

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Tue, 26 Mar 2019 07:19:04 +0000

Hi Mark,

Thanks much and now it is me to say sorry for the delay in responding. We'll add the additional paragraphs you suggest to the next revision.
Based on the outcome of the WG discussion on Friday we can decide whether folks are interested in broadening the scope of the doc to become more generic.

Thanks again, Frank

-----Original Message-----
From: Mark Smith <[email-masked]> 
Sent: Sonntag, 24. März 2019 03:00
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Hi Frank,

Apologies not to get back to you sooner, I missed this email (and now have a filter to highlight emails directly sent to me). Also apologies for sending feedback so close to IETF104.

<snip>

Thanks very much to you and the other authors for writing up that this Internet Draft.

Regarding

that's an excellent list and set of descriptions of what I think are the requirements for when information is added to an existing IPv6 packet as it transits a network domain that wishes to add and then remove local domain supplementary information.

I think those requirements also apply to the addition of Segment Routing information to existing packets as they transit an SR domain, so one thought could be to extract them out and publish them as a more general Internet Draft, titled something like "Considerations For Adding Supplemental Information to IPv6 Packets Transiting a Network Sub-Domain". (Actually, reading more, I think most of the ID is not really that specific to IOAM, so most of the ID could probably be published as a more generic document on considerations for adding supplemental information to transit IP packets).

Regarding,

where the outer encapsulation header destination IPv6 address is set to the same as the inner, original IPv6 destination address, for packets transiting the IOAM domain, I think that method would be quite vulnerable to C3:

As the outer header IPv6 address is exterior to the IOAM domain as it is also the inner packet's DA, it means that standard and default IPv6 destination address based forwarding would forward the packet as is (i.e. encapsulated with IOAM information) external to the domain, without removing the outer IPv6 encapsulation and the IOAM information. In other words, IOAM information would leak if standard
IPv6 forwarding was used, which of course is enabled by default for
IPv6 routers.

To prevent that, the egress IOAM device would have to use non-standard, non-default IPv6 forwarding, which means deeper inspection of each and every egress packet regardless of its destination address, to see if it is an encapsulated IOAM packet, and to then remove the outer IPv6 header and IOAM information if it exists, before then standard IPv6 forwarding the original inner packet.

The advantage of using a separate and IOAM domain local and interior address space as in "5.1.2.  IP-in-IPv6 encapsulation with ULA", is that the outer packet ULA destination address is distinguishing which packets need IOAM encapsulation processing from those that don't (and of course, as the ULA address space is local to and unique to the IOAM domain, if they do leak externally, they have no valid destination to reach).

As those encapsulated IOAM packets are distinguished using the (outer) Destination Address, the selector for standard IPv6 forwarding, there is now no need for any special, new and more complex packet handling by the IOAM domain egress device. Using standard IPv6 forwarding avoids the complexity and implementation that would then be imposed on every router implementation that could be used as an egress IOAM domain device.

is suggesting a solution to the leaked IOAM issue. However I think it is always better to avoid a problem using an alternative method if possible, rather than to continue to let it the problem exist and then mitigate or eliminate its effects with even further mechanisms. It's more complexity to mitigate or eliminate, and that's more things that can break.

Regarding,

Some suggested text -

add something like

"This would create an internal IPv6 forwarding topology using the IOAM domain's interior ULA address space that is parallel with the forwarding topology that exists with the non-IOAM address space (the topology and address space that would be followed by packets that do not have supplemental IOAM information) . Establishment and maintenance of the parallel IOAM ULA forwarding topology could be automated in the future, similar to how LDP [RFC5036] is used in MPLS to establish and maintain an LSP forwarding topology that is parallel to the network's IGP forwarding topology."

add something like

"If the packets do leak through IOAM egress device misconfiguration or partial IOAM egress device failure, the packets' ULA destination address is invalid outside of the IOAM domain. There is no exterior destination to be reached, and the packets will be dropped when they encounter either a router external to the IOAM domain that has a packet filter that drops packets with ULA destinations, or a router that does not have a default route."

Regards,
Mark.

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Wed, 27 Mar 2019 10:43:32 +0000

It is very important to specify the default behavior () with IOAM option processing (by an IPv6 node) to meaningfully progress on this draft. This would ensure that the nodes act I.e. drop or receive/transmit, according to what is in the header and what nodes are configured for.

I suggest the below text along that line -

//
Section 3

IOAM data can be encoded inside “option data” field of two extension headers in IPv6 packets - either Hop-by-Hop Options header or Destination options header. The selection of a particular extension header type depends on IOAM usage, as described in section 4 of [I-D.ietf-ippm-ioam-data].  The codepoints for IPv6 Destination Options Type and Hop-by-Hop Options Type are to be assigned by IANA.
In order for IOAM to work in IPv6 networks, IOAM MUST be explicitly enabled per interface on every node within the IOAM domain. This means that if a particular interface is not enabled (I.e. configured) for IOAM, then the entire extension header carrying IOAM data MUST be removed from that IPv6 packet before accepting or forwarding on that interface. This is the default behavior and is independent of whether the Hop-by-Hop options or Destination options are used to encode the IOAM data. This ensures that IOAM data does not unintentionally get forwarded outside the IOAM domain.
An IPv6 packet carrying IOAM data in an Extension header can have other extension headers, compliant with RFC8200.
//

Note - Of course, dropping the entire packet (instead of just the extension header) is an option, though it may be somewhat undesired for obvious reasons.

Cheers,
Rajiv

On Mar 14, 2019, at 7:50 PM, Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Tom,

Thanks, Frank

-----Original Message-----
From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 13. März 2019 17:50
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: Mark Smith <[email-masked]<mailto:[email-masked]>>; C. M. Heard

Hi Frank,

Tom

Hi Tom,

Hi Frank,

Tom

Thanks, Frank

Thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 21. November 2018 02:28
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;

Hi Frank,

Hi Mark,

Close, but not quite.

Regards,
Mark.

Many thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Donnerstag, 8. November 2018 12:40
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;

Hi Frank,

Hi Mark, Mike,

In a nutshell:

c)       Caveats:

Regards,
Mark.

Thoughts?

Thanks, Frank

Hi Frank,

From section 3:

"Designers of

Regards,

Mark.

Frank

-----Original Message-----
From: C. M. Heard <[email-masked]<mailto:[email-masked]>>
Sent: Montag, 29. Oktober 2018 16:03
To: 6man <[email-masked]<mailto:[email-masked]>>; IPPM <[email-masked]<mailto:[email-masked]>>
Cc: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Subject: Re: v6 option types for IOAM data fields

Mike Heard

---

From: Tom Herbert <[email-masked]>
Date: Wed, 27 Mar 2019 13:01:08 +0100

> Note - Of course, dropping the entire packet (instead of just the extension header) is an option, though it may be somewhat undesired for obvious reasons.

Hi Rajiv,

RFC8200 prohibits insertion/deletion of extension headers from packets
in flight (there has been a lot of discussion about this on 6man). The
alternative to achieve the same effect is to encapsulate packets and
put extension headers in the outer IP header that are needed to
traverse the domain. At the domain egress, decapsulation is done which
removes the extension headers before the packet is forwarded outside
of the domain. The nice part about the encapsulation method is that
encapsulated packets are targeted to destinations within the domain
which then automatically contains the extension headers within the
domain.

Tom

>
> Cheers,
> Rajiv
>
>
> On Mar 14, 2019, at 7:50 PM, Frank Brockners (fbrockne) <[email-masked]> wrote:
>
> Hi Tom,
>

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 28 Mar 2019 09:18:11 +0000

Hi Rajiv,

Thanks – this is a good suggestion and we can add it to the next rev of the draft. IMHO we can even go all the way to the stronger statement that you suggest to help mitigate concerns, i.e. specify the default behavior as “drop all packets with IOAM unless an interface is explicitly configured to process IOAM packets”.

Frank

Sent: Mittwoch, 27. März 2019 11:44

//
Section 3

Cheers,
Rajiv

Thanks, Frank

-----Original Message-----
From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 13. März 2019 17:50
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: Mark Smith <[email-masked]<mailto:[email-masked]>>; C. M. Heard

Hi Frank,

Tom

Hi Tom,

Hi Frank,

Tom

Thanks, Frank

Thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 21. November 2018 02:28
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;

Hi Frank,

Hi Mark,

Close, but not quite.

Regards,
Mark.

Many thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Donnerstag, 8. November 2018 12:40
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;

Hi Frank,

Hi Mark, Mike,

In a nutshell:

c)       Caveats:

Regards,
Mark.

Thoughts?

Thanks, Frank

Hi Frank,

From section 3:

"Designers of

Regards,

Mark.

Frank

-----Original Message-----
From: C. M. Heard <[email-masked]<mailto:[email-masked]>>
Sent: Montag, 29. Oktober 2018 16:03
To: 6man <[email-masked]<mailto:[email-masked]>>; IPPM <[email-masked]<mailto:[email-masked]>>
Cc: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Subject: Re: v6 option types for IOAM data fields

Mike Heard

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 28 Mar 2019 19:47:48 +0000

FYI. A new draft building on Rajiv’s suggestions just got posted:
https://tools.ietf.org/html/draft-ioametal-ippm-6man-ioam-ipv6-options-02
adds text which states that an interface must be explicitly configured to carry extension headers with IOAM – default would be to drop packets with extension headers carrying IOAM.

Frank

From: Frank Brockners (fbrockne)
Sent: Donnerstag, 28. März 2019 10:18
To: Rajiv Asati (rajiva) <[email-masked]>
Cc: Tom Herbert <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: RE: v6 option types for IOAM data fields

Hi Rajiv,

Frank

Cc: Tom Herbert <[email-masked]<mailto:[email-masked]>>; C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>

//
Section 3

Cheers,
Rajiv

Thanks, Frank

-----Original Message-----
From: Tom Herbert <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 13. März 2019 17:50
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: Mark Smith <[email-masked]<mailto:[email-masked]>>; C. M. Heard

Hi Frank,

Tom

Hi Tom,

Hi Frank,

Tom

Thanks, Frank

Thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Mittwoch, 21. November 2018 02:28
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;

Hi Frank,

Hi Mark,

Close, but not quite.

Regards,
Mark.

Many thanks, Frank

-----Original Message-----
From: Mark Smith <[email-masked]<mailto:[email-masked]>>
Sent: Donnerstag, 8. November 2018 12:40
To: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Cc: C. M. Heard <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>;

Hi Frank,

Hi Mark, Mike,

In a nutshell:

c)       Caveats:

Regards,
Mark.

Thoughts?

Thanks, Frank

Hi Frank,

From section 3:

"Designers of

Regards,

Mark.

Frank

-----Original Message-----
From: C. M. Heard <[email-masked]<mailto:[email-masked]>>
Sent: Montag, 29. Oktober 2018 16:03
To: 6man <[email-masked]<mailto:[email-masked]>>; IPPM <[email-masked]<mailto:[email-masked]>>
Cc: Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Subject: Re: v6 option types for IOAM data fields

Mike Heard

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 28 Mar 2019 19:50:06 +0000

-----Original Message-----
From: Tom Herbert <[email-masked]> 
Sent: Mittwoch, 27. März 2019 13:01
To: Rajiv Asati (rajiva) <[email-masked]>
Cc: Frank Brockners (fbrockne) <[email-masked]>; C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: Re: v6 option types for IOAM data fields

Hi Rajiv,

RFC8200 prohibits insertion/deletion of extension headers from packets in flight (there has been a lot of discussion about this on 6man). The alternative to achieve the same effect is to encapsulate packets and put extension headers in the outer IP header that are needed to traverse the domain. At the domain egress, decapsulation is done which removes the extension headers before the packet is forwarded outside of the domain. The nice part about the encapsulation method is that encapsulated packets are targeted to destinations within the domain which then automatically contains the extension headers within the domain.

Tom

Hi Tom,

My understanding was that Rajiv was referring to draft-ioametal-ippm-6man-ioam-ipv6-deployment, section 5.1.1 - i.e. v6-in-v6 encap.

Cheers, Frank

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 28 Mar 2019 19:50:30 +0000

FYI - A new draft including Mark's comments below just got posted:
https://tools.ietf.org/html/draft-ioametal-ippm-6man-ioam-ipv6-deployment-01 

Frank

-----Original Message-----
From: ipv6 <[email-masked]> On Behalf Of Frank Brockners (fbrockne)
Sent: Dienstag, 26. März 2019 08:19
To: Mark Smith <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: RE: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Hi Mark,

Thanks again, Frank

-----Original Message-----
From: Mark Smith <[email-masked]> 
Sent: Sonntag, 24. März 2019 03:00
To: Frank Brockners (fbrockne) <[email-masked]>
Cc: C. M. Heard <[email-masked]>; 6man WG <[email-masked]>; [email-masked]
Subject: draft-ioametal-ippm-6man-ioam-ipv6-deployment-00 feedback (Re: v6 option types for IOAM data fields)

Hi Frank,

<snip>

Regarding

Regarding,

Regarding,

Some suggested text -

add something like

add something like

Regards,
Mark.