Subject: Re: Insertion of IPv6 Segment Routing Headers in a Controlled Domain

From: Mark Smith <[email-masked]>
Date: Tue, 4 Apr 2017 06:34:00 +1000

On 31 Mar. 2017 01:34, "Dirk Steinberg" <[email-masked]> wrote:

> Am 29.03.2017 um 22:08 schrieb Mark Smith <[email-masked]>:
>
> On 30 March 2017 at 06:58, Mark Smith <[email-masked]> wrote:
>> So having further about this, I have fundamental question that it
>> isn't answering.
>>
>> Why can't IPv6-in-IPv6 encapsulation be used for this? What is missing
>> from IPv6 that "requires" that EH insertion is used instead of much
>> simpler, off-the-shelf encapsulation/decapsulation?
>>
>>
>>
>> For example, using IPv6-in-IPv6 encapsulation, I don't think an SRH is
>> needed at all in the "2. Source Domain and Packet Journey" example.
>>
>> When node 2 realises that the link between itself and 3 has failed, it
>> encapsulates the original SA=1, DA=9 packet in a new IPv6 header
>> ("tunnelling" it). The new IPv6 header has an SA=2, DA=5, and is sent
>> towards node 4.
>>
>> Node 4 forwards the packet onto node 5 using conventional destination
>> address based IPv6 forwarding.
>>
>> Node 5 receives the packet, and as it is DA=5, decapsulates the inner
>> SA=1, DA=9 packet. Node 5 then submits that packet to the standard
>> IPv6 forwarding table, resulting in it being forwarded to Node 3,
>> which then forwards it to Node 9. All of this is happening using
>> conventional IPv6 destination based forwarding.
>>
>> Encapsulation is solving this problem by effectively creating a
>> on-demand virtual link or tunnel between Node 2 and Node 5, getting
>> the original packet past the failure point to a point along the
>> original packet's forwarding path. Once there, it pops out of the
>> virtual link and is sent along its way. I don't think the insertion of
>> the EH and DA swapping method could be described the same way or could
>> be described as simply.
>>
>> I think this IP-in-IPv6 encapsulation solution to the above example is
>> better because:
>>
>> - there is no need for an additional header of any type - the path
>> information is inherent in the DA of the outer IPv6 header when sent
>> to node 5, and in the DA of the inner IPv6 packet (now "the packet")
>> when being sent by node 5 to node 9.
>
> To clarify, what I mean by "no need for an additional header of any
> type" is no additional header beyond the IPv6 encapsulation header
> i.e. no SRH EH or similar between the outer and inner IPv6 packets. It
> is the original IPv6 packet inside a vanilla IPv6 packet with the
> outer IPv6 packet’s next header field set to 41.

Following this logic of not using „an additional header of any type“ leads
to
the conclusion that for every entry in an SR policy represented by the
SID list of an SRH, you would instead use one more nested IPv6
encapsulation header.

In the simple example of section 2 the TI-LFA policy only needs to specify
one additional node, namely 5 (9 being the original destination), but
conceivably the backup policy could be more complex including 3, 4, 5,
or more SIDs. Would you really want to propose that the PLR should
impose, say 4 nested IPv6 encapsulation headers at the same time
while still claiming this to be simpler and/or more efficient?

Let’s face the reality: the cost of adding an IPv6 encapsulation header
is fairly high and while doing this once may be acceptable in certain
situations
doing this N times nested is prohibitive both in terms of the burden on the
forwarding hardware and in terms of MTU growth. One more entry in
the SID list is just so much more efficient.

That is ignoring the complexity and cost of processing introduced at each
hop.

The current (and decades old) IPv6 forwarding model involves hop-by-hop
destination address matching and hop-by-hop link-layer endcap/decap to get
the packet to its destination.

MPLS too follows this model. Labels aren't inserted into the IPv6 packet,
they're added to the outside of it.

EH insertion is fundamentally changing this forwarding model, because
forwarding devices have to do more complicated processing of EH chain
processing to forward IPv6 packets.

SR will be a compelling replacement for MPLS if it leverages existing
commodity IPv6 methods and hardware. If it requires a fork lift upgrade of
all devices in the network, then it's much less attractive compared to MPLS.

This is why in general the recurring argument to just use IPv6 encaps
whenever all you really need is one more entry in the SRH SID list
is completely missing the point.

I haven't gone through all examples to determine what the implications of
IPv6-in-IPv6 encapsulation are, and I acknowledge that the per-packet
overheard is high.

What I have done though is proven that EH insertion is not required or
essential to solve the problem presented in the example. That is the claim
that is being made in this draft.

So please do not make that claim, please do not use giant lists of authors
that could not have contributed enough to the document to have earned it,
effectively making an appeal to authority, and please look to leverage
existing IPv6 processing models and methods to avoid invalidating existing,
well known and widely deployed and used IPv6 devices and techniques.

The bigger and more different the claim from the norm, the more effort you
have to put into proving it.

Regards,
Mark.

/Dirk

>> - Encapsulation/Decapsulation performed by nodes 2 and nodes 5 is
>> conventional IPv6-in-IPv6 encapsulation, specified in RFC2473, now
>> nearly 20 years old.
>>
>> - In this example, nodes 4, 5 and 3 could be off-the-shelf IPv6
>> devices that support conventional IPv6 forwarding and conventional
>> IPv6-in-IPv6 encapsulation/decapsulation (i.e., "tunnelling").
>>
>> - Encapsulation/decapsulation is a simpler, universal and proven
>> operation (literally being done every time an IPv6 packet is being
>> sent and received over any and all layer 2 links)  compared to the DA
>> address swapping that occurs at Nodes 2 and Nodes 5 in the described
>> method.
>>
>> More complicated examples would may require more information to be
>> carried about the path between the inner and outer IPv6 headers
>> (although encapsulation inside encapsulation/tunnelling inside
>> tunnelling probably be used at a greater packet overhead host, with
>> simpler per-hop processing), however I don't think this example
>> demonstrates why EH insertion is required.
>>
>>
>> Regards,
>> Mark.
>
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Robert Raszuk <[email-masked]>
Date: Tue, 4 Apr 2017 08:42:02 +0200

Hi Mark,

> EH insertion is fundamentally changing this forwarding model, because
forwarding devices
> have to do more complicated processing of EH chain processing to forward
IPv6 packets.

Oh really ? Isn't EH insertion explicitly allowed by *all* IPv6 documents
as long as src hosts do it ?

If so how does it change your above assertion on the "complicated
processing of EH chain" ?

So if hosts do it - it is all great, but if network element dares to do it
- it is now just a disaster ?

Are all IPv6 implementations implicitly assume that src host will be too
stupid to ever do header insertion ;) ?
Well surprise ... linux kernel 4.10 can do it.

Best,
Robert.

---

From: Mark Smith <[email-masked]>
Date: Tue, 4 Apr 2017 17:29:24 +1000

If you think fiddling with packet structure and size in the network is
perfectly fine, then I wonder if you've had any operational experience
with troubleshooing NAT, implementation bugs or devices that partially
failure such that they partially corrupt packets.

I don't appreciate the condescending sarcasm either.

On 4 April 2017 at 16:42, Robert Raszuk <[email-masked]> wrote:
> Oh really ? Isn't EH insertion explicitly allowed by *all* IPv6 documents as
> long as src hosts do it ?
>
> If so how does it change your above assertion on the "complicated processing
> of EH chain" ?
>
> So if hosts do it - it is all great, but if network element dares to do it -
> it is now just a disaster ?
>

---

From: Mark Smith <[email-masked]>
Date: Tue, 4 Apr 2017 17:44:54 +1000

> failure such that they partially corrupt packets.

You may also want to read RFC7045, " Transmission and Processing of
IPv6 Extension Headers",

which says, among other related things,

"The IPv6 Hop-by-Hop Options header SHOULD be processed by
   intermediate forwarding nodes as described in [RFC2460].  However, it
   is to be expected that high-performance routers will either ignore it
   or assign packets containing it to a slow processing path.  Designers
   planning to use a hop-by-hop option need to be aware of this likely
   behaviour."

---

From: Satoru Matsushima <[email-masked]>
Date: Tue, 4 Apr 2017 17:38:10 +0900

Hello Mark,

Thank you, it looks very interesting discussion. I’d clarify a bit to your following point:

> [...]

> conventional IPv6 destination based forwarding.

This behavior of node 5 assumes that there’s on-demand virtual tunnel between node 2 and node 5 prior to the failure. Is that correct?

> the EH and DA swapping method could be described the same way or could

So that looks node 9 should define DA=9 semantics which the node pops an outer header and then forward the payload to the inner IPv6 destination based on the routing table. Basically an IP address of a node represents the node itself and receiving packets will be punt to the control-plane. So it seems that the node behavior doesn't work for conventional nodes.

To that behavior works, I think that it requires additional spec to define such semantics and some control-plane to signal that semantics to other node. If I comment to that it would be a certain volume of load for signaling and maintaining the states to the control-planes, even the data-plane behavior can be kept in conventional. And it would also require tunnel configurations to the all nodes in each node which I don’t want to do in our operations.

I have to admit my ignorant, but I’m really appreciate if you let me know about that kind of on-demand tunneling technologies without bunch of signaling to maintain states and configuration burdens.

Best regards,
--satoru

---

From: Satoru Matsushima <[email-masked]>
Date: Tue, 4 Apr 2017 17:41:42 +0900

Oops, please s/9/5/; to the following sentence:

> So that looks node 9 should define DA=9 semantics which the node pops an outer header and then forward the payload to the inner IPv6 destination based on the routing table.

Best regards,
--satoru

---

From: "Stefano Previdi (sprevidi)" <[email-masked]>
Date: Tue, 4 Apr 2017 08:51:15 +0000

> On Apr 4, 2017, at 10:38 AM, Satoru Matsushima <[email-masked]> wrote:
> 
> Hello Mark,
> 

this is part of TI-LFA (and more general fast re-rpoute) technology.

yes, the repair path is pre-computed in the form of a segment list.

> So that looks node 9 should define DA=9 semantics which the node pops an outer header and then forward the payload to the inner IPv6 destination based on the routing table. Basically an IP address…

correct. Please read draft-filsfils-spring-srv6-network-programming and you’ll see we defined an explicit behavior for that.

Suc behavior is signaled in the control plane of the infrastructure (isis, ospf). See extensions described in draft-bashandy-isis-srv6-extensions.

> To that behavior works, I think that it requires additional spec to define such semantics and some control-plane to signal that semantics to other node.

correct. See above.

> If I comment to that it would be a certain volume of load for signaling and maintaining the states to the control-planes, even the data-plane behavior can be kept in conventional. And it would also require tunnel configurations to the all nodes in each node which I don’t want to do in our operations.

good point, but I’m afraid you’re not familiar with TI-LFA. There’s very little state to keep and there’s no tunnel provisioning. This is traditional LFA/IPFRR.

> I have to admit my ignorant, but I’m really appreciate if you let me know about that kind of on-demand tunneling technologies without bunch of signaling to maintain states and configuration burdens.

draft-filsfils-spring-srv6-network-programming 
draft-bashandy-isis-srv6-extensions
RFC7855
draft-ietf-spring-resiliency-use-case
draft-francois-rtgwg-segment-routing-ti-lfa

Keep in mind that LFA technologies have been deployed for a few years already and TI-LFA is being deployed as we speak.

s.

---

From: Satoru Matsushima <[email-masked]>
Date: Tue, 4 Apr 2017 18:18:46 +0900

Hello Stefano,

Thank you for pointing the references in your prompt response. That seems SRv6 can define all the forwarding semantics to support TI-LFA, a fast protection technology without signaling and configuration burdens. So my question here is do you think it is possible to achieve that with whether purely IPv6-in-IPv6 tunneling without the burdens same as SRv6. As Mark said, if it is possible with IPv6-in-IPv6 tunneling only, what else required to do that seems to me is bunch of signaling and configuration burdens.

If SRv6 is a good choice, or only choice for that without the burdens, SRH processing capability must be required to the nodes. I think it would be obvious. So my second question here is that: do we really need to put additional outer IPv6 header just to place the SRH for TI-LFA even all the nodes are fully SRv6 capable nodes which are able to insert SRH to the packets.

Best regards,
--satoru

> 2017/04/04 17:51、Stefano Previdi (sprevidi) <[email-masked]> のメール：
> 
>

---

From: "Stefano Previdi (sprevidi)" <[email-masked]>
Date: Tue, 4 Apr 2017 09:41:15 +0000

> On Apr 4, 2017, at 11:18 AM, Satoru Matsushima <[email-masked]> wrote:
> 
> Hello Stefano,
> 
> Thank you for pointing the references in your prompt response. That seems SRv6 can define all the forwarding semantics to support TI-LFA, a fast protection technology without signaling and configuration burdens. So my question here is do you think it is possible to achieve that with whether purely IPv6-in-IPv6 tunneling without the burdens same as SRv6.

TI-LFA as any other traffic engineering and/or fast-reroute technology, uses the concept of a source routed path.

There are multiple ways to express a source routed path.

Indeed you could express a path by stacking ipv6 headers on on top of each other. This option is clearly unpractical, inefficient and non-scalable.

The best approach is to express a source routed path with the right tool: the routing header, and more precisely the segment routing header (SRH).

> As Mark said, if it is possible with IPv6-in-IPv6 tunneling only, what else required to do that seems to me is bunch of signaling and configuration burdens.

I think everyone in the industry agrees that stacking ipv6 headers as a source routing technique is really unpractical and it’s really a poor choice compared to the excellent tool we have with extensions headers.

I have nothing against bicycles, but If I have to transport freight, I clearly prefer a truck...

> If SRv6 is a good choice, or only choice for that without the burdens, SRH processing capability must be required to the nodes.

to the nodes that are segments and to the nodes that are TI-LFA PLR (point of local repair). This does not mean “all nodes” in the network.

> I think it would be obvious. So my second question here is that: do we really need to put additional outer IPv6 header just to place the SRH for TI-LFA even all the nodes are fully SRv6 capable nodes which are able to insert SRH to the packets.

that is an excellent question.

No, you don’t need an outer encapsulation and in many (if not most) of the cases, the TI-LFA PLR node will not add an outer encapsulation.

What we propose in draft-voyer-6man-extension-header-insertion is to add an outer encapsulation at the ingress of the domain so that, when the PLR node inserts an SRH, it does so under the outer IPv6 header that has been added at ingress. Therefore, the insertion of the SRH happens into a packet that has been sourced within the domain.

Thanks.
s.

---

From: Satoru Matsushima <[email-masked]>
Date: Tue, 4 Apr 2017 19:17:12 +0900

Hello Stefano,

Thank you for the clear answer to my questions. I really appreciate it.

> What we propose in draft-voyer-6man-extension-header-insertion is to add an outer encapsulation at the ingress of the domain so that, when the PLR node inserts an SRH, it does so under the outer IP…

I understand that. Adding outer header at the ingress of the domain seems make sense to me because the incoming packets to the domain include not only IPv6, but also IPv4 or L2 frames. My understanding is that once the packets come into the domain, there’s no need to put additional outer IPv6 header on somewhere within the domain when the packets meet TI-LFA forwarding during a link or node failure. And then the pushed SRH will be popped out at the MP (Merge Point?) of the protection path with in the domain.

It looks very clear and clean behavior, and also make sense to me. The pushed SRH will be confined within the source domain of the SRH, which does no impact to the end-to-end principle.  Oh that's the source domain concept you mentioned. Is that correct?

Best regards,
--satoru

> 2017/04/04 18:41、Stefano Previdi (sprevidi) <[email-masked]> のメール：
> 
>

---

From: "Stefano Previdi (sprevidi)" <[email-masked]>
Date: Tue, 4 Apr 2017 12:44:31 +0000

> On Apr 4, 2017, at 12:17 PM, Satoru Matsushima <[email-masked]> wrote:
> 
> Hello Stefano,
> 

it doesn’t need to.

The inserted SRH will anyway be removed at the same time the outer encapsulation is removed by the egress node of the domain.

> It looks very clear and clean behavior, and also make sense to me. The pushed SRH will be confined within the source domain of the SRH,

correct.

> which does no impact to the end-to-end principle. 

correct.

>  Oh that's the source domain concept you mentioned. Is that correct?

yes, that is correct.

Thanks.
s.

---

From: Dirk Steinberg <[email-masked]>
Date: Tue, 4 Apr 2017 15:49:02 +0200

> Am 03.04.2017 um 22:34 schrieb Mark Smith <[email-masked]>:
> 
> On 31 Mar. 2017 01:34, "Dirk Steinberg" <[email-masked] <mailto:[email-masked]>> wrote:
> 
> > Am 29.03.2017 um 22:08 schrieb Mark Smith <[email-masked] <mailto:[email-masked]>>:
> >
> > On 30 March 2017 at 06:58, Mark Smith <[email-masked] <mailto:[email-masked]>> wrote:
> Following this logic of not using „an additional header of any type“ leads to
> is fairly high and while doing this once may be acceptable in certain situations
> That is ignoring the complexity and cost of processing introduced at each hop. 
> 
> The current (and decades old) IPv6 forwarding model involves hop-by-hop destination address matching and hop-by-hop link-layer endcap/decap to get the packet to its destination.
> 
> MPLS too follows this model. Labels aren't inserted into the IPv6 packet, they're added to the outside of it.
> 
> EH insertion is fundamentally changing this forwarding model, because forwarding devices have to do more complicated processing of EH chain processing to forward IPv6 packets.

There seems to be a fundamental misunderstanding: The normal forwarding of 
SRv6 packets still happens based on the dest address field in the IPv6 header.
The forwarding lookup (for intermediate nodes) does not involve looking at the SRH 
at all, if fact, such nodes do not need to support or know anything about SRv6 at all.

Only segment endpoint nodes, the nodes performing SR-specific functions may have
to look at the SRH and even then, for the common operation that is analogous to 
the POP operation in MPLS, only a new dest address needs to be copied from the 
SRH  and a single counter needs to be decremented. Never is the forwarding based
directly on the SRH.

A node that wants to do SRH insertion of course has to do some heavy lifting, 
that is understood. A 5 year old hardware engine may not be able to do this.
But then we are talking about highly sophisticated and desirable functionality 
(e.g. TI-LFA) and there is always a price: there is no such thing as a free lunch.

> SR will be a compelling replacement for MPLS if it leverages existing commodity IPv6 methods and hardware. If it requires a fork lift upgrade of all devices in the network, then it’s much less attractive compared to MPLS.

Again, no forklift upgrade, but where you want significant new functionality,
there you may have to pay a price, maybe a hardware upgrade of a line card.

And, BTW, when MPLS was introduced, it was not that any IP router could be
made to do MPLS by a simple software upgrade. On the contrary, the situation
was quite similar: where you wanted to deploy new capabilities such as MPLS
FRR, you most likely needed a hardware upgrade.

So I do not see how MPLS is (has been) any better in this respect. Especially 
for the operation that is analogous to SRH insertion, namely MPLS PUSH,
I vividly remember early hw platforms only being able to push 2 or 3 labels
at a time, limiting what you could do with MPLS. Only today, after more than 
15 years of MPLS have we come to a point where we can safely assume that 
new hardware will be able to push „enough“ MPLS labels.

/Dirk

> I haven't gone through all examples to determine what the implications of IPv6-in-IPv6 encapsulation are, and I acknowledge that the per-packet overheard is high.
> 
> What I have done though is proven that EH insertion is not required or essential to solve the problem presented in the example. That is the claim that is being made in this draft.
> 
> So please do not make that claim, please do not use giant lists of authors that could not have contributed enough to the document to have earned it, effectively making an appeal to authority, and please look to leverage existing IPv6 processing models and methods to avoid invalidating existing, well known and widely deployed and used IPv6 devices and techniques.
> 
> The bigger and more different the claim from the norm, the more effort you have to put into proving it.
> 
> Regards,
> Mark.
> 
> 
> /Dirk
> 
> > [email-masked] <mailto:[email-masked]>
> > Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6 <https://www.ietf.org/mailman/listinfo/ipv6>

---

From: John Leslie <[email-masked]>
Date: Tue, 4 Apr 2017 11:04:48 -0400

> failure such that they partially corrupt packets.

   I admit to no such experience within IPv6.

   Can you point out anyone publishing such experience?

--
John Leslie <[email-masked]>

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 5 Apr 2017 09:47:43 +1200

Robert,
> as long as src hosts do it ?

Then it's not *insertion*, which makes an existing packet longer on the fly.
It's simply the creation of a packet that happens to include a particular
extension header. Which is why draft-ietf-6man-segment-routing-header-06
is not problematic.

> processing of EH chain" ?

There's a problem described in RFC7045: what happens when an unknown new
extension header type transits a middlebox that drops packets with
unknown headers? But that applies regardless of whether the header
was inserted on the fly, so I think Mark was wrong there.
 
> - it is now just a disaster ?

Yes, because it changes the packet size mid-way, and that definitely breaks
PMTUD, as has been discussed here often enough. Until the rules of how
to configure the "controlled domain" are written in such a way as to avoid 
this being a problem, which I expect we'll see in later versions of
draft-voyer-6man-extension-header-insertion.

Regards
   Brian

---

From: Robert Raszuk <[email-masked]>
Date: Wed, 5 Apr 2017 00:03:23 +0200

​Hi Brian,

> draft-voyer-6man-extension-header-insertion.

​How about alternative out of the box idea to accommodate expressed above
concern ?​

Would you and WG agree that is it legal to perform SID insertion,
modification and deletion if source hosts add to the packet NULL SRH and
NULL SIDs (marked as such in a well defined way) ?

So packet size will *never be increased* while all flexibility and
innovation provided by SRv6 could be easily accomplished regardless of the
definition and boundaries of "controlled domain" ?

SRv6 aware nodes would be free to use that space as it seems fit without
any additional encapsulation anywhere as long as the original placeholder
is sufficient for all network functions.

Comments ? Opinions ?

Kind regards,
Robert Raszuk.

PS. Flavor of the above would be to encapsulate only once when packet
enters SR domain and leave enough NULL space in the SRH to accommodate what
is expected to be needed during forwarding via given domain. But here the
requirement to reassemble is a bit of a challenge for various reasons
already stated.

---

From: Mark Smith <[email-masked]>
Date: Wed, 5 Apr 2017 09:43:46 +1000

> extension header. Which is why draft-ietf-6man-segment-routing-header-06

Exactly. Existing "insertion" is only permitted by the host that
originates the packet. IPv6 packet origination is a host function
(frame origination is too - routers encapsulating layer 3 packets in
link-layer frames are hosts on the link at layer 2)

In the network, nothing is currently splitting apart IPv6 packets and
inserting something in them while in flight. 6PE doesn't do this - it
encapsulates to add information to the packet, and it isn't adding
IPv6 relevant information to the packet - MPLS labels are MPLS
protocol information, not IPv6 protocol information, and the MPLS
label information isn't inserted anywhere inside the IPv6 packet that
is transported across the 6PE domain.

> was inserted on the fly, so I think Mark was wrong there.

I was using NAT as example of something that mangles packets in
flight, beyond standard IP forwarding, and that if they don't work as
expected, which can include because they've failed, they can be hard
to troubleshoot because the mangling isn't expected or well defined,
and in one direction (outside to inside), the NAT device's identity
isn't recorded in the packet (in the inside to outside direction, the
NAT device updates the source address, effectively making it a host on
the outside network and effectively making it the originator of a new
packet.)

The complicated processing I was referring to was that insertion of
EHs would involve processing going beyond the fixed IPv6 header, and
the complexity of processing (examining, evaluating, skipping,
inserting, deleting) a chain of one or more EHs. I wouldn't expect an
SR domain to have middle boxes internally, so I don't think the
unknown EH issue with middle boxes would be an issue.

Current high speed IPv6 forwarding devices don't need to look past the
IPv6 fixed header to perform their IPv6 forwarding function. If they
happen to e.g., TCP/UDP ACLs, then they assume that the TCP/UDP header
(EH) is directly after the IPv6 fixed header, with that information
also in a fixed location in the packet. If it isn't there, they give
up and either drop the packet or forward it regardless.

> draft-voyer-6man-extension-header-insertion.

I just don't believe this is possible to do, unless there is a
statement such as the following:

"SR domains MUST NOT be attached to the Internet or any other network
where a inserted EH, when received, would be unexpected, due to the
consequences of failure to remove inserted EHs as the domain
boundary."

Even if that was included, there would be no way to police this, and
the victims of the failed EH removal may be far removed from the
people who decided to insert them and who thought they were removing
them.

Regards,
Mark.

---

From: Mark Smith <[email-masked]>
Date: Wed, 5 Apr 2017 10:40:20 +1000

>    I admit to no such experience within IPv6.

It's not an experience exclusive to IPv6. IPv6 implementations are not
and are not going to be impervious to implementation bugs or partial
device failures.

The trouble with NAT is pretty well known, here is my recent
presentation and writings on it (I think the difference from other
presentations was that I used what I consider Network Critical Success
Factors to be the NAT evaluation criteria):

AusNOG 2016 Presentation - "The Trouble with NAT"
"http://www.ausnog.net/sites/default/files/ausnog-2016/presentations/1.2_Mark_Smith_AusNOG2016.pdf

APNIC series of 3 blog articles for the above presentation
https://blog.apnic.net/author/mark-smith/

>    Can you point out anyone publishing such experience?

I have a number of first hand experiences with devices not behaving as
specified or expected.

"Residential IPv6 CPE - What Not To Do and Other Observations"
http://www.ausnog.net/sites/default/files/ausnog-05/presentations/ausnog-05-d02p02-mark-smith.pdf

In ~2008/2009 an organisation I was working for bought a mid-range
router from a very well known vendor. It was using a 20+ year old code
base that had been widely deployed. It suffered from a fault where it
was leaking ethernet frames between the 3 physical interfaces, even
though they'd been configured as separate layer 3 interfaces, causing
a link to be taken off-line by the provider it was attached to due to
MAC address limits. That was a significant failure to the business, as
the money that was being saved by using it was going to pay for the
router within less than a month.

So when people say or effectively say that all implementations will
follow specifications to the letter, or be configured 100% accurately
in all cases, I know that won't be the case. A statement such as "An
inserted SRH EH MUST be removed when the packet exits the SR domain"
is a specification instruction on mandatory behaviour, but isn't a
guarantee it will happen 100% of time when the specification is
implemented or deployed.

IPv6-in-Ipv6 encapsulation implementations will not behave be
perfectly 100% of the time either, however the actions it is
performing (encapsulation, tunnelling) are well known and widely
deployed, and have known failure modes that are expected and can be
handled with by hosts and routers that may receive those packets in
error.

Regards,
Mark.

---

From: Mark Smith <[email-masked]>
Date: Wed, 5 Apr 2017 12:19:51 +1000

Hi,

> This behavior of node 5 assumes that there’s on-demand virtual tunnel between node 2 and node 5 prior to the failure. Is that correct?

No. I was using the term "virtual" to mean "conceptual", as they
pretty much mean the same thing, although thinking about it "virtual"
does have a more loaded meaning here in the IETF e.g., VLLs.

"on-demand" was probably a mistake too, I didn't mean a tunnel was
setup or signalled on demand, what I meant was there wasn't anything
pre-existing that specifically needed to be be created prior to the
packet being encapsulated and sent to node 5.

So what is being created is a conceptual link or tunnel between node 2
and node 5 to bypass the failure, there is no configuration or
signalling required for it, support for this is inherent in node 5
being able to receive and process an IPv6-in-IPv6 packet that node 2
sent to node 5 (i.e., match incoming DA, match on Next Header Type 41,
decapsulate inner IPv6 packet and submit that packet to the standard
IPv6 forwarding function for further forwarding.)

> So that looks node 9 should define DA=9 semantics which the node pops an outer header and then forward the payload to the inner IPv6 destination based on the routing table.

I may not be understanding, however going by the example scenario,
DA=9 is the final destination of the packet originated by SA=1. This
SA=1, DA=9 packet is the one initially sent by Node 1. What ever is in
it is what ever SA=1 intended for DA=9 to process, the example doesn't
say.

The goal of the example seems to be to demonstrate how the SA=1, DA=9
packet bypasses the failure of the link between node 2 and 3, reaching
its final destination of DA=9. The example shows how it would be
achieved via SRH EH insertion, I've shown how it could be achieved via
IPv6-in-IPv6 encapsulation.

This is in no way innovative on my part, a variety of things and
experiences have made me realise the value in tunnels to get packets
to other places in networks where the underlying network can't or
won't:

- IPsec VPNs over the Internet carrying RFC1918 addressed traffic

- Many years of using a 6to4 tunnel to gain IPv6 Internet access over
a residential IPv4 only service

- IP-in-IP tunnels to transport AS transit traffic between iBGP peers
where the underlying network doesn't carry the full route table,
mentioned in "A.2.3 Encapsulation" of RFC1772, "Application of the
Border Gateway Protocol in the Internet"

- I think a draft briefly looked at a few years ago of RFC7490,
"Remote Loop-Free Alternate (LFA) Fast Reroute (FRR)" which mentions
that IP-in-IP tunnels could be used.

I like the idea of using tunnelling to "teleport" packets to somewhere
else because the idea of doing so is at least as old as the earliest
version of RFC1772, RFC1164 from 1990.

> Basically an IP address of a node represents the node itself and receiving packets will be punt to the control-plane. So it seems that the node behavior doesn't work for conventional nodes.
>

For IPv6, by my count there are at least 6, described in

RFC7059, "A Comparison of IPv6-over-IPv4 Tunnel Mechanisms"

namely,

Automatic Tunneling
IPv6 over IPv4 without Explicit Tunnels (6over4)
Connection of IPv6 Domains via IPv4 Clouds (6to4)
Intra-Site Automatic Tunnel Addressing (ISATAP)
Tunneling IPv6 over UDP through NATs (Teredo)
Peer-to-Peer IPv6 on Any Internetwork (6bed4)

Regards,
Mark.

---

From: Stefano Salsano <[email-masked]>
Date: Wed, 5 Apr 2017 08:56:37 +0200

Hi Mark,

in the case discussed in draft-voyer-6man-extension-header-insertion
the EH is inserted in a outer packet with a destination IPv6 address 
within the "controlled domain".

The original packet will exit the controlled domain only when it is 
decapsulated by the exit node in the controlled domain. So actually, 
there is no an "EH removal" operation that a node may fail to execute: 
either the original packet is decapsulated and forwarded outside the 
domain (and for sure it will have no EH inside) or if there are problems 
they are confined in the controlled domain... as the outer packet has 
only IPv6 addresses belonging to the controlled domain

regards,
Stefano

-- 
*******************************************************************
Stefano Salsano
Professore Associato
Dipartimento Ingegneria Elettronica
Universita' di Roma Tor Vergata
Viale Politecnico, 1 - 00133 Roma - ITALY

http://netgroup.uniroma2.it/Stefano_Salsano/

E-mail  : [email-masked]
Cell.   : +39 320 4307310
Office  : (Tel.) +39 06 72597770 (Fax.) +39 06 72597435
*******************************************************************

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 6 Apr 2017 01:29:21 +1200

> only IPv6 addresses belonging to the controlled domain

Also Mark - let's not prejudge the way the controlled domain might be
defined and managed. And also let's not overstate the dangers of a
rogue packet with an SR header escaping; in the end it will either
be dropped by a paranoid firewall, or ignored by an uninterested
destination. The rfc2460bis rule is to enable traffic that is
*intended* to cross the Internet on an arbitrary path. Here we are
talking about traffic that is in any case not intended to do so.

   Brian