Subject: Re: [Int-area] I-D Action: draft-ietf-intarea-gre-ipv6-07.txt

From: Ronald Bonica <[email-masked]>
Date: Thu, 16 Apr 2015 22:43:22 +0000

Fred,

In Section 3.2, we put implementation details of the probing mechanism out of scope.

Regardless of MTU and fragmentation issues, a robust GRE implementation requires a probing mechanism. Without such a probing mechanism, a GRE ingress node might activate a GRE tunnel when the egress was not reachable at all. This would result in black-holing.

Since RFC 2784 doesn't specify a probing mechanism (or even require one) most vendors have developed proprietary probing mechanisms. These proprietary mechanism must deal with the problems of LAG and ECMP. However, because the probing mechanisms aren't IPv6 specific, and because they are proprietary, we put them out of scope.

                                                                                                                               Ron

> ------------------------------
> 
> Message: 2
> Date: Tue, 14 Apr 2015 17:12:53 +0000
> From: "Templin, Fred L" <[email-masked]>
> To: "[email-masked]" <[email-masked]>
> Subject: Re: [Int-area] I-D Action: draft-ietf-intarea-gre-ipv6-07.txt
> Message-ID:
> 	<2134F8430051B64F815C691A62D9831832E495AD@XCH-BLV-
> 504.nw.nos.boeing.com>
> 
> Content-Type: text/plain; charset="us-ascii"
> 
> I will say also that the act of probing in general is a bit troublesome. In
> particular, when there is Equal Cost MultiPath (ECMP) how can you tell that
> the probe packets will follow the same path as the data packets?
> 
> Thanks - Fred
> [email-masked]
> 

[RPB] 
***

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 17 Apr 2015 02:14:48 +0000

Hi Ron,

> -----Original Message-----
> From: Int-area [mailto:[email-masked]] On Behalf Of Ronald Bonica
> Sent: Thursday, April 16, 2015 3:43 PM
> To: [email-masked]

But, Section 3.2 is all about probing for a specific size - 1280 bytes.

> Regardless of MTU and fragmentation issues, a robust GRE implementation requires a probing mechanism. Without such a probing
> mechanism, a GRE ingress node might activate a GRE tunnel when the egress was not reachable at all. This would result in black-
> holing.

Probing for liveness, I can understand. Like maybe send a little 100 byte packet and
see if it boomerangs back to you. But, probing with 1280 might not tell the whole
story if there are ECMP or LAG branches on the path.

Also, while probing, why not also check to see if it supports GRE fragmentation?

> Since RFC 2784 doesn't specify a probing mechanism (or even require one) most vendors have developed proprietary probing
> mechanisms. These proprietary mechanism must deal with the problems of LAG and ECMP. However, because the probing
> mechanisms aren't IPv6 specific, and because they are proprietary, we put them out of scope.

Your document Section 3.2 is asking for probing the IPv6 minMTU of 1280 plus
the encapsulation overhead. That might work on one path of a multipath but
actual data packets might take a different path. This does not make me happy
because I also want to do some probing of my own. But, it seems like
something we may need to worry about.

Thanks - Fred

> _______________________________________________
> Int-area mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/int-area

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 22 Apr 2015 21:14:22 +0000

Hi Ron,

> something we may need to worry about.

I just stumbled across RFC6438 ("Using the IPv6 Flow Label for Equal Cost
Multipath Routing and Link Aggregation in Tunnels") and it seems to be a
proposed standard for ensuring that tunneled packets are assigned an
appropriate flow label according to the traffic flows they encapsulate.
Meaning that ECMP and LAG between the tunnel ingress and egress can
spread tunneled traffic over multiple paths. Meaning that probing the
tunnel to determine if a particular size is supported only tests one path
when there may in fact be many.

Based on that, probing from the source via something like RFC4821 should
be fine because the probes look exactly like ordinary data packets. But,
probing from a tunnel ingress is problematic when there is ECMP/LAG
because the ingress may have to tunnel the data packets of many flows.

AFAICT, the only way around this is for the tunnel to assign the same
flow label for every encapsulated packet (i.e., the "pipe" model). But,
that would defeat the ECMP/LAG function that the network operators
paid good money for.

Sorry I don't have better news,

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Wed, 22 Apr 2015 22:31:59 +0000

Hi Ron,

So, this gives way to the following, which may be the best we
can do if we can't probe:

1) When there is operational assurance that path MTU discovery between
    the egress, ingress and original source will work correctly, the ingress
    handles packets as follows:
    - for packets no larger than 1280-ENCAPS, encapsulate and send
     without fragmentation
    - for packets larger than 1280-ENCAPS but no larger than 1280,
      encapsulate and fragment into two approximately equal-length
      fragments. (Or, if there is operational assurance that there will
      be no links with MTU smaller than 1280+ENCAPS in the path,
      encapsulate and send without fragmentation.)
    - for packets larger than 1280 but no larger than the tunnel MTU,
      encapsulate and send without fragmentation

2) When there is no operational assurance that path MTU discovery
    between the egress, ingress and original source will work correctly,
    the ingress handles packets as follows:
    - for packets no larger than 1280-ENCAPS, encapsulate and send
     without fragmentation
    - for packets larger than 1280-ENCAPS but no larger than 1500,
      encapsulate and fragment into two approximately equal-length
      fragments. (Or, if there is operational assurance that there will
      be no links with MTU smaller than 1500+ENCAPS in the path,
      encapsulate and send without fragmentation.)
    - for packets larger than 1500, encapsulate and send without
      fragmentation

Thanks - Fred

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 23 Apr 2015 15:17:40 +0000

Fred,

If we can't probe a tunnel, we can't do GRE at all. Ever. Not with IPv4, not with IPv6, not with anything.

MTU issues aside, before activating a tunnel, we must be certain that the tunnel can carry a 1-byte packet from GRE ingress to egress. If we don't do that, we risk black-holing traffic. This is why nearly all GRE implementations have some type of proprietary probing mechanism.

Beyond suggesting that the probing mechanism use a 1280-byte payload, we have put the details of that probing mechanism out of scope for this document.

                                                                                                 Ron

> From: Templin, Fred L [mailto:[email-masked]]
> Sent: Wednesday, April 22, 2015 6:32 PM
> To: Ronald Bonica; [email-masked]
> Subject: RE: [Int-area] I-D Action: draft-ietf-intarea-gre-ipv6-07.txt
> 
> Hi Ron,
> 
> So, this gives way to the following, which may be the best we can do if we
> can't probe:
>

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 15:58:44 +0000

Hi Ron,

> From: Ronald Bonica [mailto:[email-masked]]
> Sent: Thursday, April 23, 2015 8:18 AM
> To: Templin, Fred L; [email-masked]
> MTU issues aside, before activating a tunnel, we must be certain that the tunnel can carry a 1-byte packet from GRE ingress to egress.
> If we don't do that, we risk black-holing traffic. This is why nearly all GRE implementations have some type of proprietary probing
> mechanism.

I said earlier that I am fine with going ahead and probe with a small packet
(I said 100 bytes, but 1 byte would be fine too) to see if the GRE ingress is
alive. But, that probe does nothing to test for a particular MTU.

> Beyond suggesting that the probing mechanism use a 1280-byte payload, we have put the details of that probing mechanism out of

If a 1280 byte probe (which becomes 1280+ENCAPS bytes following
encapsulation) succeeds along one path of a multipath, but would
have failed along another path, then there is possibility for black
holing. And, there is no way for the ingress to know that there
may be ECMP or LAG on the path to the egress. Hence, having the
ingress probe for the purpose of minimum MTU detection is
problematic if there may be some kind of multipath.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Apr 2015 09:15:46 -0700

On 4/23/2015 8:58 AM, Templin, Fred L wrote:
...
> problematic if there may be some kind of multipath.

Probe success tells you only that. Probe loss tells you only that.

IP allows loss, so strictly there's no reason to shut a tunnel down due
to loss. Burst lost or patterns of loss might be undesirable, but
they're allowed by IP.

However, it might be useful to note that probe success isn't the only
issue; that probe loss over 1% might be considered problematic (using
the typical performance needed for 'reasonable' TCP progress).

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 16:27:46 +0000

Hi Joe,

> From: Joe Touch [mailto:[email-masked]]
> Sent: Thursday, April 23, 2015 9:16 AM
> To: Templin, Fred L; Ronald Bonica; [email-masked]

That is fine, but the troubling condition is when the tunnel ingress
successfully probes for 1280 (or 1500) but that same probe would
have failed if it travelled along a different path. And, the ingress
has no way of knowing whether there is ECMP or LAG on the path
to the egress. Which returns us to fragmentation as I said a couple
of messages back.

Thanks - Fred

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 23 Apr 2015 16:34:30 +0000

Fred,

The following are failure modes for a GRE tunnel that includes ECMP/LAG:

- one ECMP/LAG member is dropping 100% of traffic, regardless of packet size
- one ECMP/LAG member is dropping all packets with MTU > N

Why is a probing mechanism OK to detect the first and not the second?

                                                                 Ron

> Sent: Thursday, April 23, 2015 11:59 AM
> > MTU issues aside, before activating a tunnel, we must be certain that the
> tunnel can carry a 1-byte packet from GRE ingress to egress.
> > If we don't do that, we risk black-holing traffic. This is why nearly
> > all GRE implementations have some type of proprietary probing
> mechanism.
> 
> I said earlier that I am fine with going ahead and probe with a small packet (I
> said 100 bytes, but 1 byte would be fine too) to see if the GRE ingress is alive.
> But, that probe does nothing to test for a particular MTU.
> 
> > Beyond suggesting that the probing mechanism use a 1280-byte payload,
> > we have put the details of that probing mechanism out of scope for this
> document.
> 
> encapsulation) succeeds along one path of a multipath, but would have failed
> along another path, then there is possibility for black holing. And, there is no
> way for the ingress to know that there may be ECMP or LAG on the path to
> the egress. Hence, having the ingress probe for the purpose of minimum
> MTU detection is problematic if there may be some kind of multipath.
> 
> Thanks - Fred
> > > So, this gives way to the following, which may be the best we can do
> > > if we can't probe:
> > >

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 16:49:22 +0000

Hi Ron,

> Sent: Thursday, April 23, 2015 9:35 AM

These are two very different conditions. In the first case, if one or more
paths are dropping all packets, then any communications will black hole
and not just GRE tunneling. The network operations staff needs to do
something to ensure that all paths are working.

In the second case, it could black hole even when all paths are working
and using standards-compliant MTUs on all links.

AFAICT, when there may be ECMP or LAG on the path (which the ingress
has no way of knowing) about all you can tell from probing is whether
the egress is alive - probing can't be relied on to tell you anything about
the path(s).

Again , this does not make me happy but AFAICT it is the reality we
need to deal with.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Apr 2015 09:54:38 -0700

On 4/23/2015 9:27 AM, Templin, Fred L wrote:
...
> have failed if it travelled along a different path.

I repeat: Probe success tells you only that. Probe loss tells you only that.

What "would have" happened is irrelevant.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 17:01:09 +0000

> Sent: Thursday, April 23, 2015 9:55 AM

Then, let me also repeat that "about all you can tell from probing is whether
the egress is alive - probing can't be relied on to tell you anything about the
path(s)"

Again, this does not make me happy because I have been trying to figure
out for years how to make tunnels probe the MTU.  The only alternative
I know of is to fragment when necessary per what I said a couple of
messages back.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Apr 2015 10:13:48 -0700

On 4/23/2015 10:01 AM, Templin, Fred L wrote:
...
> the egress is alive - probing can't be relied on to tell you anything about the

We're not doing routing. We're determining whether packets make it to
the egress.

> out for years how to make tunnels probe the MTU. 

If the probe makes it, it makes it.

If the probe "would have failed" on another path, you'll either see that
over several probes or not. If you do, you have what is isomorphic to a
lossy link, and you decide how you want to handle that - if you can
fragment to get around it, then you might try that. If you have FEC, you
might try that.

The cause of the losses is irrelevant.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 17:39:50 +0000

Hi Joe,

> Sent: Thursday, April 23, 2015 10:14 AM
> over several probes or not.

Only if you craft your probes in such a way that they eventually test
all paths in a multipath arrangement. That might require quite a lot
of probes (e.g., if every conceivable flow label value needs to be
probed).

> If you do, you have what is isomorphic to a
> lossy link,

It would actually be much worse than a lossy link, because some
traffic would receive good service while other traffic would be
dropped completely.

> and you decide how you want to handle that - if you can
> fragment to get around it, then you might try that.

Fragmentation works in all cases, AFAICT.

> If you have FEC, you

Sure, but then probing for a specific MTU size is hopeless.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Apr 2015 10:51:05 -0700

On 4/23/2015 10:39 AM, Templin, Fred L wrote:
...
...
> all paths in a multipath arrangement. 

I need to do no such thing.

I only care about what happens as it happens. If the path changes and it
affects loss, then I react to loss. If the path doesn't change and the
loss changes, I react to loss.

There's nothing that requires that any network path "test all paths" in
a multipath system.

,,,
> traffic would receive good service while other traffic would be

If the loss starts depending on the contents of the packets, then we'd
need a new mechanism to provide feedback on existing packets as if they
were all probes.

That's clearly out of scope for this doc.

Although I agree that fragmentation should be used *if available* to
avoid this issue, we're clearly talking about when that's not the case
either because of hardware or protocol limitations.

Joe

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 23 Apr 2015 17:57:08 +0000

Fred,

Let me restate the question slightly.....

- one ECMP/LAG member is dropping 100% of IPv6 over GRE traffic, regardless of packet size. It is doing this because of a firewall filter.
- one ECMP/LAG member is dropping 100% of IPv6 over GRE traffic when the IPv6 payload is larger than 1280 bytes. It is doing this due to an MTU issue.

                                                             Ron

> Sent: Thursday, April 23, 2015 12:49 PM
> paths are dropping all packets, then any communications will black hole and
> not just GRE tunneling. The network operations staff needs to do something
> to ensure that all paths are working.
> 
> In the second case, it could black hole even when all paths are working and
> using standards-compliant MTUs on all links.
> 
> AFAICT, when there may be ECMP or LAG on the path (which the ingress has
> no way of knowing) about all you can tell from probing is whether the egress
> is alive - probing can't be relied on to tell you anything about the path(s).
> 
> Again , this does not make me happy but AFAICT it is the reality we need to
> deal with.
> 
> Thanks - Fred
> > > > If we can't probe a tunnel, we can't do GRE at all. Ever. Not with
> > > > IPv4, not
> > > with IPv6, not with anything.
> > > >
> > > > If we don't do that, we risk black-holing traffic. This is why
> > > > nearly all GRE implementations have some type of proprietary
> > > > probing
> > > mechanism.
> > >
> > > I said earlier that I am fine with going ahead and probe with a
> > > small packet (I said 100 bytes, but 1 byte would be fine too) to see if the
> GRE ingress is alive.
> > > > Beyond suggesting that the probing mechanism use a 1280-byte
> > > > payload, we have put the details of that probing mechanism out of
> > > > scope for this
> > > document.
> > >
> > > holing. And, there is no way for the ingress to know that there may
> > > be ECMP or LAG on the path to the egress. Hence, having the ingress
> > > probe for the purpose of minimum MTU detection is problematic if there
> may be some kind of multipath.
> > >
> > > Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 18:04:51 +0000

Hi Joe,

> Sent: Thursday, April 23, 2015 10:51 AM

You are missing the point. With ECMP or LAG, a middlebox in the path
from the ingress to the egress selects a path for the tunneled packet
by examining information in the headers such as the flow label. If all
tunneled packets look the same to the middlebox (i.e., the pipe model)
then they will all take the same path and it becomes conceivable to test
the (uni-path) MTU through probing. But, if tunneled packets are crafted
to reflect the flow information in the data packets they are carrying, then,
the middlebox can send different tunneled packets across different
paths unbeknownst to the tunnel ingress.

Let's say that there are 2^20 paths in the multipath (i.e., one for each
flow label) - how can the tunnel test the MTU of all paths?

> need a new mechanism to provide feedback on existing packets as if they

The data packets themselves as probes is the only way to test the paths
that the data packets take. The probe reply is a Packet Too Big message.
But, if you get a PTB for a data packet that is smaller than 1280 you are
back to fragmentation.

> either because of hardware or protocol limitations.

Huh?

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 18:10:03 +0000

Hi Ron,

> Sent: Thursday, April 23, 2015 10:57 AM

That would be a case of a broken ECMP/LAG configuration. If one path of an
equal cost multipath is blocked while the others are open, the blocked path
needs to be operationally removed from the ECMP/LAG path selection engine.
This is true for both tunneled and non-tunneled traffic.

> - one ECMP/LAG member is dropping 100% of IPv6 over GRE traffic when the IPv6 payload is larger than 1280 bytes. It is doing this due

I think you mean "when the GRE payload packet is larger than 1280 bytes minus
the GRE overhead"?

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Apr 2015 11:17:40 -0700

...
> by examining information in the headers such as the flow label. 
...

>> If the loss starts depending on the contents of the packets,

I.e., "the point".

>> then we'd

I.e., out of scope.

> But, if you get a PTB for a data packet that is smaller than 1280 you are

Or you don't support the necessary fragmentation and the link is
declared down.

Or you don't get the PTB (ICMP blocking), you're back to a black hole.

Again, we'd need new mechanism and that's out of scope.

Joe

---

From: Ronald Bonica <[email-masked]>
Date: Thu, 23 Apr 2015 18:21:07 +0000

Fred, 

Let's try again.....

- one ECMP/LAG member is dropping 100% of IPv6 over GRE traffic, regardless of packet size. It is doing this because of a firewall filter.
- one ECMP/LAG member is dropping 100% of IPv6 over GRE traffic when the IPv6 payload is 1280 bytes or larger. It is doing this due to an MTU issue. If the MTU were one byte larger, we would be OK.

                                                                                                  Ron

> Sent: Thursday, April 23, 2015 2:10 PM
> > - one ECMP/LAG member is dropping 100% of IPv6 over GRE traffic,
> regardless of packet size. It is doing this because of a firewall filter.
> 
> > - one ECMP/LAG member is dropping 100% of IPv6 over GRE traffic when
> > the IPv6 payload is larger than 1280 bytes. It is doing this due to an MTU
> issue.
> 
> I think you mean "when the GRE payload packet is larger than 1280 bytes
> minus the GRE overhead"?
> 
> Thanks - Fred
> > > These are two very different conditions. In the first case, if one
> > > or more paths are dropping all packets, then any communications will
> > > black hole and not just GRE tunneling. The network operations staff
> > > needs to do something to ensure that all paths are working.
> > >
> > > In the second case, it could black hole even when all paths are
> > > working and using standards-compliant MTUs on all links.
> > >
> > > has no way of knowing) about all you can tell from probing is
> > > whether the egress is alive - probing can't be relied on to tell you
> anything about the path(s).
> > >
> > > > > > If we can't probe a tunnel, we can't do GRE at all. Ever. Not
> > > > > > with IPv4, not
> > > > > > payload, we have put the details of that probing mechanism out
> > > > > > of scope for this
> > > > > document.
> > > > >
> > > > > have failed along another path, then there is possibility for
> > > > > black holing. And, there is no way for the ingress to know that
> > > > > there may be ECMP or LAG on the path to the egress. Hence,
> > > > > having the ingress probe for the purpose of minimum MTU
> > > > > detection is problematic if there
> > > > > > > So, this gives way to the following, which may be the best
> > > > > > > we can do if we can't probe:
> > > > > > >

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 19:05:06 +0000

Hi Joe,

> Sent: Thursday, April 23, 2015 11:18 AM

So, if the tunnel is working but you all of a sudden get a PTB that reports
a size that would be too small to accommodate a 1280 byte payload packet,
shut the tunnel down? You had better be able to trust that the source
of the PTB is trustworthy and not sending bogus PTBs. This sounds like
a condition that can only hold when the tunnel ingress, egress and all
routers in the path between are under the same administrative control.

> Or you don't get the PTB (ICMP blocking), you're back to a black hole.

Right, which as we know can happen when the ingress, egress and
all routers in between are not under the same administrative control.

This strongly suggests that operational considerations are extremely
important to the way the ingress must conduct its business. We have
1) the path from the original source to the tunnel ingress, and 2) the
path from the tunnel ingress to the tunnel egress.

If path 1) is susceptible to either lost or bogus PTBs, then the tunnel
ingress must ensure that packets as large as 1500 bytes get through
even if some fragmentation is needed. If 1) is OK, but 2) allows for
lost or bogus PTBs, then the tunnel ingress must ensure that packets
as large as 1280 get through.

> Again, we'd need new mechanism and that's out of scope.

The mechanism is fragmentation; that is not new.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 19:11:59 +0000

Hi Ron,

> Sent: Thursday, April 23, 2015 11:21 AM
> - one ECMP/LAG member is dropping 100% of IPv6 over GRE traffic when the IPv6 payload is 1280 bytes or larger. It is doing this due to

Again, I think you mean "when the IPv6 payload is larger than 1280 bytes minus
the GRE encapsulation overhead".

> an MTU issue. If the MTU were one byte larger, we would be OK.
> 

The first condition is a network operational misconfiguration, while the second
condition is not necessarily a misconfiguration. You can't use fragmentation
to mask the first condition, but you can to overcome the second condition.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Apr 2015 12:17:37 -0700

On 4/23/2015 12:05 PM, Templin, Fred L wrote:
...
> shut the tunnel down?

Yes.

> You had better be able to trust that the source
> of the PTB is trustworthy and not sending bogus PTBs.

That's always been true.

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 19:29:05 +0000

Hi Joe,

> Sent: Thursday, April 23, 2015 12:18 PM

Only in tightly-controlled operational environments, where there are
no bad routers on the path, no ICMP PTB blockages, and no opportunity
for an attacker to inject a spurious PTB. And also, only when the path
from the tunnel ingress to the original source is equally well behaved
(otherwise, the original source would see a black hole).

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Apr 2015 12:48:06 -0700

On 4/23/2015 12:29 PM, Templin, Fred L wrote:
...
> (otherwise, the original source would see a black hole).

Bad routers can always send bad PTBs, and PTBs can be (and have been)
blocked.

None of that is new, and it can impact all packet sources including
tunnels and real endpoints.

That's also why the mechanism we discussed does NOT rely on PTBs - it
relies on liveness detection.

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 19:56:17 +0000

Hi Joe,

> Sent: Thursday, April 23, 2015 12:48 PM

But, good PTBs are useful when you can get them.

> relies on liveness detection.

Unless you want the egress to ACK each data packet received, I don't
see how that squares with ECMP/LAG. If your liveness detection is
only checking some paths and not others, data packets may be black
holing over other (untested) paths.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Thu, 23 Apr 2015 21:41:55 +0000

Hi Joe,

> holing over other (untested) paths.

More importantly, failure to receive an ACK is not necessarily a sign
of a path MTU limitation whereas receipt of a (good) NACK is.

An ICMP PTB provides such a NACK, but can only be accepted if there
is operational assurance that it is authentic.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Thu, 23 Apr 2015 15:38:34 -0700

On 4/23/2015 2:41 PM, Templin, Fred L wrote:
...
> of a path MTU limitation whereas receipt of a (good) NACK is.

The reasons this is not true are addressed in detail in RFC 4821.

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 24 Apr 2015 15:11:17 +0000

Hi Joe,

> Sent: Thursday, April 23, 2015 3:39 PM

That is not correct; RFC4821 is about starting with a small size and
advancing to larger sizes based on successful probes for a given
(src, dst, flow-label)-tuple.

Re-read what I said again in both of my last two messages:

>> If your liveness detection is

> > of a path MTU limitation whereas receipt of a (good) NACK is.

Deterministic actions to shut down the tunnel can only be based on an
actual report of packet loss due to a size restriction, i.e., a good PTB
message that is not dropped due to filtering. Shutting down the tunnel
when it should stay up can make vast expanses of networks go
unreachable. Failure to shut down the tunnel when it should be down
can result in black hole loss of packets within a certain size range. None
of these concerns are problematic when operational assurances are
in place and/or when fragmentation is used.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 24 Apr 2015 15:46:02 +0000

To respond to myself, I will say that RFC6438 permits the tunnel ingress
to set the flow label in the way in which it sees fit (i.e., in the spirit of
RFC6437) which may or may not depend on the flow label values in
the actual payload packets.

So, if the ingress uses a module function that maps the tuples of the
packets it carries into just a few flow labels to be used in the
encapsulation header (say, 4 or 5 values) then the ingress can test all
of those paths and can shut down the tunnel if any one or more path
consistently fails to acknowledge a 1280 byte probe. The drawbacks
are that there may be many ECMP/LAG paths that go unused, and
the more distinct flow label values the ingress uses the more probes
will be necessary.

Still, why not keep the tunnel up by using fragmentation and alert
the network operations staff? The staff can then take action to either
find and fix the link(s) that have a too-small MTU or they can shut
down the tunnel in a managed fashion. Or, they can just let it
continue to fragment if fragmentation isn't a problem.

Thanks - Fred

> From: Int-area [mailto:[email-masked]] On Behalf Of Templin, Fred L
> Sent: Friday, April 24, 2015 8:11 AM
> To: Joe Touch; Ronald Bonica; [email-masked]

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Apr 2015 08:53:10 -0700

> On Apr 24, 2015, at 8:11 AM, Templin, Fred L <[email-masked]> wrote:
> 
> Hi Joe,
> 

It is about positive vs negative confirmation.

The rest are details.

> message that is not dropped due to filtering.

That is negative confirmation and we've already reviewed why that is not robust or safe. 

Joe

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 24 Apr 2015 16:33:59 +0000

Hi Joe,

Then, instead of saying "shut the tunnel down" you should say "shut the path
down". As in, test over all of the distinct flows the tunnel will spread traffic over
then shut down only those paths that consistently fail to pass a 1280 packet.

Only if all paths consistently fail to pass a 1280 should you consider shutting
the tunnel down. Or, instead, start fragmenting.

That is for when the original source can reasonably expect to receive PTBs
from the path to the tunnel ingress and the tunnel ingress can reasonably
expect to receive PTBs from the path to the tunnel ingress. When that is
not the case, the probe size should be 1500. 

Thanks - Fred

> Sent: Friday, April 24, 2015 8:53 AM
> To: Templin, Fred L
> Cc: Ronald Bonica; [email-masked]

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 24 Apr 2015 16:39:38 +0000

Hi Joe,

> Sent: Friday, April 24, 2015 9:34 AM
> To: Joe Touch

BTW, the AERO spec already does this kind of probing. Only, its
fallback position is to fragment instead of shut down paths that don't
work. The augmentation to the AERO spec therefore is to use RC6438
and test multiple flows and only shut down those that can't pass a probe.

Thanks - Fred

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Apr 2015 10:13:02 -0700

On 4/24/2015 9:33 AM, Templin, Fred L wrote:
> down". 

A tunnel is defined by its ability to transfer packets between ingress
and egress. How the tunnel decides what loss rate is sufficient is its
decision.

A tunnel has no way to know "path" at all; that concept doesn't exist in
the Internet except using LSR or SSR. What it does know is loss rate and
whether that correlates to packet properties - flow IDs, ports (if
visible), packet length, etc.

Any of those can be a consideration in deciding when to declare a tunnel
down. Short of sending every bit combination repeatedly and using
positive acknowledgement, an ingress can't know whether the packet
contents are causing loss.

However, since I don't hear anyone else on this list raising this red
herring, it doesn't seem critical to even explain in detail as to why it
need not be addressed in this doc.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Apr 2015 10:13:39 -0700

On 4/24/2015 9:39 AM, Templin, Fred L wrote:
...
> BTW, the AERO spec already does this kind of probing.

And yet we are *still* not talking about AERO.

Can we please get back to the subject of this doc?

---

From: "Templin, Fred L" <[email-masked]>
Date: Fri, 24 Apr 2015 18:30:28 +0000

HI Joe,

> Sent: Friday, April 24, 2015 10:14 AM
> To: Templin, Fred L

I have pointed you to AERO repeatedly and you seem to be reaching
some of the same conclusions that are already in AERO. You are saying
use a positive confirmation of a successful probe. AERO has been
saying that for a long time now. You are saying do something drastic
if probes fail - AERO says fragment and you say shut down. But, both
are just drastic measures that are taken only if probes fail.

>From this discussion, I have learned that AERO should really be doing
RFC6438-style flow label assignments, meaning that ECMP/LAG can
take the packets across different paths - all of which need to be
probed. But, that is really just an augmentation of what is already
in AERO. I have also learned that probing for 1280 is sufficient in
environments where it is operationally reasonable to expect that
PMTUD will work, whereas probing for 1500 is needed when there
is no such operational assurance.

> Can we please get back to the subject of this doc?

I have indulged your line of discussion in significant detail.
You owe me a critique of AERO.

Thanks - Fred

---

From: Ronald Bonica <[email-masked]>
Date: Fri, 24 Apr 2015 19:04:37 +0000

Are we still on topic?

                         Ron

> Sent: Friday, April 24, 2015 2:30 PM
> To: Joe Touch
> I have pointed you to AERO repeatedly and you seem to be reaching some of
> the same conclusions that are already in AERO. You are saying use a positive
> confirmation of a successful probe. AERO has been saying that for a long time
> now. You are saying do something drastic if probes fail - AERO says fragment
> and you say shut down. But, both are just drastic measures that are taken
> only if probes fail.
> 
> RFC6438-style flow label assignments, meaning that ECMP/LAG can take the
> packets across different paths - all of which need to be probed. But, that is
> really just an augmentation of what is already in AERO. I have also learned
> that probing for 1280 is sufficient in environments where it is operationally
> reasonable to expect that PMTUD will work, whereas probing for 1500 is
> needed when there is no such operational assurance.
>

---

From: Brian Haberman <[email-masked]>
Date: Fri, 24 Apr 2015 17:39:28 -0400

> Are we still on topic?

No.

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 27 Apr 2015 15:30:09 +0000

Hi Ron,

ECMP and LAG are likely to operate on the (src, dst, flow label)-tuple of encapsulated
packets. This means that MTU probing needs to be done on a per (src, dst, flow label)
basis where src is that of the tunnel ingress, dst is that of the tunnel egress, and flow
label is the value set by the tunnel ingress.

RFC6438 specifies the manners in which the tunnel ingress may set the flow label.
But, if the ingress sets the flow label to a uniformly-distributed 20 bit value, the
ingress would have to send 2^20 probes to ensure that there are no paths with a
too-small MTU in the ECMP/LAG arrangement which is far too many for practical
purposes.

It would be in keeping with RFC6438 to have your document recommend a more
conservative approach. For example, calculate a hash of the payload packet's IP
header fields but then map the hash into one of only N values where N is
something small like 4, 5, etc. Then, the ingress would only need to send a small
number of probes but there would still be some ECMP/LAG benefit gained from
using the flow label.

Alternatively, your document could set the flow label to a constant value (e.g., 0)
in which case only one (src, dst, flow label) value would need to be probed. But,
then ECMP/LAG path selection may be hampered by only ever seeing a single
constant value.

Thanks - Fred

---

From: "Templin, Fred L" <[email-masked]>
Date: Mon, 27 Apr 2015 15:33:57 +0000

Hi Brian,

> From: Int-area [mailto:[email-masked]] On Behalf Of Brian Haberman
> Sent: Friday, April 24, 2015 2:39 PM
> To: [email-masked]

I don't understand your comment. Are you saying that we should ignore
existing works that have undergone years of development?

Thanks - Fred

---

From: Brian Haberman <[email-masked]>
Date: Mon, 27 Apr 2015 12:27:05 -0400

> existing works that have undergone years of development?

Ron asked if this discussion was still on-topic.  Given that the thread
started with the comment:

"In Section 3.2, we put implementation details of the probing mechanism
out of scope."

and yet the entire thread has been about implementing a probing
mechanism, my view is that it is off-topic.

Regards,
Brian