Subject: Re: [v6ops] I-D Action: draft-jaeggli-v6ops-pmtud-ecmp-problem-00.txt

From: Brian E Carpenter <[email-masked]>
Date: Wed, 04 Jun 2014 13:18:38 +1200

Hi,

>    A problem common to the approach of distribution through hashing is
>    its impact on path MTU discovery.  An ICMPv6 type 2 PTB message
>    generated on the path between a client and an ECMP load balanced
>    server will have the anycast address as the destination and will be
>    statelessly load balanced to one of the anycast servers. 

This may seem picky, but I think the reader's brain will run
more smoothly if it's explicit that it's a PTB triggered by
a packet *from* the server and therefore directed *to* the server.
"on the path" doesn't quite contain that information.

>             Because of this, the results of
>    the ICMPv6 ECMP hash do not match that of the corresponding TCP or
>    UDP ECMP hash.

Again picky, but if there are (say) 2 paths, there's a 50% chance
that it gets the right path, etc. So "might not match" is probably
better.

General comment: I'm not sure what is specific to ECMP
about this problem. We identified it as a general (but out of
scope) problem in RFC 7098. We did include this comment there:

 o  Note that correct handling of ICMPv6 for Path MTU Discovery
    requires the layer 3/4 balancer to keep state for the client
    source address, independently of either the port numbers or the
    flow label.

because, indeed, the PMTU is a function of the address pair only.

    Brian

---

From: joel jaeggli <[email-masked]>
Date: Tue, 03 Jun 2014 21:48:34 -0700

> "on the path" doesn't quite contain that information.

I think that's a reasonable observation and fairly straight forward to
adjust. To my mind the destination address being the anycast address
does communicate the direction of this ptb message.

> that it gets the right path, etc. So "might not match" is probably

If there are 64 hash buckets the probability of ending up in the same
one is 1.5% if there are 255 it's ~.4%. so may not is potentially a near
certainty. It's entirely possible to have more hash buckets than
next-hops in order to facilitate load balancing without rehashing when
adding and removing devices in which case the distribution may vary, and
on those cases a packet hashed to the wrong bucket may well arrive on
the correct server/load-balancer.

> scope) problem in RFC 7098. We did include this comment there:

So it may be out of scope for your rfc, I'm pretty sure that doesn 't
make the problem go away. I've been dinking around with large scale
load-balancing of this flavor for some time, more than a decade in on
form or another. We found it to be a commercial necessity to address the
problem of how to make PTB work as part of deploying consumer facing
internet applications services. it's not unique to ICMP, e.g. my
interest in the fragdrop discussion is due to similar problems.

>     source address, independently of either the port numbers or the

So there are two problems with this assertion.

1. state sharing is presently impossible at that scale I'm operating at.
so any solution that presumes it across an entire population of devices
isn't going to work. state sharing between pairs of devices in otherwise
stateless clusters (which describes the internal architecture of a
number of high-end firewalls and load-balancer products doesn't solve
this either).

2. the ptb packet doesn't have the parts of the flow associated within
the ip and icmp header so the flow that is was associated with cannot be
reconstructed by a pure l3/l4 device stateful or otherwise, it can as
noted in the draft be derived by inspecting the icmp payload for the
IP/TCP/UDP header of outgoing packet which is pretty deep packet
inspection (normally done by the end system). Finding the ICMP header is
not the same as being able to parse the payload. In any event we were
using rather high-end but but otherwise normal router silicon, so we get
to do this the same as anyone else who fronts a load-balancer or server
tier containing ecmp nexhops with a router...

> because, indeed, the PMTU is a function of the address pair only.

The ptb packet has the source address of the device that emitted it.

In any event, our goal isn't really to boil the ocean, with respect to
parsing the packet or deriving correct host, if we could get close
enough, it was to not break PMTUD, and therefore customers in a service
that supported millions of  users.

>     Brian
> 
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 05 Jun 2014 08:32:06 +1200

> the correct server/load-balancer.

My logic is the number of hash buckets doesn't matter - it's the number
of paths, because if there are N paths, ~1/N of the traffic will go
along each path regardless of the number of hash buckets.

> interest in the fragdrop discussion is due to similar problems.

Fully agreed. It just wasn't a problem we could ameliorate using the
flow label, so it didn't belong in 7098.

> number of high-end firewalls and load-balancer products doesn't solve

Agreed. Our comment was not intended to suggest that it was practical
to synchronise state in that way.

> tier containing ecmp nexhops with a router...

Agreed. Again, we didn't mean to imply that line-speed dissection
of ICMP was practical; only that it seems to be needed to send
PTBs where they need to go.

> The ptb packet has the source address of the device that emitted it.

Yes, of course. That's why you have to dissect the packet.

(One fanciful idea that the authors of 7098 discussed was to
specify that the outgoing flow label value should be reflected
back in the ICMPv6 packet, but that didn't seem like a
deployable idea.)

> that supported millions of  users.

Fully understood.

   Brian

---

From: Nick Hilliard <[email-masked]>
Date: Wed, 04 Jun 2014 22:49:01 +0100

> along each path regardless of the number of hash buckets.

on a matter of nits, the number of hash buckets matters greatly if it's too
small, and there is lots of hardware deployed out on the internet which
suffers from not enough hash buckets.

Nick

---

From: Mark Andrews <[email-masked]>
Date: Thu, 05 Jun 2014 08:19:35 +1000

Setting IPV6_USE_MIN_MTU to 1 should be a effective fix at the
application layer for IPv6.  That said FreeBSD doesn't do the right
thing by taking IPV6_USE_MIN_MTU into consideration when performing
MSS negotiations.

http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/173444

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: joel jaeggli <[email-masked]>
Date: Wed, 04 Jun 2014 16:40:50 -0700

> along each path regardless of the number of hash buckets.

yeah and I had 64 paths in one deployment so...

> flow label, so it didn't belong in 7098.

If I were hashing on the flow label (which frankly I wouldn't mind
doing) I'd still have to find it. it can probably can be found  fairly
easily since theoretically since the packet in the payload originated
with me, and I emitted no fragments, and used no extension headers the
payload in the icmp  after offset 32 should be the ip header, which is
conveniently fixed. and the upper layer header. doing this in silicon
depends on how much of the packet I can sluice off with the header (and
vendor implmentation of course).

> back in the ICMPv6 packet, but that didn't seem like a

In practice  even if many of them weren't zero, 20 bits isn't enough for
the flow label by itself to positively identify the flow in my case.
notwitstanding the lack of state sharing.

if it used the destination adress and the flow label, that would
actually do it, but then it's spoofing.