Subject: Questions about QUIC and Anycast

From: William Herrin <[email-masked]>
Date: Mon, 10 Dec 2018 16:18:24 -0800

Hi folks,

I've been reading draft-ietf-quic-transport-16 and seek some
clarification about how QUIC is intended to work with Anycast routing.
For anyone unfamiliar with Anycast routing, that's where multiple
servers located around the world are all configured with the same IP
address. The client's packet routes to the "nearest" of those servers
as selected by the layer-3 routing protocols.

>From what I've read there are two ways to robustly support anycast
with QUIC: A. The responding server can issue a unicast preferred
address. The client switches to the preferred address. B. The
responding server can pick a connection ID according to a
non-overlapping pattern for all servers within the anycast cluster.
The pattern can be used to route each client packet to the correct
server.

Question 1: Did I miss anything? Do the QUIC authors envision any
additional ways of supporting Anycast-routed servers?

Section 9.6 of the draft describes the server issuing a preferred
address. So once anycast finds the best server, use of the server's
unicast address lets the client stick with it. However, as I read it
the client and server must complete TLS negotiation before the client
can attempt a connection migration.

Question 2: Which packet contains the server preferred address? Is
this in the first response packet to the client? Can the client start
trying to use the preferred address with its second packet or does it
have to wait? Is it allowed to continue using the non-preferred
address and, if so, will it fail over to attempting the preferred
address if its communication fails in some way (no response or
rejection from the server at the non-preferred address)?

I ask this because Anycast routing has a packet spread case in which
per-packet load balancing, ECMP anomalies and other network funkiness
cause the client's second packet to reach a different server in the
cluster than the first. Both behaviors have been observed in the wild.
If the client can't reliably switch to the preferred address on packet
#2, the preferred address will fail to solve the Anycast use case.

Section 5 describes connection IDs. As I read it, the responding
server selects one or more integers whose presence in the client's
packet along with the client's selected connection IDs associate the
packet with an ongoing connection. It's possible to constrain the
server selection of those integers such that a simple algorithm
applied to the server's connection ID (such as an AND mask) uniquely
identifies which server in an Anycast cluster handles the associated
connection. A load balancer fronting the cluster or a packet rerouting
process in the Anycast QUIC implementation could then move the packet
to the server capable of handling it.

Question 3: What picks the connection ID? Does the API allow the
application software to influence the selection of connection ID in
any way?

Question 4: Is the connection ID inside or outside the encrypted
portion of the packet?

Question 5: Fragmentation needed / Packet Too Big messages originate
from an intermediate router that often has.a different return path
than the route taken by packets from the endpoint. How is QUIC
intended to deal with packet too big messages which arrive at a
different node in the Anycast cluster than the one which sent the
overlarge packet?

Thanks,
Bill Herrin

-- 
William Herrin ................ [email-masked]  [email-masked]
Dirtside Systems ......... Web: <http://www.dirtside.com/>

---

From: Martin Thomson <[email-masked]>
Date: Tue, 11 Dec 2018 11:40:55 +1100

Thanks for the questions Bill,

I'll provide some short answers.  Others might be able to expand a little.

> additional ways of supporting Anycast-routed servers?

The two methods you describe are what I understand to be the primary
mechanisms.  You might find that connection ID-based routing (your
second option) to be necessary though for the reasons discussed
below...

> Question 2: Which packet contains the server preferred address?

In practice, taking some of the less-used exchanges (Retry, Version
Negotiation, a TLS HelloRetryRequest) out of the picture, the server
will often include the value in the first flight of packets it sends.
However, that can be a couple of round trips away from the end of the
handshake, because the handshake is multiple packets and might need to
be repaired due to packet loss.

> Can the client start
> have to wait?

The client has to wait, because - until the handshake is complete - it
can't authenticated the preferred address value provided by the
server.  Also, validation of the path to that address likely takes
time.  That basically forces the server to use connection IDs to
ensure continuity.  A server might rely on the route being stable for
the duration of the handshake, though that would rely more on having
timely termination of the connection; keeping in mind that our best
mechanism, stateless reset, probably isn't viable at this point.

> Is it allowed to continue using the non-preferred

Yes, a client can choose to ignore the preferred address.  And yes, I
believe that the working group understands the implications of this.
If someone didn't and finds this distressing, yell.

> application software to influence the selection of connection ID in

The server picks connection IDs that the client puts in packets.  That
selection really has to be done in consultation with load balancers or
the folks managing routing configurations.

There's work on a protocol for managing this:
https://tools.ietf.org/html/draft-duke-quic-load-balancers-03

> Question 4: Is the connection ID inside or outside the encrypted

The value is unencrypted, but it is authenticated.  So it can be read,
but not changed.

> different node in the Anycast cluster than the one which sent the

The details of how to deal with PTB messages is in
https://quicwg.org/base-drafts/draft-ietf-quic-transport.html#packet-size

We realize that there is a limitation here: the PTB message won't
contain the connection ID selected by the server, so it can get
misrouted.  Kazuho suggested using a dummy packet with a long header
on PMTUD probes for that reason (those packets can be skipped and
contain both connection IDs), but I don't think that we've agreed to
formalize that technique, so there might be some privacy implications
we haven't considered.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 11 Dec 2018 01:39:07 +0000

> The details of how to deal with PTB messages is in https://quicwg.org/base-
> drafts/draft-ietf-quic-transport.html#packet-size
> 
> We realize that there is a limitation here: the PTB message won't contain the
> connection ID selected by the server, so it can get misrouted.  Kazuho
> suggested using a dummy packet with a long header on PMTUD probes for
> that reason (those packets can be skipped and contain both connection IDs),
> but I don't think that we've agreed to formalize that technique, so there
> might be some privacy implications we haven't considered.

Bill, note that this problem is not unique to Anycast (assuming that the ICMP PTB landed in the correct anycast cluster).  Any load balanced cluster that uses Connection IDs to route packets within the cluster will need to address this.

If the cluster is using a Consistent Hash algorithm to route the initial packets based on the 4-tuple of the packet, the ICMP can also be routed using the same hash.  Otherwise, one can keep state on the cluster's load balancer (active 4-tuples) or do rate-limited broadcast of the ICMP within the cluster.  None of the above (except for keeping centralized state) is a guarantee of success but does improve the chances.

If the Server's Connection ID is critical to you, you can use the long-header dummy packet coalesced with a short header packet trick for PMTU probe packets.  If a UDP packet containing these coalesced QUIC packets gets through, the long-header packet will be ignored but the short-header packet will be ACKed (just make sure to include at least one ACK-required frame -- a single PING frame will do the trick).  If that UDP packet elicits an ICMP PTB message, and that message contains more than 8 bytes of the original UDP datagram (the old IPv4 ICMP standard), the ICMP PTB message will contain your Server's Connection ID.

> -----Original Message-----
> From: Martin Thomson <[email-masked]>
> Sent: Monday, December 10, 2018 7:41 PM
> To: [email-masked]
> Cc: QUIC WG <[email-masked]>
> Subject: Re: Questions about QUIC and Anycast
> 
> mechanisms.  You might find that connection ID-based routing (your second
> option) to be necessary though for the reasons discussed below...
> 
> Negotiation, a TLS HelloRetryRequest) out of the picture, the server will
> often include the value in the first flight of packets it sends.
> handshake, because the handshake is multiple packets and might need to be
> repaired due to packet loss.
> 
> > Can the client start
> The client has to wait, because - until the handshake is complete - it can't
> authenticated the preferred address value provided by the server.  Also,
> validation of the path to that address likely takes time.  That basically forces
> the server to use connection IDs to ensure continuity.  A server might rely on
> the route being stable for the duration of the handshake, though that would
> rely more on having timely termination of the connection; keeping in mind
> that our best mechanism, stateless reset, probably isn't viable at this point.
> 
> > Is it allowed to continue using the non-preferred address and, if so,
> > will it fail over to attempting the preferred address if its
> > communication fails in some way (no response or rejection from the
> > server at the non-preferred address)?
> 
> Yes, a client can choose to ignore the preferred address.  And yes, I believe
> that the working group understands the implications of this.
> The server picks connection IDs that the client puts in packets.  That selection
> really has to be done in consultation with load balancers or the folks
> managing routing configurations.
> 
> The value is unencrypted, but it is authenticated.  So it can be read, but not
> changed.
> 
> The details of how to deal with PTB messages is in https://quicwg.org/base-
> drafts/draft-ietf-quic-transport.html#packet-size
> 
> We realize that there is a limitation here: the PTB message won't contain the
> connection ID selected by the server, so it can get misrouted.  Kazuho
> suggested using a dummy packet with a long header on PMTUD probes for
> that reason (those packets can be skipped and contain both connection IDs),
> but I don't think that we've agreed to formalize that technique, so there
> might be some privacy implications we haven't considered.

---

From: William Herrin <[email-masked]>
Date: Tue, 18 Dec 2018 10:47:05 -0800

Thanks Martin, Igor:

I appreciate your insight. This leads me to a couple of additional questions.

> the folks managing routing configurations.

How does short header versus long header impact this? Can the server
instruct the client to use the full destination connection ID in both
headers so that the full connection ID is available to a middlebox in
every packet?

> be repaired due to packet loss.

Hmm. I'm curious what the reasoning is here. TLS would have to satisfy
the same cryptography regardless of which address is used, would it
not? Presumably the client could return to the original address upon
failed communication with the preferred address, including a failure
to negotiate TLS. How does requiring the TLS negotiation to complete
before attempting the preferred address end up enhancing security in
the bigger picture?

Does the preferred address have another use case besides load
balancing? The load balancing process could be stateless if it could
rely on the client shifting to the preferred address on packet #2. It
seems to me that making the load balancing process stateless,
especially in the anycasted CDN case, would be a big win if it could
be done securely.

Regards,
Bill Herrin

---

From: Martin Thomson <[email-masked]>
Date: Wed, 19 Dec 2018 08:24:18 +1100

> headers so that the full connection ID is available to a middlebox in

The destination connection ID included, in full, in every packet.  The
point of that is to ensure that that any middlebox that cooperates
with the server can use it.  (I say cooperates because the value can
change over time.)

> before attempting the preferred address end up enhancing security in

Yes, TLS can complete using multiple paths - it doesn't care.
However, we treat it like a black box, only releasing information once
it has been authenticated.  So while the preferred address might
appear in a packet, it won't be available until the handshake
completes.  Best case, that's immediately, because the server's
handshake fits into our amplification limit and there is no packet
loss.  However, that's not guaranteed.

We also simplify the protocol analysis by limiting the handshake to
the same address tuple.  Migration to a different address comes with a
bunch of other problems that we decided weren't worth grappling with,
like ensuring that the other endpoint genuinely can receive packets at
that address, and so forth.  Perhaps that is something that might be
improved with some work, but work is the scarce resource right now and
we're spending all we can on completing the other bits of the protocol
that we know need to be fixed.

> especially in the anycasted CDN case, would be a big win if it could

Yep, that "done securely" qualifier is the real catch.  See above.