Subject: PROXY protocol

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 23 Apr 2019 23:06:31 -0700

Various cloud load balancers appear to be adding the PROXY protocol which
is a single line of text inserted in front of routed packets.

https://blog.digitalocean.com/load-balancers-now-support-proxy-protocol/

I’m not deep into this issue, but it appears to also apply to binary
protocols.

I wonder if there is something QUIC should address here in order to avoid
having to parse text headers routed packets.

For reverse proxy HTTP there is the X-Forwarded family of headers but the
PROXY protocol is for cases where this does not apply.

In particular, an application might want to know the original IP and port
of a QUIC connection before it passed through a load balancer, for example
for IP whitelisting.

Mikkel

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Tue, 23 Apr 2019 23:07:05 -0700

some background

On 24 April 2019 at 08.06.31, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Mikkel

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Apr 2019 08:58:54 +0200

Hi Mikkel,

On Tue, Apr 23, 2019 at 11:06:31PM -0700, Mikkel Fahnøe Jørgensen wrote:
> I'm not deep into this issue, but it appears to also apply to binary
> protocols.
> 

Well, if you want the original, it's here :

     https://www.haproxy.org/download/2.0/doc/proxy-protocol.txt

As you see the text-only version was deprecated 5 years ago in favor of
a binary one which is smaller, easier to parse and can be extended. Users
want to have some TLS information transmitted with the header for example,
which v2 supports but not v1. Since it is designed to be prepended to any
payload, it would not be surprised if some QUIC implementations would
manage to adapt it there. You would probably not want it for each packet
though.

A few years ago I've been asked to write a draft on it but sadly it went
too low on my priority list so I never started this task :-/

I've found a list of products known for implementing it, I don't know if
it's up to date :

     https://www.haproxy.com/blog/haproxy/proxy-protocol/

Hoping this helps,
Willy

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Wed, 24 Apr 2019 09:02:34 +0200

hi Mikkel,

Thanks, I hadn't seen this.

Looking at the description here, ISTM that this protocol is solely applied between the front-end and the back-end within a provider, and seems pretty strongly to assume the FE-BE protocol is HTTP/1.1. 

Certainly someone will probably try to encap H2 or H3 in a plaintext protocol like this one, and if they do that, well, it seems a form of self-harm we're not well-placed to prevent.

Martin Duke's load balancers draft could be expanded to provide a QUIC-friendlier way to do the same thing (e.g. by encoding indexes to metadata available by a third service in the CID), or if we really want an encapsulation-based alternative (because it seems like PROXY will deploy widely around QUIC regardless), then we can define one that has less fiddly framing and the possibility to add encapsulation protection between the FE and the BE.

Cheers,

Brian

> On 24 Apr 2019, at 08:07, Mikkel Fahnøe Jørgensen <[email-masked]> wrote:
> 
> some background
> 
> On 24 April 2019 at 08.06.31, Mikkel Fahnøe Jørgensen ([email-masked]) wrote:
> 
>> Various cloud load balancers appear to be adding the PROXY protocol which is a single line of text inserted in front of routed packets.
>> 
>> I’m not deep into this issue, but it appears to also apply to binary protocols.
>> 
>> I wonder if there is something QUIC should address here in order to avoid having to parse text headers routed packets.
>> 
>> For reverse proxy HTTP there is the X-Forwarded family of headers but the PROXY protocol is for cases where this does not apply.
>> 
>> In particular, an application might want to know the original IP and port of a QUIC connection before it passed through a load balancer, for example for IP whitelisting.
>> 
>> 
>> Mikkel

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Apr 2019 09:11:57 +0200

Hi Brian,

On Wed, Apr 24, 2019 at 09:02:34AM +0200, Brian Trammell (IETF) wrote:
> Looking at the description here, ISTM that this protocol is solely applied
> between the front-end and the back-end within a provider, and seems pretty
> strongly to assume the FE-BE protocol is HTTP/1.1. 

No in fact it was designed for anything TCP-based in part as a simpler
alternative to Postfix's XCLIENT for dumb reverse proxies, and in order
to work with HTTP as a first real use case, but it is quite used with
SMTP and SQL. Please check the links I sent in the other e-mail in this
thread. I find it a bit sad that DigitalOcean didn't put the references
to the protocol's definition by the way.

> Certainly someone will probably try to encap H2 or H3 in a plaintext protocol
> like this one, and if they do that, well, it seems a form of self-harm we're
> not well-placed to prevent.

The v2 of the protocol dropped the plaintext approach. I'm not saying that
it's the best solution for QUIC either, maybe we'll need to encode stuff
that wasn't planned there or which would make the protocol inefficient for
this stuff.

Cheers,
Willy

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 24 Apr 2019 00:17:51 -0700

Perhaps QUIC should recommend load balancers that are QUIC aware to inject
PROXY headers only for initial packets and that they should use the binary
format
and initial packets should allow space for such headers in the initial
padding. Perhaps the 1200 byte requirement is already sufficient.

The problem with this approach is that the detection of initial packets is
version dependent and versions want to hide from the world, it seems.
So this appears to tied to invariants.

Tarreau covered what I was about to respond to Trammel on HTTP/1.1. As I
also said before, X-Forwarded-* is already widely used in that scenario,
and also for HTTPS TLS terminated at load balancer.

Mikkel

On 24 April 2019 at 08.58.57, Willy Tarreau ([email-masked]) wrote:

Hi Mikkel,

Hoping this helps,
Willy

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Wed, 24 Apr 2019 00:21:23 -0700

Google Cloud also uses the text version (version 1) of the PROXY protocol:

https://cloud.google.com/load-balancing/docs/tcp/setting-up-tcp#proxy-protocol

On 24 April 2019 at 09.17.51, Mikkel Fahnøe Jørgensen ([email-masked])
wrote:

Mikkel

Hi Mikkel,

Hoping this helps,
Willy

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 24 Apr 2019 10:35:17 +0100

I have just read QUIC transport, prior to doing a detailed read of the 
congestion control spec) and have a number of comments on the text and 
one more involved question on the decision on what to do with congestion 
control when there is a path change. I 'll send the latter issue later. 
(The editors can break these up into issues in the git if this seems 
useful to them).

Gorry

—————
The following comments/questions mainly relate to text:
—
NiT: Section 4:
“ It is necessary to limit the amount of data that a receiver could
    buffer,”
- Is this correct? (i.e. do we wish to limit what it can do or what it 
needs to buffer - is this a maximum limit or a minimal requirement? (I 
suspect the latter).
---
Question: Section 4:
  “The implementation SHOULD provide an interface to QUIC to tell it
    about its buffering limits so that there is not excessive buffering
    at multiple layers.”
- This seems an odd place to me to make a subtle new requirement on the 
IP-layer/interface: This lower-layer implementation interface 
requirement is buried in the spec. This seems that it could go 
unnoticed, and isn’t really a requirement of the protocol, but of the 
layer below.
—
Nit: Section 4:
   “that the sender receives an update before running out of flow control
    credit, even if one of the packets is lost.”
- the requirement could be misread as ONLY one packet is lost, whereas I 
think it means if one or more packets are lost.
—
Nit: Section 4:
  “That is, once a receiver
    advertises an offset, it MAY advertise a smaller offset, but this has
    no effect.”
- No effect on what? - presumably something at the sender?
—
Nit: Section 4:
   “Implementations might choose to increase limits as
    streams close to keep the number of streams available to peers
    roughly consistent.”
- could this be changed to “are closed” or something to avoid misreading 
of the word “close”.
—
NiT: Section 5:
“A Version Negotiation (Section 17.2.1) packet echoes the connection
    IDs selected by the client, both to ensure correct routing “:
- not strictly routing? Isn’t this “forwarding”. (The word routing is 
used again in the section and in section 7.2
—
Section 5:
NiT: “different IP or port”
- could you say:
“different IP address or transport port”
---
Section 5.1.1:
NiT: “The connection
    ID randomly selected by the client in the Initial packet and any
    connection ID provided by a Retry packet are not assigned sequence
    numbers unless a server opts to retain them as its initial connection
    ID.”
- could this sentence be simplified, it is hard to parse and long. (Or 
insert a comma)
—
Section 5.2:
NiT: “  or - for servers -“
- what is the meaning of the minus signs? does it imply “particularly” 
or “only”, I was unsure.
—
Section 5.2.1:
Nit: “Packets that don't match an existing connection are discarded.”
- can we remove “don’t” and explain what does match?
- Is this something like: “Packets with a Connection ID that does not 
match an existing
    connection are discarded.”
---
Section 5.2.1:
Nit: “Due to packet reordering or loss, clients might receive packets for a
    connection that are encrypted with a key it has not yet computed.”
“
- Is it possible to say something like: “Packet reordering or loss, 
might cause a client  to receive packets for a connection that were 
encrypted with a key that it has not yet computed.”
—
Section 8.1.2:
- The title seems a little odd to me? Future?
—
Section 9.3.3.:
?: “ An off-path attacker that can observe packets might forward copies of
    genuine packets to endpoints.  “
- This may make total sense to some people, but to me an “off path” 
observer has no access to the data on the path. Maybe someone could work 
this text in a way that doesn’t cause this confusion.
—
  Section 9.3.3.:
NiT: “If the copied packet arrives before
    the genuine packet, this will appear as a NAT rebinding.”
I think the pathology is the same as a NAT rebinding, but it isn’t one. 
The text could be clear that this appears as if there had been a NAT 
rebidding?
—
Section 9.3.3.:
?: “The attack is more reliable “
- The term reliable does have certain implications in transport 
documents, if this isn’t intended could this phrased as a higher 
likelihood of success or something like that?
- similarly “attack is reliably faster “
- explain how the path can be reliable, or please choose another word?
—
Issue: Section 10.4.2.:
?" “using a second iteration of a preimage-resistant function.
- Maybe I am alone, but I  did not understand what was intended by this, 
and where this was defined: is this I-D:
---
Section 19.1:
Question: “Padding can be used to
    increase an initial client packet to the minimum required size, or to
    provide protection against traffic analysis for protected packets.”
- I suggest this also can be used to perform path packet probing in 
DPLPMTUD, and the text should somehow be improved to permit this.
—
Section 9.4:
“The capacity available on the new path might not be the same as the
    old path.”
- I disagree in the words here, because I think the normal IETF 
transport assumption is much stronger than “might”, and the draft should 
say something like:
“The endpoint should not assume the capacity available on the new path 
isbe the same as the
    old path.”
—
Section 9.4:
NiT: “On confirming a peer's ownership of its new address, an endpoint
    SHOULD immediately reset the congestion controller and round-trip
    time estimator for the new path.”
- To be clear, does this reset require setting the CC to the initial 
value for a connection?
—
RFC2119:  “A sender can make exceptions for probe packets”
- Is that “MAY” - that would seem entirely reasonable.
—
“   While multiple paths might be used during connection migration, a
    single congestion control context and a single loss recovery context
    (as described in [QUIC-RECOVERY]) may be adequate. “
- I think the present I-D should not make these claims, because this 
seems like a CC decision. I actually do not understand this specific 
claim as currently presented.
—
Section 9.7:
Ref: “The IPv6 flow label SHOULD be a pseudo-random function of the source
    and destination addresses, source and destination UDP ports, and the
    destination CID. “
- Please cite RFC3976 (which contains details on how to set flow labels).—
—
Section 12.3:
NiT: “and acknowledged in packets which are also Initial packets”
- Change /which/ to /that/.
—
Section 14.2:
Ref: “An endpoint MUST NOT increase PMTU based on ICMP messages.”
- please add citation to  [RFC1191] [RFC8201], so the reader knows this 
is true of any protocol.
—
Section 21.2:
Question: “   Once a connection is established QUIC endpoints might 
accept some
    unauthenticated ICMP packets (see Section 14.2), but the use of these
    packets is extremely limited. “
- Why is this a good idea, please explain what these would be used for?
—
Section 21.5:
“   An adversarial sender might intentionally send fragments of stream
    data in order to cause disproportionate receive buffer memory
    commitment and/or creation of a large and inefficient data structure.”
- Is this memory commitment at the target? (I assume so, but it would be 
easier to read if it said this).
-

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 25 Apr 2019 09:55:05 +0900

Thanks for your detailed read, Gorry, will respond soon.

---

From: Martin Duke <[email-masked]>
Date: Fri, 26 Apr 2019 14:48:16 -0700

Injection implies messing with the packet stream, which will break
everything unless we do it on the Initial Packet or put it outside the UDP
payload (using the UDP option draft or something)

QUIC-LB, as a framework, in not particularly well-suited to per-connection
signaling. But if someone has a design I'm happy to consider integrating it
into my draft.