Subject: Re: Alt-Svc related Chromium bug report (proxy related)

From: Mark Nottingham <[email-masked]>
Date: Wed, 2 Apr 2014 11:32:11 +1100

I think that phrase probably needs to at least be modified to

"Proxies MUST NOT change or append Alt-Svc values."

On 1 Apr 2014, at 2:51 pm, Martin Thomson <[email-masked]> wrote:

> On 31 March 2014 20:08, Amos Jeffries <[email-masked]> wrote:
>> This is not "mis-advertising". The origin server producing the response
>> eminently *does* support that service.
>> 
>> I then draw your attention to draft-ietf-httpbis-alt-svc-00 section 3...
>> 
>> " Intermediaries MUST NOT change or append Alt-Svc values. "
>> 
>> There is no distinction regarding intermediary type(s), nor exemption
>> clauses. CDN is an intermediary just as much as any other.
> 
> 
> That would be fine, except for the fact that - in this case - it seems
> likely that the CDN is acting as an origin server that serves a copy
> of the content, not an intermediary.  The URI includes the domain name
> of the CDN.  Will, amirite?
> 

--
Mark Nottingham   http://www.mnot.net/

---

From: Erik Nygren <[email-masked]>
Date: Thu, 17 Apr 2014 18:22:01 -0400

> "Proxies MUST NOT change or append Alt-Svc values."

Agreed.  I guess CDNs will need to remove Alt-Svc values that may be
returned by an origin if they can't handle the upgrade.  This is a little
unfortunate from a compatibility perspective.

It may also be worth us thinking through the Alt-Svc scenarios in the proxy
world a little more carefully, perhaps with some concrete examples of flows
to make sure we're on the same page.  For example, the upgrade case is
fairly straight-forward:

1) Upgrade to HTTP/2-over-TLS through a proxy:

        - client makes HTTP request through proxy (GET
http://www.example.com/)
        - server returns AltSvc to the client via the proxy (Alt-Svc:
http2=443)
        - client makes a HTTP connect request through the proxy (CONNECT
www.example.com:443)

However, some of the other cases are less clear to me:

2) What happens for clients talking HTTP/2 to a proxy with the proxy
talking HTTP/2 to the origin?  Would an ALTSVC frame referencing a
different server and using TLS get passed through meaning that the client
needs to now CONNECT through to do TLS?

3) What happens with clients talking HTTP/1 to a proxy when the proxy is
talking HTTP/2 to the origin and gets an ALTSVC frame pointing to a
different host?  Presumably the proxy needs to drop it?

Are there other similar scenarios we need to work through?

      Erik

---

From: Martin Thomson <[email-masked]>
Date: Thu, 17 Apr 2014 15:32:33 -0700

On 17 April 2014 15:22, Erik Nygren <[email-masked]> wrote:
> 2) What happens for clients talking HTTP/2 to a proxy with the proxy talking
> HTTP/2 to the origin?  Would an ALTSVC frame referencing a different server
> and using TLS get passed through meaning that the client needs to now
> CONNECT through to do TLS?

ALTSVC frames need to be consumed, not passed through.  If the proxy
did that, then it's responsible for the consequences.  But I don't see
how this would fail.

> different host?  Presumably the proxy needs to drop it?

Same answer as above.

The real concern here is the Alt-Svc header field, which will (likely)
be passed the whole way through.  CDNs should probably remove the
header field and maybe add their own, since they either act as a new
origin server or gateway.  Proxies might want to drop the header field
in the case where they aren't able to (or don't want to) support the
alternative.

---

From: Mark Nottingham <[email-masked]>
Date: Sun, 20 Apr 2014 10:06:38 +1000

On 18 Apr 2014, at 8:22 am, Erik Nygren <[email-masked]> wrote:

> Agreed.  I guess CDNs will need to remove Alt-Svc values that may be returned by an origin if they can't handle the upgrade.  This is a little unfortunate from a compatibility perspective.

Keep in mind that CDNs are *not* proxies as per the HTTP spec; they're gateways that happen to talk HTTP on the back end.

> It may also be worth us thinking through the Alt-Svc scenarios in the proxy world a little more carefully, perhaps with some concrete examples of flows to make sure we're on the same page.  For example, the upgrade case is fairly straight-forward:
> 
>         - client makes HTTP request through proxy (GET http://www.example.com/)
>         - server returns AltSvc to the client via the proxy (Alt-Svc: http2=443)
>         - client makes a HTTP connect request through the proxy (CONNECT www.example.com:443)
> 
> 
> 2) What happens for clients talking HTTP/2 to a proxy with the proxy talking HTTP/2 to the origin?  Would an ALTSVC frame referencing a different server and using TLS get passed through meaning that the client needs to now CONNECT through to do TLS?

I think the proxy would consume and potentially use the ALTSVC frame, since it's the one managing the connection to the server. Remember, ALTSVC doesn't change the origin.

> 3) What happens with clients talking HTTP/1 to a proxy when the proxy is talking HTTP/2 to the origin and gets an ALTSVC frame pointing to a different host?  Presumably the proxy needs to drop it?

Think so.

---

From: Amos Jeffries <[email-masked]>
Date: Sun, 20 Apr 2014 18:36:54 +1200

On 20/04/2014 12:06 p.m., Mark Nottingham wrote:
> 
> On 18 Apr 2014, at 8:22 am, Erik Nygren wrote:
> 

The draft wording however is not limited to "proxies". Which was my
initial report of there being a problem.

Alt-Svc draft (editors copy from today):

3. The Alt-Svc HTTP Header Field:
"
  Intermediaries MUST NOT change or append Alt-Svc field values.
"

In HTTP/2 draft (editors copy from today):

2.2 Conventions and Terminology:
"
 intermediary:
    A "proxy", "gateway" or other intermediary as defined in Section
    2.3 of [HTTP-p1].
"

draft-ietf-httpbis-p1-messaging-26:

2.3. Intermediaries
"
  A "gateway" (a.k.a., "reverse proxy") is an intermediary that acts as
   an origin server for the outbound connection,
"

Therefore a CDN being an intermediary *MUST NOT* change the Alt-Svc
header, even to fix known problems. Despite being a gateway.

>> 3) What happens with clients talking HTTP/1 to a proxy when the proxy is talking HTTP/2 to the origin and gets an ALTSVC frame pointing to a different host?  Presumably the proxy needs to drop it?

Or not. Up to the proxy as recipient of the hop-by-hop frame.

>> Are there other similar scenarios we need to work through?

ALTSVC frame from a proxy perspective is okay since it is defined with
hop-by-hop semantics.

The problem is interaction of the Alt-Svc HTTP/1 header since it is
end-to-end but places semantics of:

case A)
  bypassing an entire chain of proxies by diverting the client to an
entire alternate path.

case B)
 breaking connectivity, by informing the client about an Alt-Svc which
is impossibel for it to contact.

These could easily be encountered at an ISP where port 80 is firewalled
and HTTP delivered through an intermediary instead. Or at a CDN such as
teh one described where the gateway server offered diffrent services
than the internel origin node.

IMO the Alt-Svc header semantic needs to be redefined such that any
recipient can "terminate" it and contact the alternative service itself.
 Doing this will avoid problems from case A with a simple software
upgrade by any of the components in the chain. But also will work
through legacy intermediaries that dont understand it yet.

The blanket "MUST NOT" on changes to its values only works if the
intermediary which are in gateway situations are empowered to perform
the action the header indicated and/or drop it completely from the traffic.

Additional to that a gateway being in the position of an origin server
should be empowered to send its own Alt-Svc header. For the reason that
its very presence may be due to the backend server being obsolete
non-upgradable software.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 21 Apr 2014 13:08:49 +1000

> initial report of there being a problem.

Yes; we've discussed this a few times on-list, but it never seemed to have made it onto the issues list.

I've created <https://github.com/http2/http2-spec/issues/462>.

Any thoughts about the suggestion there?

Cheers,

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 21 Apr 2014 16:42:34 +1200

> Any thoughts about the suggestion there?

I think we should go with the SHOULD NOT. The cases that need to be
enumerated IMO are that the proxy either;
 a) drop the header completely (consumes with decision not to use the
alt service)
 b) uses the alt-service (as if it were the client)
 c) passes it on un-touched (the case being forced by the MUST NOT)

I agree that a proxy has no place adding/changing field values the
client gets. But consuming the header is a different proposition which
has several uses already mentioned.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 21 Apr 2014 08:32:20 +0200

Hi Amos,

> has several uses already mentioned.

Well, we've had this discussion many times in the past on the list about
passing addresses/ports/protocols and whatever is routing-related from
servers to clients, which *never* works if intermediaries cannot change
them. It's always the same thing, just like with redirects where gateways
("reverse-proxies") are filled with many rules to rewrite them.

Such information can only be hop-by-hop by definition since the entity
emitting this information only knows the values to emit for the place
it is connected to (eg: private IP address, internal port, etc). And
gateways need to convert this to the new values.

Let's consider this common example : company A runs a web server in a DMZ
("safe zone") behind an apache reverse proxy acting as an SSL-offloader
and cache running in a different DMZ ("unsafe zone). The same service is
accessible from the internal LAN directly and from a partner company B
over a VPN. These DMZ and VPN are provided by one or multiple firewalls,
some of them performing NAT. This is really *common*. Many of my customers
have services built that way.

              Internet. Pub IP=1.1.1.1:80,443
                   |
                   |       Pub IP=192.168.1.1:80,443
  Company B     +-----+     +--------+
  over VPN -----| FW1 |-----| rproxy |
                +-----+     +--------+
  IP=10.0.0.1:8080 |
                   |
                +-----+     +--------+
                | FW2 |-----| server | Priv IP=192.168.2.1:8080
                +-----+     +--------+
                   |
             ------+---------  Lan with local users.
                               IP=192.168.2.1:8080

So what do we have here ?

  - a server which only knows it's running on 192.168.2.1:8080
  - local users which access it on that IP:port address
  - a reverse proxy configured to access it to that IP address:port
    and exposing it on 192.168.1.1:80 and in https at 192.168.1.1:443.
    This reverse proxy knows its public-facing IP address
  - a firewall which nats the public address 1.1.1.1 to the reverse
    proxy's 192.168.1.1 and keeps the port
  - the same firewall which NATs accesses over the VPN from parner
    company B to 10.0.0.1:8080 to 192.168.2.1:8080 (direct server access).

In such a common scenario, it is clear that it is not possible for the
server to provide a correct information about its location. Having the
public and private accesses (the most common cases) already makes this
impossible without letting each gateway adjust the values. Here it would
easily work if the server just provided http://192.168.2.1:8080 as info
and the front reverse proxy would simply adjust this to become 1.1.1.1:80
or https://1.1.1.1:443 depending on what's expected.

The company B's access over VPN is more annoying because there is no
gateway there. That means that any such information provided to this
place is wrong and cannot be used or must be modified there. It also
discloses internal information which is not always acceptable in some
places.

So at best, information provided in Alt-Svc can only be hop-by-hop and
can only be emitted/modified/filtered by gateways and must not be emitted
by origin servers if they're not authoritative. And quite frankly these
days, origin servers that can claim to be authoritative about their public
address/port/protocol become extremely rare.

Best regards,
Willy

---

From: Martin Thomson <[email-masked]>
Date: Mon, 21 Apr 2014 11:12:29 -0700

> initial report of there being a problem.

Mark already suggested that the wording needed to be changed from
"intermediary" to "proxy".  I think that suffices.

> is impossibel for it to contact.

Neither is an issue because Alt-Svc expressly states that:

"The client is not required to block requests; the origin's connection
can be used until the alternative connection is established."

That means that if the new path doesn't work (network policy, broken
header, or for any reason...), then the existing connection is good.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 Apr 2014 11:12:42 +1000

> "intermediary" to "proxy".  I think that suffices.

It might also be helpful to note that a client with a configured proxy isn't expected to use alternative services (no matter how discovered).

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 22 Apr 2014 18:39:54 -0700

​​

On Tue, Apr 22, 2014 at 6:12 PM, Mark Nottingham <[email-masked]> wrote:

> It might also be helpful to note that a client with a configured proxy
> isn't expected to use alternative services (no matter how discovered).

​That doesn't sound right to me. Couldn't the client use a CONNECT tunnel
to reach the alternate service? If the alternate service is HTTP/2, and the
main service is HTTP/1, then there could be a big performance win from
using the alternate service through the proxy, wouldn't there?​

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 Apr 2014 11:42:24 +1000

On 23 Apr 2014, at 11:39 am, Ryan Hamilton <[email-masked]> wrote:

> ​That doesn't sound right to me. Couldn't the client use a CONNECT tunnel to reach the alternate service? If the alternate service is HTTP/2, and the main service is HTTP/1, then there could be a big performance win from using the alternate service through the proxy, wouldn't there?​

... isn't expected to use alternative services (no matter how discovered) to establish new connections; however, existing connections through the proxy might be affected (e.g., CONNECT tunnels).

;)

---

From: Ryan Hamilton <[email-masked]>
Date: Tue, 22 Apr 2014 20:00:26 -0700

​​

On Tue, Apr 22, 2014 at 6:42 PM, Mark Nottingham <[email-masked]> wrote:

> > ​That doesn't sound right to me. Couldn't the client use a CONNECT
> tunnel to reach the alternate service? If the alternate service is HTTP/2,
> and the main service is HTTP/1, then there could be a big performance win
> from using the alternate service through the proxy, wouldn't there?​
>
> ... isn't expected to use alternative services (no matter how discovered)
> to establish new connections; however, existing connections through the
> proxy might be affected (e.g., CONNECT tunnels).
>

​I'm not following, which suggests that I'm missing something. :> Imagine
that a client is visiting http://server/ via the proxy, which serves up an
alt-service header suggesting server:443. The client would then initiate a
new CONNECT tunnel through the proxy to server:443.

Was that what you meant?​

​Cheers,

Ryan​

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 23 Apr 2014 14:10:19 +1000

On 23 Apr 2014, at 1:00 pm, Ryan Hamilton <[email-masked]> wrote:

> Imagine that a client is visiting http://server/ via the proxy, which serves up an alt-service header suggesting server:443. The client would then initiate a new CONNECT tunnel through the proxy to server:443.

Ah, IC.

Perhaps we should just say that Alt-Svc doesn't affect whether a client uses a proxy, although it might affect how the client contacts an origin *through* its proxy/proxies.

Cheers,

---

From: Ryan Hamilton <[email-masked]>
Date: Wed, 23 Apr 2014 11:41:52 -0700

On Tuesday, April 22, 2014, Mark Nottingham <[email-masked]> wrote:

>
> On 23 Apr 2014, at 1:00 pm, Ryan Hamilton <[email-masked] <javascript:;>>
> wrote:
>
> > Imagine that a client is visiting http://server/ via the proxy, which
> serves up an alt-service header suggesting server:443. The client would
> then initiate a new CONNECT tunnel through the proxy to server:443.
>
> Ah, IC.
>
> Perhaps we should just say that Alt-Svc doesn't affect whether a client
> uses a proxy, although it might affect how the client contacts an origin
> *through* its proxy/proxies.

Sounds good to me!

Cheers,

Ryan