Subject: Fwd: New Version Notification for draft-cdn-loop-prevention-00.txt

From: Mark Nottingham <[email-masked]>
Date: Mon, 2 Jul 2018 17:00:05 +1000

(Co-author hat on)

For interest / discussion. This is a proposal for a minimal mechanism to avoid loop attacks and misconfigurations against CDNs. Feedback appreciated.

Cheers,

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: New Version Notification for draft-cdn-loop-prevention-00.txt
> Date: 27 June 2018 at 2:12:46 pm AEST
> To: "Stephen Ludin" <[email-masked]>, "Mark Nottingham" <[email-masked]>, "Nick Sullivan" <[email-masked]>
> 
> 
> A new version of I-D, draft-cdn-loop-prevention-00.txt
> has been successfully submitted by Mark Nottingham and posted to the
> IETF repository.
> 
> Name:		draft-cdn-loop-prevention
> Revision:	00
> Title:		CDN Loop Prevention
> Document date:	2018-06-27
> Group:		Individual Submission
> Pages:		5
> URL:            https://www.ietf.org/internet-drafts/draft-cdn-loop-prevention-00.txt
> Status:         https://datatracker.ietf.org/doc/draft-cdn-loop-prevention/
> Htmlized:       https://tools.ietf.org/html/draft-cdn-loop-prevention-00
> Htmlized:       https://datatracker.ietf.org/doc/html/draft-cdn-loop-prevention
> 
> 
> Abstract:
>   This specification defines the CDN-Loop request header field for
>   HTTP.
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Thomas Peterson <[email-masked]>
Date: Mon, 02 Jul 2018 09:10:12 +0100

Thanks Mark.

Should this specification be specifically named and emphasised on CDNs exclusively? I would think that many web services with more than 1 tier of load balancing/traffic routing have also shot themselves in the foot at least once with traffic looping.

From: Mark Nottingham <[email-masked]>
Date: Monday, 2 July 2018 at 08:04
To: HTTP Working Group <[email-masked]>
Cc: "Ludin, Stephen" <[email-masked]>, Nick Sullivan <[email-masked]>
Subject: Fwd: New Version Notification for draft-cdn-loop-prevention-00.txt
Resent-From: <[email-masked]>
Resent-Date: Mon, 02 Jul 2018 07:00:48 +0000

(Co-author hat on)

Cheers,

Begin forwarded message:

Abstract:

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 2 Jul 2018 18:19:11 +1000

Hi Thomas,

That's a good question. The focus is on CDNs here because it's a problem and a solution that's pretty specific to them.

In other words, it's easy to prevent loops when you control all of the layers (or can closely coordinate with them); you just define a loop prevention mechanism. AFAIK most CDNs do this internally already. 

The problem is when you a) have multiple uncoordinated layers (like CDNs), and b) they can be configured by a potential attacker, or misconfigured by a well-meaning customer who's using multiple CDNs. 

In that case, the CDNs need to agree on a mechanism that can prevent loops amongst them; once we settle on it, we can prevent customer configuration from modifying that header field, and prevent any attack / misconfiguration.

In theory it could be used by others. However, given the misuse of Vary that the draft cites, I'm reluctant to make its use too broad; if sites start doing other things with it, we won't be able to use it in CDNs, and we'll have to create Yet Another Header.

Does that make sense?

Cheers,

> On 2 Jul 2018, at 6:10 pm, Thomas Peterson <[email-masked]> wrote:
> 
> Thanks Mark.
>

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 2 Jul 2018 18:35:31 +1000

Yes :)

> On 2 Jul 2018, at 6:35 pm, Thomas Peterson <[email-masked]> wrote:
> 
> You mean Via and not Vary, right?
> 
> Regardless, it makes sense.
> 
> Thank you.
> 
> -----Original Message-----
> Date: Monday, 2 July 2018 at 09:19
> To: Thomas Peterson <[email-masked]>
> Cc: HTTP Working Group <[email-masked]>, "Ludin, Stephen" <[email-masked]>, Nick Sullivan <[email-masked]>
> Subject: Re: New Version Notification for draft-cdn-loop-prevention-00.txt
> 
>    Hi Thomas,
>

---

From: Thomas Peterson <[email-masked]>
Date: Mon, 02 Jul 2018 09:35:01 +0100

Thank you.

-----Original Message-----
From: Mark Nottingham <[email-masked]>
Date: Monday, 2 July 2018 at 09:19
To: Thomas Peterson <[email-masked]>
Cc: HTTP Working Group <[email-masked]>, "Ludin, Stephen" <[email-masked]>, Nick Sullivan <[email-masked]>
Subject: Re: New Version Notification for draft-cdn-loop-prevention-00.txt

    Hi Thomas,

    

    
    Does that make sense?
    
    Cheers,

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 02 Jul 2018 08:47:38 +0000

--------
In message <[email-masked]>, Mark Nottingham wri
tes:

>For interest / discussion. This is a proposal for a minimal mechanism to 
>avoid loop attacks and misconfigurations against CDNs. Feedback 
>appreciated.

This problem is as old as packet networks, and why IP have the TTL field.

I think it would be better and more robust to define a "max-hops"
header with a single numerical field, which each (conforming) proxy
decrements and if it becomes zero, 50x error is returned.

CDNs can create a max-hops header if there is none, and even if
nobody else implements the max-hops header, it will eventually count
down to zero if there is a loop.

In difference from the proposed draft, this doesn't reveal the
architecture to the client.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 2 Jul 2018 20:48:56 +1200

> In theory it could be used by others. However, given the misuse of Vary that the draft cites, I'm reluctant to make its use too broad; if sites start doing other things with it, we won't be able to…

s/Vary/Via/

But no, the rational for adding a new header does not make much sense
yet. There is already Forwarded header created for this same use-case
(amongst others) overlapping with Via.

And why not push to get the problematic servers fixed instead of
polluting HTTP header space with more features everybody with middleware
has to support?

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 2 Jul 2018 21:56:32 +1200

> architecture to the client.

Max-Forwards goes almost there. Would just need to revise this part of
RFC 7231 section 5.1.2 to make all recipients SHOULD decrement instead:

"
 A recipient MAY ignore a Max-Forwards header field received with any
 other request methods.
"

... and nothing is preventing us middleware and CND people decrementing
on any method right now.

Amos

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Mon, 02 Jul 2018 10:03:13 +0000

--------
In message <[email-masked]>, Amos Jeffries 

>RFC 7231 section 5.1.2 to make all recipients SHOULD decrement instead:

Well 4.3.7 prevents CDN's from adding it:

   A proxy MUST NOT generate a Max-Forwards header
   field while forwarding a request unless that request was received
   with a Max-Forwards field.

---

From: Amos Jeffries <[email-masked]>
Date: Mon, 2 Jul 2018 22:25:02 +1200

>    with a Max-Forwards field.

That is where CDN / reverse-proxy acting as origin servers exempts them
from general proxy rules.

Amos

---

From: Patrick McManus <[email-masked]>
Date: Mon, 02 Jul 2018 16:29:24 +0000 (UTC)

Hi, thanks for the draft! Some comments as an individual ...

* I agree with the comment that, as proposed, this isn't really a CDN
specific solution. Its a more opaque version of via.. so it makes sense to
me to construct it independent of CDNs and instead focus it on any element
that wants to participate in loop prevention. CDNs serve as the primary
illustrator.

* I would be interested in more text on why Via failed and therefore this
should succeed.. you mention that via is used in practice as a control
signal incompatibly enumerating intermediaries.. I agree, but I also think
via in practice reveals information people don't want revealed.. is
CDN-loop perhaps more robust to this because the token is opaque (and
perhaps not consistent between exchanges)?

* the draft focuses on creation of the cdn-loop request header but has
nothing to say on who/how/when that is policed.

* "the token defines the CDN as a whole." feels like it needs more
guidance.. is it even the right advice? Is it more about the scope of the
token? are there use cases where cascading cdns make sense that might
involve CDN repeats without forming a loop?

* it would really help the wg to understand the scope of the problem a
little more.. "not unknown for CDNs to be configured in a loop" makes me
wonder if the authors believe this is a serious problem (though I suspect
they just didn't want to over-assert).. but given that the document
basically wants to restandardize something that has been standardized
(Via), a little discussion of the operational imperative makes sense. (this
is not necessarily something that would have to be in the doc)

-P

On Mon, Jul 2, 2018 at 3:00 AM, Mark Nottingham <[email-masked]> wrote:

> avoid loop attacks and misconfigurations against CDNs. Feedback appreciated.
>
> Cheers,
>
>
> Begin forwarded message:
>
> *From: [email-masked]
> *Subject: **New Version Notification for draft-cdn-loop-prevention-00.txt*
> *Date: *27 June 2018 at 2:12:46 pm AEST
> *To: *"Stephen Ludin" <[email-masked]>, "Mark Nottingham" <
> [email-masked]>, "Nick Sullivan" <[email-masked]>
>
>

---

From: Frederik Deweerdt <[email-masked]>
Date: Mon, 2 Jul 2018 11:06:57 -0700

> For interest / discussion. This is a proposal for a minimal mechanism to avoid loop attacks and misconfigurations against CDNs. Feedback appreciated.
I like the fact that this essentially is just a requirement to make
the request slightly bigger every time it goes through a host, and
ends up relying on the fact that some implementation defined limit
(individual header size or total header size) will eventually be hit,
which means that it doesn't have to specify how loops need to be
enforced.

Frederik

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jul 2018 09:46:23 +1000

Did you want to hint at what you might do when you receive one of
these things?  That is, is there a 5xx-series status code that a CDN
might use if it recognizes itself in the header field?

As Patrick mentioned, Via is abused in ways that can be fairly
invasive.  Aside from the obvious question of how this time it will be
different, which might be addressed with more text, are there some
missing privacy considerations?

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Jul 2018 12:08:07 +1000

Hi PHK,

The problem with something like Max-Forwards is that it requires all participants to actively process it. CDN-Loop works because I can insert my token and look for it in requests; if it appears, I know there's some sort of loop.

Cheers,

> On 2 Jul 2018, at 6:47 pm, Poul-Henning Kamp <[email-masked]> wrote:
> 
> --------

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Jul 2018 12:16:20 +1000

Hi Amos,

> On 2 Jul 2018, at 6:48 pm, Amos Jeffries <[email-masked]> wrote:
> 

A new, single-purpose header is preferable, because CDNs need to bar customer configuration/manipulation/etc of the mechanism, and since Forwarded serves multiple purposes, that might be more difficult.

> polluting HTTP header space with more features everybody with middleware

Like this?
  https://trac.nginx.org/nginx/ticket/1362

Even if that were fixed tomorrow, there are a *lot* of deployed nginx servers that will behave this way for the foreseeable future, and AIUI other implementations who do similar things. 

As to "polluting" - the HTTP header name space is large, so I don't think that's a valid argument. Intermediaries aren't forced to implement this; again, a single CDN can get some benefit, since it can insert its token and then recognise it in incoming requests.

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Jul 2018 12:32:41 +1000

Hi Patrick,

On 3 Jul 2018, at 2:29 am, Patrick McManus <[email-masked]> wrote:
> * I agree with the comment that, as proposed, this isn't really a CDN specific solution. Its a more opaque version of via.. so it makes sense to me to construct it independent of CDNs and instead focus it on any element that wants to participate in loop prevention. CDNs serve as the primary illustrator.

If it were made it into a generic loop-detection mechanism among all kinds of intermediaries, I'd be concerned that it becomes less robust for its intended purpose. Right now its primary use case is to avoid looping  in intermediaries that are uncoordinated in action but able to be directed to each other by a single party (either a potential attacker or a genuine mistake by a user). 

That's a fairly unique situation. People who want to avoid loops within their own infrastructure can already do so by inserting a new header field -- just as CDNs currently do. Or they can use Via, provided that they don't run into the same origin problems that I mentioned. It's only when the intermediaries are uncoordinated *but* able to be controlled by a potential attacker when you need to nominate a header that can't be modified by the user of the intermediary.

Another header like Forwarded could be used -- but since it's useful for other purposes, there will be some level of demand to remove / modify / etc. that header field, for various purposes. CDNs are very obliging like that. Likewise, if the header we choose were available for other purposes / parties, I'm concerned that there would be pressure from customers to allow it to be changed, precisely *because* other components are acting upon it; people tend to use their CDNs to do all sorts of fixups, routing, etc.

I know we try to make mechanisms as general as possible here, and that's a virtue. In this case, however, making it general makes it an attractive nuisance. 

> * I would be interested in more text on why Via failed and therefore this should succeed.. you mention that via is used in practice as a control signal incompatibly enumerating intermediaries.. I agree, but I also think via in practice reveals information people don't want revealed.. is CDN-loop perhaps more robust to this because the token is opaque (and perhaps not consistent between exchanges)?

I think it's more robust because it's specifically designed as a single-purpose mechanism with a very limited scope of applicability (CDNs); that's why I'm pushing back on broadening it.

> * the draft focuses on creation of the cdn-loop request header but has nothing to say on who/how/when that is policed.

Yes, I think some advisory text would help here.

> * "the token defines the CDN as a whole." feels like it needs more guidance.. is it even the right advice? Is it more about the scope of the token?

Since the primary use case is avoiding loops *between* CDNs, it seemed to be the right granularity. I'm open to changes here because at the end I think CDNs will put what they want into it; the only intended audience is themselves.

> are there use cases where cascading cdns make sense that might involve CDN repeats without forming a loop?

Could you give a concrete example of what you're thinking about here?

> * it would really help the wg to understand the scope of the problem a little more.. "not unknown for CDNs to be configured in a loop" makes me wonder if the authors believe this is a serious problem (though I suspect they just didn't want to over-assert).. but given that the document basically wants to restandardize something that has been standardized (Via), a little discussion of the operational imperative makes sense. (this is not necessarily something that would have to be in the doc)

I don't think this is considered to be a *severe* problem, but it's one where most people (without speaking for the entire industry!) would like to see a coordinated solution for. 

Personally, I also want to see CDNs work together more; they're a large part of the Internet now, and more standards for how they behave is IMO a good thing for everyone. This seemed like a small, easy (hah) problem to start on.

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 3 Jul 2018 12:34:38 +1000

Hi Martin,

> On 3 Jul 2018, at 9:46 am, Martin Thomson <[email-masked]> wrote:
> 

I think that's entirely up to the CDN and its configuration by customers.

> different, which might be addressed with more text

Based on the discussion so far, I think a FAQ might help, yes.

> , are there some missing privacy considerations?

Such as? It's only a request header. I suppose the CDN could put sensitive information in the payload if it wanted to, but that's no different from any other header field that allows extensibility, or unregistered fields. Did you have something else in mind?

Cheers,

---

From: Martin Thomson <[email-masked]>
Date: Tue, 3 Jul 2018 13:50:53 +1000

> Such as? It's only a request header. I suppose the CDN could put sensitive information in the payload if it wanted to, but that's no different from any other header field that allows extensibility,…

Yeah, it's not inherently bad, unless the CDN decides to make it so.
A gentle admonishment to keep this to information that directly
identifies the CDN would help.  Failing that, any information the CDN
adds should only be for its own consumption and it should therefore be
constructed in a way that ensures that.

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 3 Jul 2018 18:30:34 +1200

On 03/07/18 04:29, Patrick McManus wrote:
> specific solution. Its a more opaque version of via.. so it makes sense
> to me to construct it independent of CDNs and instead focus it on any
> element that wants to participate in loop prevention. CDNs serve as the
> primary illustrator.
> 
> * I would be interested in more text on why Via failed and therefore
> this should succeed.. you mention that via is used in practice as a
> control signal incompatibly enumerating intermediaries.. I agree, but I
> also think via in practice reveals information people don't want
> revealed.. is CDN-loop perhaps more robust to this because the token is
> opaque (and perhaps not consistent between exchanges)?

That is wrong and I think a derivative of people not reading the specs
properly.

RFC 7230 section 5.7.1:
"
   The received-by portion of the field value is normally the host and
   optional port number of a recipient server or client that
   subsequently forwarded the message.  However, if the real host is
   considered to be sensitive information, a sender MAY replace it with
   a pseudonym.
"

Or do you mean that the information "1.1" versus "1.0" is so important
for privacy that it cannot be shared around?

> nothing to say on who/how/when that is policed.

Aye, and this seems to also be part of the root problem with Via usage.
Lack of guidance on use for the particular cases, just statement that it
can be.

> guidance.. is it even the right advice? Is it more about the scope of
> the token? are there use cases where cascading cdns make sense that
> might involve CDN repeats without forming a loop?
> 

Seems like meaning of the token to me and also another thing which Via
already has.

> wonder if the authors believe this is a serious problem (though I
> suspect they just didn't want to over-assert).. but given that the
> document basically wants to restandardize something that has been
> standardized (Via), a little discussion of the operational imperative
> makes sense. (this is not necessarily something that would have to be in
> the doc)
> 

The more I hear about the use-case the more I think reusing Via is still
the better choice, but perhapse an informational document explaining how
CDNs should best use Via for loop detection while protecting privacy.

Amos

---

From: Amos Jeffries <[email-masked]>
Date: Tue, 3 Jul 2018 18:34:01 +1200

> I think it's more robust because it's specifically designed as a single-purpose mechanism with a very limited scope of applicability (CDNs); that's why I'm pushing back on broadening it.

Huh?

RFC 7230 section 5.7.1:
"
   Via can be used for tracking message forwards, avoiding
   request loops, and identifying the protocol capabilities of senders
   along the request/response chain.
"

That is multiple purposes and this CDN loop detection case is one of them.

> Since the primary use case is avoiding loops *between* CDNs, it seemed to be the right granularity. I'm open to changes here because at the end I think CDNs will put what they want into it; the onl…

Exactly the same with Via received-by token. Which can also be
aggregated at the CDN gateway to hide internal structure. Internal nodes
MAY add Via entries for that gateway to use. Which then get
elided/aggregated on use.

> Personally, I also want to see CDNs work together more; they're a large part of the Internet now, and more standards for how they behave is IMO a good thing for everyone. This seemed like a small,…

Cheers
Amos

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Tue, 03 Jul 2018 11:30:21 +0000

--------
In message <[email-masked]>, Mark Nottingham writes:

>The problem with something like Max-Forwards is that it requires all 
>participants to actively process it. CDN-Loop works because I can insert 
>my token and look for it in requests; if it appears, I know there's some 
>sort of loop.

As long as nobody in the loop removes the header, it will eventually
do its job if just one proxy in the loop decrements and tests it for zero.

A header which exposes the layering of the full sandwich is at the
very least worrisome from a security point of view.