Subject: Fwd: draft-richsalz-httpbis-https-downgrade-00.txt

From: Erik Nygren <[email-masked]>
Date: Mon, 11 Mar 2019 22:52:55 -0400

This draft on "Best practices for TLS Downgrade" is intended as a starting
point for discussion on a topic that many people would like to ignore but
which introduces risk into the ecosystem.  We'd like to bring some
co-authors onboard (especially from other CDNs and browsers/OSes) and
incorporate lessons learned elsewhere as well.  While "don't downgrade!" is
almost always the "correct" solution, it isn't always viable.  Getting
alignment on best practices may at least help provide better visibility
into the associated risks, such as by exposing to clients when an insecure
downgrade to cleartext is happening and by stripping request data most
likely leak private information.

Feedback and suggestions for additions are most welcome, and we're also
interested in discussing more in Prague.

    Erik

---------- Forwarded message ---------
From: <[email-masked]>
Date: Mon, Mar 11, 2019 at 4:08 PM
Subject: I-D Action: draft-richsalz-httpbis-https-downgrade-00.txt
To: <[email-masked]>

A New Internet-Draft is available from the on-line Internet-Drafts
directories.

        Title           : Best practices for TLS Downgrade
        Authors         : Brian Sniffen
                          Mike Bishop
                          Erik Nygren
                          Rich Salz
        Filename        : draft-richsalz-httpbis-https-downgrade-00.txt
        Pages           : 8
        Date            : 2019-03-11

Abstract:
   Content providers delivering content via CDNs will sometimes deliver
   content over HTTPS (or both HTTPS and HTTP) but configure the CDN to
   pull from the origin over cleartext and unauthenticated HTTP.  From
   the perspective of a client, it appears that their requests and
   associated responses are delivered over HTTPS, while in reality their
   requests are being sent across the network in-the-clear and responses
   are delivered unauthenticated.  This exposes user request data to
   pervasive monitoring [RFC7258]; it also means response data may be
   tampered with by active adversaries.  Terminating TLS connections on
   a load balancer and contacting a backend over cleartext has long been
   common within data centers, but doing this TLS termination and
   downgrade to HTTP at a CDN introduces additional risk when the
   unprotected traffic is sent over the general Internet, sometimes
   across national boundaries.

   While it would be nice to say "never do this," customer demand,
   content provider use-cases, and market forces today make it
   impossible for CDNs to not support downgrade.  However, following a
   set of best practices can provide visibility into when this is
   happening and can reduce some of the risks.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-richsalz-httpbis-https-downgrade/

There are also htmlized versions available at:
https://tools.ietf.org/html/draft-richsalz-httpbis-https-downgrade-00
https://datatracker.ietf.org/doc/html/draft-richsalz-httpbis-https-downgrade-00

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

_______________________________________________
I-D-Announce mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/i-d-announce
Internet-Draft directories: http://www.ietf.org/shadow.html
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

---

From: Alessandro Ghedini <[email-masked]>
Date: Tue, 12 Mar 2019 11:44:05 +0000

Hello,

> interested in discussing more in Prague.

Some of the recommendations seem very hard if not impossible to implement:

1. Don't downgrade X / Only downgrade Y: it just ends up breaking customers'
   websites and if CDNs were allowed to break customers' websites and wanted
   to reduce use of plain HTTP to origins they could just forbid HTTP
   altogether.

   It also becomes hard to debug problems caused by this. E.g. if customers
   look at say, browsers' developer tools and see a bunch of resouces that fail
   to be fetched, or an app that doesn't work properly, because the CDN decided
   to downgrade something but not something else.

   Also, how does the "not downgrade" work? Does the CDN return a specific
   status code? Which one?

2. Strip sensitive headers. Lots of websites (particularly APIs) implement
   custom authentication protocols. CDNs can't just guess what is sensitive
   and what is not.

   Not to mention that it potentially breaks customers' websites / mobile apps
   / whatever depends on those headers, and problems coming from that would be
   pretty hard to debug as well.

   I mean, the draft says so itself: "Mechanisms that rely on lists of what
   is allowed or what is banned rely on an implausibly detailed and up-to-date
   models of Web use", so it seems strange that this is a recommendation.

3. "Protocol-To-Origin: cleartext" header: this makes sense I think. I imagine
   browsers are unlikely to add a special error for this, so it would end-up
   being treated like plaintext HTTP requests, which might encourage people to
   try to fix this. Are there other clients that could use this?

   But to be able to do this the header should be standardized. Is a BCP
   enough for this? Until there's an actual standard header, it doesn't seem
   like it would be possible to implement.

Cheers

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 13 Mar 2019 05:06:12 +1300

>    like it would be possible to implement.

We already have the Forwarded header defined for this use.

CDN's typically erase such headers to hide their internal structures.
Making yet another header to expose the details will not change that
behaviour, only conflate the set of rarely used headers.

AYJ

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 13 Mar 2019 11:58:53 +1100

Hey Erik et al,

Thanks for publishing this.

I know there's been some discussion about this in the past -- especially about a CDN-to-browser signal. As I recollect, that got stuck on how to meaningfully present that to the user without complicating the security signals from the browser.

I agree with Alessandro's concern about hobbling the protocol in these situations; it seems like a very artificial kind of pain.

I've been wondering for a little while about an adjacent concern -- that origin configuration for CDNs is often ad hoc, with things like SNI and Host headers getting in the way. If we could define a profile (or profiles) of a CDN origin that is easy to configure and meets the basic use cases as well as security requirements, we might be able to encourage people towards a common goal; i.e., a carrot, rather than a stick. It might also be interesting to consider something like a dedicated origin configuration URI scheme. Are those sorts of things interesting to you?

*chair hat on* Would you like some time to talk about this in Prague? The agenda is getting a bit tight, but I think we could squeeze you in.

Cheers,

> On 12 Mar 2019, at 1:52 pm, Erik Nygren <[email-masked]> wrote:
> 
> This draft on "Best practices for TLS Downgrade" is intended as a starting point for discussion on a topic that many people would like to ignore but which introduces risk into the ecosystem.  We'd like to bring some co-authors onboard (especially from other CDNs and browsers/OSes) and incorporate lessons learned elsewhere as well.  While "don't downgrade!" is almost always the "correct" solution, it isn't always viable.  Getting alignment on best practices may at least help provide better visibility into the associated risks, such as by exposing to clients when an insecure downgrade to cleartext is happening and by stripping request data most likely leak private information..
> 
> Feedback and suggestions for additions are most welcome, and we're also interested in discussing more in Prague.
> 
>     Erik
>  
> 
> A New Internet-Draft is available from the on-line Internet-Drafts directories.
> 

--
Mark Nottingham   https://www.mnot.net/

---

From: Erik Nygren <[email-masked]>
Date: Tue, 12 Mar 2019 21:50:05 -0400

Hi Mark,

It would be great to have some time to discuss in Prague.

On the CDN-to-client signal, one major initial value may be for providing
visibility and transparency.  For example, this could be useful for
researchers and for dashboards such as https://securethe.news/ in a similar
way that the 451 response code was useful even if the target audience
wasn't end-users.

For "hobbling the protocol", it has the potential to be useful to
discourage some of the most dangerous behaviors.  Many of the biggest
challenges and drivers for downgrade seem to come from origins hosting
large media libraries (video streaming, images, user-generated content,
etc).  Supporting downgrade but while stripping cookies and the like can
help reduce risks from misconfigurations or miscommunication.  I'd agree
that this won't work as well with small HTML websites (vs
images/media/video libraries), and other strategies may be more appropriate
there.  Anecdotally, the latter are potentially easier to just get to
deploy HTTPS to the origin, especially with the existence of LetsEncrypt,
but the mix of these challenges may vary for CDNs with different customer
bases.

Most of the challenges we've seen on this front have come from folks with
custom / legacy origin infrastructure.  It's unclear whether a profile
would help, although I can see some best practices there being useful in
other ways.

    Erik

On Tue, Mar 12, 2019 at 8:59 PM Mark Nottingham <[email-masked]> wrote:

> I know there's been some discussion about this in the past -- especially
> about a CDN-to-browser signal. As I recollect, that got stuck on how to
> meaningfully present that to the user without complicating the security
> signals from the browser.
>
> I agree with Alessandro's concern about hobbling the protocol in these
> situations; it seems like a very artificial kind of pain.
>
> I've been wondering for a little while about an adjacent concern -- that
> origin configuration for CDNs is often ad hoc, with things like SNI and
> Host headers getting in the way. If we could define a profile (or profiles)
> of a CDN origin that is easy to configure and meets the basic use cases as
> well as security requirements, we might be able to encourage people towards
> a common goal; i.e., a carrot, rather than a stick. It might also be
> interesting to consider something like a dedicated origin configuration URI
> scheme. Are those sorts of things interesting to you?
>
> *chair hat on* Would you like some time to talk about this in Prague? The
> agenda is getting a bit tight, but I think we could squeeze you in.
>
> Cheers,
>
>
> > This draft on "Best practices for TLS Downgrade" is intended as a
> starting point for discussion on a topic that many people would like to
> ignore but which introduces risk into the ecosystem.  We'd like to bring
> some co-authors onboard (especially from other CDNs and browsers/OSes) and
> likely leak private information..
> >