Subject: Some proxy needs

From: "Nicolas Mailhot" <[email-masked]>
Date: Sun, 8 Apr 2012 14:14:15 +0200

I'll attempt a quick summary of what's been written on the list those past weeks

A proxy needs:

1. discoverability, to handle network guests (right now taken care of wpad+pac
though a lot of clients do not handle those, and they are not really
satisfactory I guess). Automated discoverability is not the same thing as
applying those blindly in the web client behind the user back. The web client
is free to try a direct connection anyway, but if it fails it should notify
the user of a proxy presence and have it accept it, or not. Right now no
discoverability means settings are copied blindly by users from one web client
to another, and I fail to see how that improves security.

2. a way to signal the web client a specific URL access (any access at any
time) needs proxy authorization, and where this authorization happens (on
complex network topologies there is no single auth portal that works for every
access). Probably a way to communicate the terms of service associated with
this gateway, in a non-spoofable way. To avoid spoofing this requires specific
web client chrome.

3. a way to signal the web client a request is being processed (there is no
way a multi-GB iso is going to pass through the anti-malware system
instantaneously, and users will press retry if the download bar does not move
after a few seconds)

4. A way to inspect most of the client communication for malware. I say most
because :
 a. a lot of operators would probably accept not to look at the user dialog
with bank sites or webmails operated by entities with known good security
(such as gmail)
 b. though not ideal, it would be an acceptable risk not to look at
authentication dialogs as long as they were too small to be an efficient
malware or botnet channel (I suppose some rate-limiting would need to occur
proxy-side)

5. a way for distribution sites to signal a resource is duplicated on a CDN,
and what the root resource is (systems like sourceforge are killing caching,
every new request is redirected to a different distribution server)

1 needs signing for any user white-listing to be secure

2-3 need out-of-band communication, probably multiplexed with the main stream,
so there is no doubt in the web client what message originates from the web
site and what message originates from the proxy. And the proxy messages need
at least signing (to avoid untrusted interception), and the proxy auth needs
full encrytion.

4. means there would be at least two levels of communication security:
 a. communication in clear
 b. crypted communication, with a middleman authorized to look at it (what
Willy proposes)

and ideally
 c. crypted end-to-end communication (but it needs clear separation from b.
and will be subject for specific proxy whitelisting, and probably require
proxy auth)

Except maybe for 1. everything else seems in http/2 scope to me.

-- 
Nicolas Mailhot

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 08 Apr 2012 12:41:51 +0000

In message <[email-masked]>, "Nicol
as Mailhot" writes:

>though a lot of clients do not handle those, [...]

I would have thought this was a job for DHCP ?

>instantaneously, and users will press retry if the download bar does not move

That sounds like serious scope-creep to me.

>4. A way to inspect most of the client communication for malware. I say most

If the site policy is "everything gets inspected", the protocol must support
that, either by allowing inspection, or by preventing the communication.

It site administrators choose not to, because of sound use of
decretion/legally requiments etc, that is not a relevant factor in
the standardization.

>every new request is redirected to a different distribution server)

Etags could almost do this, if there were a way to say blind the Host:
header ("Vary: -Host" ?)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
[email-masked]         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Sun, 08 Apr 2012 12:50:36 +0000

------ Original Message ------
From: "Poul-Henning Kamp" <[email-masked]>
To: "Nicolas Mailhot" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>
Sent: 9/04/2012 12:41:51 a.m.
Subject: Re: Some proxy needs
>I would have thought this was a job for DHCP ?
DHCP has some issues with this, it's not standardised, and option 252 
vs DNS lookups for WPAD... it's a bit of a mess.
  
and has no capability for communicating policy, or requirements for 
enforcement.
>That sounds like serious scope-creep to me.
  
http://tools.ietf.org/html/draft-decroy-http-progress-04
  
We actually deployed a similar (evolved version) to this in WinGate 7. 
If you install the AV plugin (free trials available for both), download 
a big file and throw it a progress header.  We patched Chromium source 
to get the client working.  It actually works really well.  Certainly 
stops human timeouts.
  
We didn't test it via a bunch of other proxies though.
  
I've an outstanding task to write up the modified protocol (contrary to 
that I-D it uses 103, name=value pairs, and only sends deltas).
  
Adrien

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Sun, 8 Apr 2012 21:44:01 +0200

>>That sounds like serious scope-creep to me.

Unfortunately a single user that pressed refresh half a dozen times because
he's not seeing progress on its multi-GB file can consume as much in a few
minutes as a normal user would over weeks.

> that I-D it uses 103, name=value pairs, and only sends deltas).

Could you clean it up and get it included in HTTP/2 ? That's the only way to
get it implemented by browsers and proxies.

Thanks!

-- 
Nicolas Mailhot

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 08 Apr 2012 20:01:28 +0000

In message <[email-masked]>, "Nicol
as Mailhot" writes:
>
>Le Dim 8 avril 2012 14:50, Adrien W. de Croy a Ã©crit :
>

Ohh, I don't dispute that.  I just don't see the solution being adding
a userinterface on top of HTTP...

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Sun, 8 Apr 2012 22:01:50 +0200

> decretion/legally requiments etc, that is not a relevant factor in

Real-world is not black-and-white. A big proxy setup is a compromise between
what the security people want (inspect everything for malware) and the user
happiness (some privacy). For some kinds of web sites the legal risks of
inspecting will outweigh the legal risks of not inspecting (user bank accesses
almost certainly fall there). That only reflects the ambivalence of general
law on this subject. Any law-abiding operator will try to match law as much as
possible.

Exceptions that won't be inspected even though the general policy is to
inspect will always be a minority because setting up exception lists is
administrative hell but the protocols should permit such lists to be put in
place.

Like Willy wrote previously, a typical proxy setup is a tiered config of
general rules, positive exceptions (do it even though the general rules say
you should not), and negative exceptions (don't do it anyway). There is no
reason choosing to inspect or not encrypted coms won't be handled the same
way.

Regards,

-- 
Nicolas Mailhot

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Sun, 8 Apr 2012 22:24:24 +0200

> a userinterface on top of HTTP...

Then if the protocol does not permit signaling progress, what the solution
would be? (educating users does not work, they've been brainwashed to refresh
at the slightest pause)

-- 
Nicolas Mailhot

---

From: "Poul-Henning Kamp" <[email-masked]>
Date: Sun, 08 Apr 2012 20:35:10 +0000

In message <[email-masked]>, "Nicol

>would be? (educating users does not work, they've been brainwashed to refresh

I think it is a design-mistake for the proxy to react to the forced
reload of a huge object, when is already busy processing that object
and have not delivered it to any clients yet.

Given that the client has not seen the object yet, the client has
no basis for conclusing it is out of date, so the reloads should
be ignored, the requests queued and once the object is processed,
you can deliver it.

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Sun, 8 Apr 2012 23:12:08 +0200

> be ignored, the requests queued and once the object is processed,

Ah, but on a big proxy farm setup, the load balancer may orient the second
request to an new proxy (since the first one is busy). And the origin web site
may also be part of a cdn, and the second request may not end on the same
delivery server as the first one

Either way once you pile up several layers where load is shared between
multiple parallel nodes, implicit signaling breaks down badly

-- 
Nicolas Mailhot

---

From: Mark Nottingham <[email-masked]>
Date: Mon, 9 Apr 2012 10:14:36 -0500

Hi Nicolas,

> get it implemented by browsers and proxies.

This is firmly out of scope for the work of this group.

Cheers,

--
Mark Nottingham
http://www.mnot.net/

---

From: Eliot Lear <[email-masked]>
Date: Mon, 09 Apr 2012 17:55:03 +0200

On 4/8/12 2:50 PM, Adrien W. de Croy wrote:
> [discoverability]

DHCP's problems aren't really that.  It's easy to write a new option. 
DHCP's problems are the following:

 1. Will O/S vendors adopt a new option?
 2. Will browsers tie to the O/S for the information (it's inherently an
    O/S function and you wouldn't want http clients making DHCP calls on
    their own)?

Let's assume for a moment that a PAC file or something similar is
pointed to.  That's probably rich enough for most.  Assuming the file is
shared over SSL, then there's the matter of whether the source is
trusted.  That's ALSO not a DHCP problem, but a problem with ANY
discovery mechanism.

I'll argue that if you can get O/S vendors into the game, then [2]
should follow because it probably doesn't make sense to have more than
one of these things on a host, and could lead to unintended inconsistent
behavior.

Eliot

---

From: Per Buer <[email-masked]>
Date: Mon, 9 Apr 2012 18:30:28 +0200

> delivery server as the first one

So, if the proxy farm fails to hash incoming requests on source IP or
target URL then this might happen. But either of these methods will
easily help avoid the problem. So, I personally don't think there is
need to further complicate things that are somewhat easy to solve
without making HTTP 2.0 even more complicated.

--
Per Buer
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
Varnish makes websites fly!
Whitepapers | Video | Twitter

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Tue, 10 Apr 2012 08:56:26 +0200

Le Lun 9 avril 2012 18:30, Per Buer a écrit :
> target URL then this might happen.

That breaks load balancing as soon as your network is big enough, with
different parts that get active at different points of the day.

> But either of these methods will
> easily help avoid the problem.

No they won't.
To scale network equipments need to be as stupid as possible, and as much
smarts as possible kept in the endpoints. You're breaking this principle
there.

And anyway even if your solution was possible, you still get unhappy users
that serial refresh because they're not seeing initial progress in their web
clients

-- 
Nicolas Mailhot

---

From: Per Buer <[email-masked]>
Date: Tue, 10 Apr 2012 11:23:15 +0200

Nicholas, list,

> different parts that get active at different points of the day.

Sorry if I seem to miss the point, but why would it break? Are you worried that
one point in the farm would get too hot?

> smarts as possible kept in the endpoints. You're breaking this principle

Having some smarts in the network to optimize cache hit rates seems to be a
reasonable optimization as long as the amount of state can be kept as low
as possible.

> that serial refresh because they're not seeing initial progress in their web

If the loadbalancer was balancing on target URLs the request would end up at
the same proxy. The proxy should be smart enough to coalesce the request
into the ongoing one and this one could feed the user a couple of bytes so
at the client understands that there is some progress on the download as
some proxies have done for years.

--
Per Buer

---

From: "Nicolas Mailhot" <[email-masked]>
Date: Tue, 10 Apr 2012 12:36:07 +0200

Le Mar 10 avril 2012 11:23, Per Buer a écrit :
> Sorry if I seem to miss the point, but why would it break? Are you worried
> that one point in the farm would get too hot?

I'm worried first that the activity distribution changes over time (so a
static table does not work), and second that any network-wide state (such as
an hashtable) is a contention point and potential spof. Making an individual
gateway smarter is simple (scaling the farms is just the matter of adding new
boxes). Making balancing smart and robust when some gateways or even some
physical processing sites can go down at any time is hard.

That's why direct browser-to-proxy communication will always be preferred and
more reliable than depending on a magical holistic hashtable balancer.
Round-robin is dumb but robust and stateless.

> some proxies have done for years.

Can't do when inspecting for malware. Malware inspection is not streamable,
you need to decompress recursively all the container formats (isos, zips,
jars, etc) to inspect the content before feeding the first byte to clients.
Just decompressing the container often requires the full file to have been
received proxy-side before starting the processing. The gateway could tell the
browser it's working on its request, but not give it part of the file.

-- 
Nicolas Mailhot

---

From: Per Buer <[email-masked]>
Date: Tue, 10 Apr 2012 14:59:02 +0200

Hi Nicolas, list,

On Tue, Apr 10, 2012 at 12:36 PM, Nicolas Mailhot
<[email-masked]> wrote:
> an hashtable) is a contention point and potential spof.

The load balancers derrive the correct  proxy from the URL through a
hash algorithm.
The only state would lie on the number of active proxies. Altering the
number of proxies
requires a bit of thinking and you might  want to rehash at some point
which will of
course be painful. Sorry if I might state the obvious here but this is
really quite
stateless and there is no SPOF here and no contention point. We've
used this on quite
large scale with reverse proxies and it works like a charm.

> Making an individual
> Can't do when inspecting for malware.  (..)

Sure. This has been done for years. It was an absolute necessity when dealing
with earlier versions of IE. Feeding it the first 10 bytes of
something was considered
safe enough and would keep the browser from timing out. So giving it
one byte every
30 seconds or so would buy us 5 minutes.

I think this is perfectly doable withing the current framework and without the
need to add further complexities to the protocol.

-- 
Per Buer

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Tue, 10 Apr 2012 13:11:02 +0000

------ Original Message ------
From: "Per Buer" <[email-masked]>
>need to add further complexities to the protocol.
  
it's a hideous hack, it's hard to describe how badly drip-feeding sucks.
  
It doesn't provide any other benefit (unlike progress notifications 
which have many applications).
  
And it doesn't work for scanning PUT or POST, where the client sends 
the whole resource to the intermediary quickly (because it's not being 
sent on) and then times out waiting for any response because the 
intermediary has to upload it to a poorly-connected server at low speed.
  
We can do a lot better.  
Adrien

---

From: Roberto Peon <[email-masked]>
Date: Tue, 10 Apr 2012 07:36:32 -0700

> large scale with reverse proxies and it works like a charm.

This isn't stateless, and claims that it is are in error. The state is
external to the connection (as you have identified earlier). The fact that
it is external doesn't make it any less important.

You are right that there isn't a SPOF... there are N points of failure
instead, any of which can cause problems.
> > physical processing sites can go down at any time is hard.

The above statement is important!
-=R

---

From: Anthony Bryan <[email-masked]>
Date: Wed, 18 Apr 2012 15:03:57 -0400

> every new request is redirected to a different distribution server)

some distribution sites already use metalink (HTTP=RFC 6249, XML=RFC
5854) for showing where resources are duplicated.

as mentioned privately, unfortunately RFC 6249 is silent on how a
proxy is supposed to behave when encountering metalinks. any
suggestions on how this should work would be helpful.

we hope to have a student working on the proxy side of things for
Google summer of code this year. :)
-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads