Subject: protocol support for intercepting proxies

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 Jun 2007 10:23:35 +1200

Hi all

I guess mentioning "intercepting proxies" is probably akin to swearing 
on this list, as they break many aspects of the spec, (and TCP).

However, in this customer-wants-customer-gets world of ours, many proxy 
vendors are commercially forced to provide implementations.

It is very appealing for system administrators to install an 
intercepting proxy, as it "solves" the issue of client browser 
configuration.  Sure, there are many other methods of auto-proxy 
configuration, but these all rely on ancillary systems and extra 
sys-admin (and sometimes customer) knowledge (i.e. DHCP option 242, 
and/or DNS WPAD lookup).

Vendors who implement intercepting proxies do so without much (if any) 
support from the spec, and so there are issues encountered  which end up 
being solved by trial and error or best guess or in some cases are not 
even solvable (not reliably or properly anyway).

For instance the issue when a proxy intercepts connections then wishes 
to force the UA to authenticate to the proxy.

This is a really common scenario.

Since the UA doesn't believe it is talking to a proxy, it will only 
process WWW-Authenticate headers, not Proxy-Authenticate ones, and sends 
Connection headers instead of Proxy-Connection.  Often a UA will behave 
quite differently if it believes it is talking to a server than a proxy 
(for instance handling of POST requests in IE).

Given that the problem is not going to go away because people are not 
going to want to stop using intercepting proxies, wouldn't it be better 
if there was some proper protocol support for the concept?

UAs at the moment don't generally know if their connections are being 
intercepted.  If they knew, then they could;

* let the user know connections were being intercepted
    - ameliorates issues relating to privacy
    - helps users decipher errors better (i.e. upstream connection failure)
    - leads towards possible user-control over whether their traffic may 
be intercepted or not
* cooperate better with the proxy.
    - move to a proxy-oriented protocol operation (can fix many issues, 
such as auth)
    - deal with errors differently

I believe that this could be achieved with either a status code, or a 
header, where an intercepting proxy could signal to a client that it 
intercepted the connection.  The proxy could even intercept connections 
and enforce the UAs to use a proxy method, provide a proxy URI in the 
response that the UA must use.   This is another case where 305 
would/could have been useful.  Another option would be a warning code to 
indicate the connection had been intercepted.  I believe system 
administrators would wish to be able to configure how to deal with the 
case from a number of options including

1. Allow the clients to operate through the intercepting proxy
    - with notification
    - silently
2. Force the clients to re-connect to the proxy and issue requests with 
proxy semantics.

Option 1 would only require a header field (perhaps Warning), option 2 
would require a Location header to be used, and most likely a status code.

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Mon, 18 Jun 2007 01:21:16 +0200

> if there was some proper protocol support for the concept?

Big fat no from this proxy vendor (Squid). Interception and
Authentication does not mix, and should not mix.

What would be better is a standardized proxy discovery mechanism. WPAD
isn't far from that today, and complemented with a DNS SRV profile it's
fairly complete for all uses..

And perhaps some way for local networks to tell clients that proxies
MUST be used.

Doing this in HTTP is just dangerous. Switching to using proxies should
not be taken lightly as there is security and privacy concerns. Doing so
in response to messages from random servers without any chain of trust
to the client is just plain wrong.

What could reasonably be done in this area using HTTP interception is an
error indication which hints the client that it should go and look for a
proxy if it want's to gain Internet access. But not which proxy. That
needs to be discovered with some mechanism where there is a reasonable
chain of trust already established.

>     - leads towards possible user-control over whether their traffic may 

This is already in the specs thanks to the Via header.

>     - deal with errors differently

And this is covered by WPAD (even if just Informal today..).

And without WPAD the UA could in theory extract what it needs from Via
to switch to proxy operation if desired, provided the proxy actually
adds what it is required to add to the Via, but doing so would be a
security nightmare due to the completely missing chain of trust.

Also many doing interception don't want their proxy to send Via because
they don't want to tell the user which proxy intercepted the request.
Quite many admins doing the interception dance even asks if it's
possible to completely hide any proxy generated error messages etc,
making things always behave as if the proxy wasn't there..

Regards
Henrik

---

From: Travis Snoozy <[email-masked]>
Date: Sun, 17 Jun 2007 16:35:10 -0700

On Mon, 18 Jun 2007 10:23:35 +1200, Adrien de Croy <[email-masked]>
wrote:

<snip>
> sys-admin (and sometimes customer) knowledge (i.e. DHCP option 242, 
<snip>
> Another option would be a warning code to indicate the connection had
> been intercepted.  I believe system administrators would wish to be
> able to configure how to deal with the case from a number of options
> including
>
> 2. Force the clients to re-connect to the proxy and issue requests
> with proxy semantics.
<snip>

What I'm getting from the first part is: admins don't want to configure
the browser. The second part, though, seems to say admins want to
configure the browser. Isn't this a little conflicted?

> Vendors who implement intercepting proxies do so without much (if
> any) support from the spec, and so there are issues encountered
> which end up being solved by trial and error or best guess or in some
> cases are not even solvable (not reliably or properly anyway).
> 
> For instance the issue when a proxy intercepts connections then
> wishes to force the UA to authenticate to the proxy.
>

And it's been solved for some time now, practically speaking[1]. Also,
one should speak about authenticating to the _network_, not the proxy --
the proxy simply provides the service for authentication. It's not an
"HTTP proxy" per se, so much as a TCP/IP filtering application that
happens to speak HTTP + HTML, because most users have something that
can speak HTTP + HTML. Actual HTTP-level filtering and/or caching is
another story altogether, but I'm sticking with simple authentication
for now.

<snip>
> going to want to stop using intercepting proxies, wouldn't it be
> better if there was some proper protocol support for the concept?

Yes, but what about backwards compatibility? The proxy still needs a
way to let browsers that *don't* implement such extensions to
authenticate and work properly. We wind up with a chicken/egg problem,
and we still have to solve the original issue with the existing
infrastructure, regardless.

>     - ameliorates issues relating to privacy

So long as the proxy-operator wants them to know, and is a decent human
being, and the software supports it .

>     - helps users decipher errors better (i.e. upstream connection

A good error message from the proxy should be adequate for this ("500
failed to connect to server at example.org"). Alternately, one could
pass the TCP/IP issues directly through (e.g., if the connection timed
out, let it time out on the client; if it was reset, reset it, etc.).
What about identifying the proxy with the client would help?

>     - leads towards possible user-control over whether their traffic
> may be intercepted or not

See prior comment about proxy op being a decent human being. Also, the
options in this scenario are go through the forced proxy, or don't get
Internet access -- a security policy wouldn't be very helpful. Users
should *always* assume their traffic is monitored (esp. on business &
school networks, where this type of proxy is likely to occur), and
vary their browsing habits based on that assumption. Users interested
in getting around eavesdroppers should already be using technologies
like Tor, VPNs, anonymizing SOCKS proxies, etc.

>     - move to a proxy-oriented protocol operation (can fix many
> issues, such as auth)

Yet another proxy-discovery technology -- but why? How are legacy
browsers going to cope?

>     - deal with errors differently

Examples?

> intercepted the connection.  The proxy could even intercept
> connections and enforce the UAs to use a proxy method, provide a
> proxy URI in the response that the UA must use.

So, instead of the admins putting the infrastructure in to allow
auto-config, we'll force the end-users to do it themselves? That seems
kind of backwards.

> This is another case where 305 would/could have been useful.  Another
> option would be a warning code to indicate the connection had been
> intercepted.  I believe system administrators would wish to be able
> to configure how to deal with the case from a number of options
> including
> 
> 2. Force the clients to re-connect to the proxy and issue requests
> with proxy semantics.

Why bother? What would either of these points gain us?

Some solid examples/scenarios would be really handy for illustrating
the issues you're coming up against. I can vaguely see where you're
going (drop a browser on the network, have it auto-configure by virtue
of simply getting routed through the proxy, with no extra setup), but I
don't see any really big win, especially when other technologies
(zeroconf, UPnP, etc.) have been explicitly written to solve the problem
generically.

-- 
Travis

[1] http://en.wikipedia.org/wiki/Captive_portal

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 Jun 2007 11:41:43 +1200

> Authentication does not mix, and should not mix.
I've got quite a few customers who would disagree with you on that 
unfortunately.

And quite a few competitors as well.  You'll find most proxy vendors 
that sell their proxies integrated with firewalls etc with packet-level 
capabilities provide interception capabilities, and with auth.  E.g. 
WinRoute, WinProxy, I think ISA server does as well, I'm sure there are 
a whole lot more.

Just because it is bad (imperfect) technically doesn't stop customers 
from wanting it, and therefore vendors from giving it to their customers.

It doesn't work poorly enough to put the customers off it.

> to the client is just plain wrong.

Obviously such a mechanism should be used only on the proxy closest to 
the client on the edge of the client's network, so the client would 
trust that proxy.  The gateway would block any such things from outside 
the network, or outside the circle of trust.  For instance many ISPs 
here use intercepting proxies.  Many.  Mainly due to lack of 
international bandwidth, so they save on bandwidth by forcing all 
customers through a great big cache.

Administrators want all issues to be solved in one place.  Having to set 
up SRV records, DHCP options etc etc, WPAD files etc is just adding more 
links to a chain which then gets weaker and weaker.  Support for this 
becomes more difficult.  Competitive forces mean one vendor offers a 
"solution", and the rest must follow or be uncompetitive.  It's all very 
well taking a moral stance on it, but when you start going hungry 
because your customers don't appreciate it, and other vendors are happy 
to provide them what they want, then you start to question whether the 
customers really are wrong.

So like it or not, intercepting proxies, and auth issues is here to 
stay.  I'm just suggesting we should accept that and address the issue 
for the good of the customers.

> chain of trust already established.

I agree totally with the trust issue.  But I believe it is possible to 
do this within HTTP

> This is already in the specs thanks to the Via header.
Many intercepting proxies don't add a Via header, since they wish to 
remain invisible.

If we address intercepting proxies, i.e. allow for them in the spec 
(recognise the reality of their existence), then we could deprecate 
invisible intercepting proxies.

Whilst the spec ignores their existence, it can do nothing about these 
issues.

> making things always behave as if the proxy wasn't there..
that's impossible, since the TCP connection was already intercepted, and 
therefore the proxy cannot always guarantee it will behave the same as a 
server (i.e. if the server isn't even available, the proxy can't go back 
in time and not accept the intercepted connection from the client).

Regards

Adrien

-- 
Adrien de Croy - WinGate Proxy Server - http://www.wingate.com

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 Jun 2007 12:03:19 +1200

> configure the browser. Isn't this a little conflicted?
Admins don't want to individually configure thousands of individual 
browsers.

They wish to configure browsers centrally with a single central 
configuration setting.

> And it's been solved for some time now, practically speaking[1]. 

largely, but there are some significant warts on it.

WinGate for example does the hideous juggling act of intercepting 
connections, running NTLM auth over them, then allowing the same request 
to go through to an origin server that then also requires NTLM auth.

But it's not pretty, or particularly robust.  There are browser variations.

> Also,
lost me there.

> authenticate and work properly. 
if a 400 series code came back from an intercepting proxy, with a page 
saying "you need to configure your browser to use a proxy", plus a 
header field with the URI of the proxy, if the client trusted the source 
of this message, a compliant client could automatically retry the 
request to the proxy, even ask the user if they wish to set their 
browser to use this proxy for future requests. A non-compliant browser 
would show the message.

> We wind up with a chicken/egg problem,

or is forced by privacy legislation to do so.  Several countries I know 
of have quite advanced privacy regulations concerning internet traffic, 
i.e. Italy.

> out, let it time out on the client; 

You can't time out at the connection phase if you already moved past 
that phase by accepting the connection.

The whole advantage of caching intercepting proxies is to avoid 
connecting to the upstream origin server if possible.  So proxies have 
to accept the connection before the client will send the request, which 
the proxy needs to check the cache by which stage it's too late if the 
upstream connection fails.  All that can be done at that stage is 
present an error page to the client, at which stage they know 
immediately that their connection is being intercepted (if they are 
aware of the significance of such things).

> if it was reset, reset it, etc.).

You'd definitely want some mechanism to assist the human to make the 
decision about whether or not to use the proxy.

> like Tor, VPNs, anonymizing SOCKS proxies, etc.
Actually I can think of scenarios where this would be useful.

For instance our ISP intercepts all connections, but allows customers to 
opt-out of this (which we had to do in order to perform useful testing 
of WinGate).

The opt-out process of the ISP is something that consumes their 
resources.  A mechanism to allow a customer to opt-out would save the 
ISP resources.

Obviously this would be a configuration option, whether or not to allow 
users to opt-out or not (off by default for corporate proxies, possibly 
on for ISPs).

Most corporate gateways won't allow such simple bypassing of HTTP 
policy, and will block VPN, SOCKS etc.

> Yet another proxy-discovery technology -- but why? How are legacy

why: because current ones rely on out-of-service solutions (i.e. DHCP / 
DNS).  It's possible to solve the problem completely in HTTP.  One stop 
shop.

Legacy browsers will cope as per mechanisms above.  I'm not proposing 
deprecating these other methods, although I think they would fall from 
grace with a decent HTTP implementation.

>>     - deal with errors differently

classic one being the upstream connection failure.

> auto-config, we'll force the end-users to do it themselves? That seems

not end-users, UAs.

Don't forget that it's a configuration option on a browser whether or 
not to use proxy auto detection as well.

> Why bother? What would either of these points gain us?

reduced burden on admins and users.

the links in the chain that can break for WPAD.

1. Browser not configured to use proxy auto detect
2. Client DNS issues
3. DNS server not providing decent records for WPAD lookups
4. WPAD URL not working (web server serving WPAD files not configured 
properly)
5. Some clients use DHCP option 252 for WPAD, not DNS.
6. DHCP implementation and configuration issues

There are quite a few links that can break, several of which are 
client-side. On a large network, supporting all this can be a huge 
burden on admins.

So, to solve the issue within HTTP would seem sensible?

> (zeroconf, UPnP, etc.) have been explicitly written to solve the problem
UPnP is often seen as a security nightmare and turned off.  I don't know 
much about zeroconf.

the main win comes in sys admin work load, see above

Regards

Adrien

---

From: Travis Snoozy <[email-masked]>
Date: Sun, 17 Jun 2007 19:29:03 -0700

On Mon, 18 Jun 2007 12:03:19 +1200, Adrien de Croy <[email-masked]>
wrote:
<snip>
> > What I'm getting from the first part is: admins don't want to
> > configure the browser. The second part, though, seems to say admins
> > want to configure the browser. Isn't this a little conflicted?
> >

And users wish to have control over their own browsers. Unfortunately,
users on average are also fairly un-savvy when it comes to making good
security decisions. Providing a pop-up that lets the user decide is not
really an answer -- most of the time, the poor user just wants it to
work, and will click "yes" to anything that gets in the way between
them and that goal.

> connections, running NTLM auth over them, then allowing the same
> request to go through to an origin server that then also requires
> NTLM auth.
>

I'm not even going to touch that; this e-mail is already too long. :)
 
> > Also,
> > one should speak about authenticating to the _network_, not the
> > proxy -- the proxy simply provides the service for authentication.
> > It's not an "HTTP proxy" per se, so much as a TCP/IP filtering
> > application that happens to speak HTTP + HTML, because most users
> > have something that can speak HTTP + HTML. Actual HTTP-level
> > filtering and/or caching is another story altogether, but I'm
> > sticking with simple authentication for now.
> lost me there.

Alice hops onto the network. Bob, the network operator, needs to verify
that the person who hopped on the network (Alice) is actually a valid,
paid-up user, and not some cad trying to bum bandwidth (Malorie). So,
Bob denies nearly all services to folks who hop on the network until
they've authenticated somehow. The "somehow" could be anything, but it
should be done at the lowest protocol level available (e.g., dial-up
connections use PPP for link-layer authentication; wireless connections
are also authenticated at the link-layer via WPA+RADIUS; SOCKS, SSH, IP
over IPSec, and other VPN-like technologies can be used for
network-layer authentication).

HTTP is a bad way to do this, but it's used because anybody with a
browser can understand HTTP + HTML. Network authentication, however, is
a *separate* problem from providing caches. Caches introduced here
would be tantamount to trying to cache TCP/IP, which is utterly silly.
The fact that a single application can combine the two functions is
understandably confusing, and it's easy to get things mixed and jumbled
in such a way as to cause havoc.

> >
> > <snip>
> >> Given that the problem is not going to go away because people are
> >> not going to want to stop using intercepting proxies, wouldn't it
> >> be better if there was some proper protocol support for the
> >> concept?
> >
> if a 400 series code came back from an intercepting proxy, with a
> page saying "you need to configure your browser to use a proxy", plus
> a header field with the URI of the proxy, if the client trusted the
> source of this message, a compliant client could automatically retry
> the request to the proxy, even ask the user if they wish to set their 
> browser to use this proxy for future requests. A non-compliant
> browser would show the message.

Yes, but the automatic configuration is just ripe for abuse. I mean,
there's a reason why nobody uses 305, and it's because you'd have to be
nuts to let some arbitrary server on the Internet tell you to route
yourself through some other arbitrary server.

DHCP is relatively safe -- it's broadcast at the link-layer, and isn't
generally routed (and when it is, it's done in a conscious manner).
That makes it great for telling you which proxy to use, granted that
you trust your local network. Browsers use HTTP over TCP/IP, which is
Internet-routable by definition -- because of this, allowing proxy
redirects would be a Very Bad Thing. All it takes is a DNS hijack of one
domain you visit (or even unintentionally visiting one bad domain) along
with an uninformed "yes" click to have all your traffic routed through a
proxy.

Easy configuration is good, but easy/auto configuration should _always_
be limited to the local network ("local" being somewhat relative, here).

> >> UAs at the moment don't generally know if their connections are
> >> being intercepted.  If they knew, then they could;
> > So long as the proxy-operator wants them to know, and is a decent
> > human being, and the software supports it .
> 
> or is forced by privacy legislation to do so.  Several countries I
> know of have quite advanced privacy regulations concerning internet
> traffic, i.e. Italy.

Yes, but law does not protection make. Fraud is illegal, but
perpetrated daily. Compromising computers, e.g., for botnets, is
similarly illegal. XSS attacks to perpetrate phishing is illegal. Not
putting a proper Via: entry could be made illegal, but it won't stop
the badness (hijacking sessions by re-routing browsers through
arbitrary proxies) from actually happening.

> > A good error message from the proxy should be adequate for this
> > ("500 failed to connect to server at example.org"). Alternately,
> > one could pass the TCP/IP issues directly through (e.g., if the
> > connection timed out, let it time out on the client; 
> 

We could if we were tunneling TCP/IP over HTTP, but we're not (that's
part of the problem with trying to use this method for network
authentication).

> connecting to the upstream origin server if possible.  So proxies
> have to accept the connection before the client will send the
> request, which the proxy needs to check the cache by which stage it's
> too late if the upstream connection fails.  All that can be done at
> that stage is present an error page to the client, at which stage
> they know immediately that their connection is being intercepted (if
> they are aware of the significance of such things).

The benefit you stated here was that a UA knowing it is being
intercepted "helps users decipher errors better." That presupposes
that there is no secrecy, or attempt at secrecy. My question was, how
does the UA knowing interception is happening have any bearing on the
quality of the error message returned? Unless the UA and proxy are
trying to conspire to keep the user from figuring out there's a proxy
(which I don't think is what you're looking for), I don't see why an
appropriate/descriptive 5xx code wouldn't do.

> decision about whether or not to use the proxy.

... who's going to provide that information? The proxy? I'm supposed to
make a security decision based on the word of the proxy that's trying
to hijack my connection? That's just about as bad as Vista's UAC, or
(more topically) SSL certificate popups. The average user will scratch
his head for a moment, figure that he has to click "yes" to continue,
and be on his merry way. Security freaks like myself enjoy the
opportunity to stop badness from happening, but the average user just
wants the darned thing to connect.

> 
> >
> >>     - leads towards possible user-control over whether their
> >> traffic may be intercepted or not
> >
> > See prior comment about proxy op being a decent human being. Also,
> > the options in this scenario are go through the forced proxy, or
> > don't get Internet access -- a security policy wouldn't be very
> > helpful. Users should *always* assume their traffic is monitored
> > (esp. on business & school networks, where this type of proxy is
> > likely to occur), and vary their browsing habits based on that
> > assumption. Users interested in getting around eavesdroppers should
> > already be using technologies like Tor, VPNs, anonymizing SOCKS
> > proxies, etc.
> For instance our ISP intercepts all connections, but allows customers
> to opt-out of this (which we had to do in order to perform useful
> testing of WinGate).
> 
> Obviously this would be a configuration option, whether or not to
> allow users to opt-out or not (off by default for corporate proxies,
> possibly on for ISPs).
> 

That sounds like an IP/routing level change, not an HTTP-level one. The
flow would have to be something like...

User: I want to connect to example.com!
Proxy: Uh... you're going through me. That all right?
User: No!
Proxy: All right; I'll poke the backend so that your packets aren't
routed through me anymore.
User: ... I want to connect to example.com!
Example.com: Welcome!
...

Now, granted, the proxy could switch to transparent mode instead, but
that wouldn't _really_ be taking the proxy out of the equation. So,
yes, it would provide the proxy with some mechanism to make decisions,
but that type of thing could already be available in the form of other
standard technologies[1].

> why: because current ones rely on out-of-service solutions (i.e.
> DHCP / DNS).  It's possible to solve the problem completely in HTTP.
> One stop shop.
> 
> deprecating these other methods, although I think they would fall
> from grace with a decent HTTP implementation.
> 
> >

500: Your request was routed through a proxy, and that proxy could not
contact the server. Please contact <[email-masked]> if you have
further questions.

What's wrong with this, unless you're trying to _hide_ the fact that
there's a proxy?

> >
> >> I believe that this could be achieved with either a status code,
> >> or a header, where an intercepting proxy could signal to a client
> >> that it intercepted the connection.  The proxy could even intercept
> > auto-config, we'll force the end-users to do it themselves? That
> > seems kind of backwards.
> 
> not end-users, UAs.
> 

Yes it is, but that's a single checkbox. All the legwork is done by the
admins (as it _ought_ to be), and the technology is fairly well locked
to the local (assumed trusted) network (see comments on DHCP).

> >
> >> This is another case where 305 would/could have been useful.
> >> Another option would be a warning code to indicate the connection
> >> had been intercepted.  I believe system administrators would wish
> >> to be able to configure how to deal with the case from a number of
> >> options including
> >>

- Browser doesn't support these extensions, and needs to be configured
manually to use proxy

> 2. Client DNS issues

- If your client has DNS issues, they're kinda screwed net-wise
anyways. DHCP (another requisite for WPAD) should take care of this.

> 3. DNS server not providing decent records for WPAD lookups

- If you have bad proxies, transparently routing to them won't solve
anything over using DNS to point at them.

> 4. WPAD URL not working (web server serving WPAD files not configured 

- Proxies and routers can be likewise misconfigured.

> 5. Some clients use DHCP option 252 for WPAD, not DNS.

- No clients support the proposed new mechanism, and (given the track
record on *other* HTTP features) support will be spotty and of
questionable interoperability.

> 6. DHCP implementation and configuration issues

- Examples?

> So, to solve the issue within HTTP would seem sensible?

Except that there will still be all the client-side issues with the
"old clients". I mean, the best thing to do would seem to be to put up
a page that instructs users on how to configure their web browser to
use a proxy.

> > going (drop a browser on the network, have it auto-configure by
> > virtue of simply getting routed through the proxy, with no extra
> > setup), but I don't see any really big win, especially when other
> > technologies (zeroconf, UPnP, etc.) have been explicitly written to
> > solve the problem generically.
> >
> UPnP is often seen as a security nightmare and turned off.  I don't
> know much about zeroconf.

UPnP is a security nightmare only insofar as allowing users to
automagically poke holes in "gateway devices" (read: NAT router
and/or firewall). It's all broadcast-based, so everything stays on the
local network, like DHCP.

> the main win comes in sys admin work load, see above

So build it all into your product and put an easy-to-use interface on
it. It's value-add. In any case, all this configuration should be more
or less one-time-only. Once it's set up, everything should just keep
working -- until you need to add a new proxy, or reconfigure the setup.
And even then, given that the procedure is well-documented, it
shouldn't be that much added work.

My biggest issue with using existing proxy auto-detection is that WPAD
is just an I-D; there is no standard. Implementation-wise, it's
de-facto a standard -- but I can't comment on the interoperability of
the implementations. So, yeah, the problem needs to be solved, but
browsers technically don't have a standard way of doing it. That said,
any solution that _does_ come up is likely going to be DHCP and/or DNS
based, and locked to the local network (a la zeroconf DNS Service
Discovery).

My second biggest issue is the implied OSI mashing. HTTP is an
application-level protocol, and it should stay that way. Trying to use
it for network authentication is just plain wrong, unless you're going
to actually tunnel a real network protocol over it (in which case, you
can't do it in-browser anymore and may as well use IPSec instead). It
works good-enough in basic cases, but it is NOT the solution that an
ISP should be using. AOL is a perfect example of the very scenario you
describe -- they distribute their own CD, with a browser tweaked with
the required proxy server settings. Now, AOL is not the shining model
of good business practices, but they do seem to have surmounted this
technical problem with relative ease. It's not like there's any
shortage of those damn CDs, so they can't be _that_ hard to make. ;)

-- 
Travis

[1] http://www.w3.org/P3P/

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 Jun 2007 15:15:09 +1200

Travis Snoozy wrote:

<snip>

None of this addresses the fact that

a) intercepting proxies exist, and exist currently unsupported by the spec
b) customers want to be able to *enforce* HTTP policy on their corporate 
networks,
c) customers don't like to have to pay sys admins to do things they can 
get around with technology.
d) the more links you put in a chain, the more chance one of them will 
break.

I'm simply proposing that it might be appropriate that *something* be 
done about this.

Adrien

---

From: Adrian Chadd <[email-masked]>
Date: Mon, 18 Jun 2007 11:31:18 +0800

> d) the more links you put in a chain, the more chance one of them will 

Explain how you can get around these issues with technology without having
at least a skeletal technical services staff to handle issues?

My last client has a 1400 desktop environment running off a single Squid
server, using central-managed browser configuration + proxy.pac for the
actual service configuration. Works fine for their requirement.
They've had no complaints since I updated their Squid environment (since
not touching an environment for 5 years does result in degraded performance;
they had no system administrator for their squid stuff which "just worked"
for said 5 years..)

Adrian

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 Jun 2007 15:55:24 +1200

I'm talking about intercepting proxies, which squid is not.

We do get requests from clients around these sorts of issues.

Many clients (esp SMB) don't have any tech support staff, or their staff 
are not very knowledgeable.

It's one thing for us to say they should have good staff, it's another 
for them to find and pay them.

We get many support requests unrelated to WinGate, regarding DNS setup, 
DHCP setup, proxy auto config etc etc.

Enough support requests to consider it a problem.

For PAC to work properly, you need everything lined up, DNS/DHCP, 
something to serve the .pac file etc etc.

When it works sure it works.  When it doesn't there are a bunch of 
places to start looking, and that's where the support cost comes in.

Regards

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 Jun 2007 16:03:24 +1200

PS, many organisations do have competent IT departments, but they still 
need to justify the cost of things they do.

Going round a bunch of workstations is something that customers are more 
and more reluctant to get their IT staff to do.  They'd rather get them 
to do things that are in their eyes more productive.

It's a recurring theme in comments from customers - we get a lot of 
requests like "if I upgrade will I need to change browser settings or 
any client settings".  That to me signals that it is a concern for 
customers.

Adrien

Adrien de Croy wrote:
> Many clients (esp SMB) don't have any tech support staff, or their 
> staff are not very knowledgeable.
>
> We get many support requests unrelated to WinGate, regarding DNS 
> setup, DHCP setup, proxy auto config etc etc.
>
>>> b) customers want to be able to *enforce* HTTP policy on their 
>>> corporate networks,
>>> c) customers don't like to have to pay sys admins to do things they 
>>> can get around with technology.
>>> d) the more links you put in a chain, the more chance one of them 
>>> will break.
>>>     
>>

---

From: Travis Snoozy <[email-masked]>
Date: Sun, 17 Jun 2007 22:26:42 -0700

> a) intercepting proxies exist, and exist currently unsupported by the

"Intercepting" is merely TCP/IP sleight-of-hand to misdirect/redirect
traffic at the network layer. Because the sleight-of-hand is invisible
to the UA (and any other clients downstream of the invisible proxy), it
has no way of knowing if there's _really_ an invisible proxy between it
and the origin server, or if the origin server or some other
intermediary just wants to hijack the user's session.

Scenario 1 ("what we want"):

UA:   Hey, I want 1.2.3.4:80!
Proxy pretending to be 1.2.3.4:80:
      I'm not really 1.2.3.4:80; you need to add 4.3.2.1 as a
      normal HTTP proxy to get there.
UA:   Uh... is that okay, user?
User: ... suuure?

Scenario 2 ("what we don't want"):

UA:    Hey, I want 1.2.3.4:80!
1.2.3.4:80:
      Um... I'm not really 1.2.3.4:80; you need to add 5.6.7.8
      (totally not a compromised server!) as a proxy to get
      there.
UA:   Uh... is that okay, user?
User: ... suuure?

So, money question: how do we tell these two scenarios apart? That is
the problem that needs to be solved in a secure, airtight fashion. It's
not intractable (the immediate thing that comes to mind is allowing
the redirect to affect the next request on the same address only --
which smells exactly like 305), but it's certainly unpleasant. It may
be more work than just instructing the client to configure their
browser to use the appropriate proxy via a 5xx message.

> b) customers want to be able to *enforce* HTTP policy on their

And ponies ;). But seriously, if you're talking about content filtering
(which I assume you are), "enforced HTTP policy" is about as achievable
as "unbreakable DRM." The Chinese government has sunk untold amounts of
money into their Great Firewall, and it's still routinely circumvented.

Most filtering can be accomplished with a combination of blacklisted
IPs, blacklisted DNS entries that fall under certain domains (with
reverse DNS lookup on IPs), blocked outgoing traffic on certain ports,
and blocked incoming traffic on all ports (with an optional DMZ) -- you
just need a firewall for all of this. Anything more (esp. trying to
filter at the application level, like HTTP) is pretty much a waste of
effort*.

> can get around with technology.

Technology has to be set up by someone. "Turnkey" is hard, hard stuff
in the _home_ network arena, let alone corporate. Yeah, I'm sure that
folks would like to be able to drop a big old SAN box on their network,
and have everything on the network that can see the SAN just start doing
backups magically. I'm sure that it'd be wonderful to just drop a
filtration box on the network, and have no more porn or games being hit
up on work time, and have everything be cached and fast. But, as nice
as that would be, it's totally insecure, and unfit for the enterprise**.

> d) the more links you put in a chain, the more chance one of them

I argue that this service is likely to be more complex implemented in
HTTP than it is to be implemented in a different protocol (like DHCP).
It's likely to be more of a security risk, and cause more trouble in
total than if it's implemented at a non-routeable fashion. While
implementing in HTTP involves one fewer protocol from your perspective,
and may make your job easier, it does not remove a link from the
overall chain (i.e. DHCP is still present and used), or actually reduce
the complexity at all (browsers have to implement a new feature; the
HTTP spec gets even bigger; still have to implement the "bad"
backwards-compatible way).

> I'm simply proposing that it might be appropriate that *something* be 

Oh, indeed, something should be done about it; as you've said, it's
common enough in practice. The question is, where is the most
appropriate place for that something to be? The secondary question is,
even if we want to do that something in HTTP, is it reasonably possible
to accomplish it in a sane and secure manner, without disrupting a
substantial portion of the spec***? The ternary question is,  might we
just write another RFC that specifies the way in which such proxies
(and optionally, user agents) should behave to achieve maximum
interoperability, and simply admit that it's not _really_ HTTP, but
"mostly HTTP compatible"? The final question is, regardless of the
chosen solution, who's going to drive all the work to make sure it gets
done? :)

-- 
Travis

* Not, mind you, that your customers care... as mentioned, it doesn't
stop China from continuing to spend gobs of money. Just make sure that
you have such customers, before putting a lot of effort in.

** Until we can swipe a smart card through it, have the HW's built-in
cert get signed, and through that let all the equipment know that the
HW's been authorized to perform the task it advertises. Mmm... pardon
me whilst I revel in that pipe dream of massive standardization,
interop and ease.

*** Though I'll be the first to tell you I'd like to see the whole
thing rewritten, disruptive changes will be hard, if not impossible, to
get through. O, how I pine for thee, HTTP/2.0!

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 Jun 2007 18:24:53 +1200

> UA:    Hey, I want 1.2.3.4:80!
I think this has been dragged a bit off topic - I'm sure it's my fault
for making a suggestion as to "how" instead of sticking to the "why".

The thing to resolve first of course is the why: is this enough of a
problem to bother doing anything about?  The what and the how comes after.

My opinion based on fielding support queries on this for the last 12
years is that it is significant for users of HTTP.

And what are we here for, why do anything to HTTP at all if it's not
going to in the end benefit the users of HTTP?

> browser to use the appropriate proxy via a 5xx message.
or 4xx message as I originally also suggested.

Anyway, let's not at this stage assume that this would be an impossible
problem.

> as "unbreakable DRM." 
enforced HTTP policy is a lot more achievable than unbreakable DRM.
When you have control over every packet going through the firewall, you
can do quite a lot.

> The Chinese government has sunk untold amounts of

maybe they need WinGate :)

> filter at the application level, like HTTP) is pretty much a waste of

I've a lot of customers and distributors and competitors who would
strongly disagree with you on that one.  Several companies I know of are
making their entire livelihood out of the fact that their customers deem
HTTP filtering to be worth-while.

> as that would be, it's totally insecure, and unfit for the enterprise**.

That's a pretty bold assertion.  I think you'd have trouble convincing
customers about that.

> backwards-compatible way).
We're getting caught up too soon in implementation details, and the
discussion doesn't deserve to be there yet.

Every parameter you assign with DHCP increases complexity.  After IP,
Default Gateway, mask and DNS you start to get into areas (e.g. option
252) which aren't universally implemented by DHCP servers.  Someone with
a network set up with manually assigned IP addresses won't benefit from
DHCP here either, so there are cases where DHCP is a real problem.  Same
with DNS.  There's no sense to it sometimes, but it doesn't help much to
tell customers they are senseless.. you tend to lose their business.

> appropriate place for that something to be? 
good question.

> The secondary question is,
> substantial portion of the spec? The ternary question is,  might we
these are all good questions, and I think this area is where the
discussion should be.

I personally believe that something should be done to recognise the
existence of intercepting proxies, and that the main problems that
exist, such as authentication issues deserve to be looked at.  Whether
the best solution is something to do in HTTP, or something else is
another matter, but we shouldn't be prejudiced.

The issues around a client using server vs proxy request semantics (i.e.
sending full URI, Proxy-Connect, Proxy-Auth tags vs partial URI,
Connect, and WWW-auth tags) can only be resolved if the UA knows it's
going through a proxy.

At the moment, the spec only contemplates explicit proxies (proxies that
are known about by their clients).  So it stands to reason that there is
a place for an intercepting proxy to upgrade its legitimacy by letting
the UA know it is there.  It doesn't matter that some people wont want
this feature.  Some people will.  Passing moral judgements over the
motives of those who may wish their transparent proxies to remain
invisible isn't particularly useful.

As to what happens once a UA learns of the existence of an intercepting
proxy that is happy to be "found out", that's yet another stage.
Resolving trust issues allows a proxy-auto-config type benefit, but
there are likely other benefits as well.  If you are concerned about the
trust issues, try re-working your sample above with more meaningful IP
addresses, like in sample 1 try using a private IP, and in 2 a public
one.  Sure private vs public IPs are crude forms of trust, but given the
un-routable nature of private IPs on the internet, it achieves security
in the same mechanism as you claim for DHCP.  More trust can be obtained
by other methods, say for instance SSL with a certificate.  I think it's
too soon to conclude that trust cannot be achieved.

How a UA learns of the existence of an intercepting proxy (i.e
standards-based notification vs heuristic forensics) impinges a lot on
what can be done with that knowledge.

Also, as other have said, there are currently some artefacts left in
communications by intercepting proxies.  Currently there's no UA that I
know of that lets users know they are going through an intercepting
proxy.  Another area where the spec could acknowledge the existence of
intercepting proxies would be to provide guidelines to UA developers in

a) how to detect the presence of intercepting proxies
b) whether or how detected presence of an intercepting proxy should be
highlighted to the users.

So, there's 2 areas where I think contemplation of intercepting proxies
could provide tangible benefits.  There are probably others.

However, if everyone feels this is not something that is worth pursuing,
then fine, I'll carry on doing it my own way, as will every other vendor
of intercepting proxies, and maybe someone that writes UAs will do
something, and hopefully we will all drift in the same direction, but
probably not :)

Cheers

Adrien

---

From: "Joris Dobbelsteen" <[email-masked]>
Date: Mon, 18 Jun 2007 10:30:39 +0200

>-----Original Message-----
>From: [email-masked] 
>[mailto:[email-masked]] On Behalf Of Adrien de Croy
>Sent: maandag 18 juni 2007 8:25
>To: HTTP Working Group
>Subject: Re: protocol support for intercepting proxies
>
>Travis Snoozy wrote:
>>> a) intercepting proxies exist, and exist currently 
>unsupported by the 
>>> spec
>>
[snip]
>>
>I think this has been dragged a bit off topic - I'm sure it's 
>my fault for making a suggestion as to "how" instead of 
>sticking to the "why".
>
>The thing to resolve first of course is the why: is this 
>enough of a problem to bother doing anything about?  The what 
>and the how comes after.

As of my experience with an intercepting proxy is that the only trouble
I had run into is connecting to servers that are not on the default
ports (they are blocked, unless using the regular proxy). In general, I
found them very useful as not all (read: too few) applications support
*auto*configuration. Now I can just put my computer on any network and
it just works, yet giving me all the advantages of a HTTP proxy.

In fact, here I have WPAD deployed, but IE gives that annoying
"Detecting proxy configuration" message that really gets on my nerves
everytime I want to do something. But this is an implementation mistake,
not a standard's mistake. Yet again it's a small tradeoff.
Alternatively, the autoconfiguration allows me to override one laptops
corporate policy, where they put in a fixed proxy server that is
reachable from only the corporate network, and not from there. With WPAD
you get arround that and don't have to mess arround with the
configuration everytime you use the browser.

What will in fact be the purpose of knowing that you intercepted HTTP
traffic? In fact, at this point the traffic must already get intercepted
(so it won't solve the non-default ports problem) and there must be a
problem doing so (proxy authentication is desired). And to be honest,
proxy authentication from some magic spot will break automatic tools
anyway and requires configuration. The only types of application that
are capable of handling this situation will be user operated ones.

>My opinion based on fielding support queries on this for the 
>last 12 years is that it is significant for users of HTTP.
>
>And what are we here for, why do anything to HTTP at all if 
>it's not going to in the end benefit the users of HTTP?

+1

[snip]

>> even if we want to do that something in HTTP, is it reasonably 
>> possible to accomplish it in a sane and secure manner, without 
>> disrupting a substantial portion of the spec? The ternary 
>question is,  
>> might we just write another RFC that specifies the way in which such 
>> proxies (and optionally, user agents) should behave to 
>achieve maximum 
>> chosen solution, who's going to drive all the work to make sure it 
>> gets done? :)
>these are all good questions, and I think this area is where 
>the discussion should be.
>
>I personally believe that something should be done to 
>recognise the existence of intercepting proxies, and that the 
>main problems that exist, such as authentication issues 
>deserve to be looked at.  Whether the best solution is 
>something to do in HTTP, or something else is another matter, 
>but we shouldn't be prejudiced.
>
>sending full URI, Proxy-Connect, Proxy-Auth tags vs partial 
>URI, Connect, and WWW-auth tags) can only be resolved if the 
>UA knows it's going through a proxy.

Don't understand Proxy-Connect/Connect.
Thinks only Proxy-Auth is a real problem here.

A good intercepting proxy should know to where the connection attempt
was made and most likely is taken from the host header in all other
cases. In fact, even most servers today rely on the value of the host
header.

>At the moment, the spec only contemplates explicit proxies 
>(proxies that are known about by their clients).  So it stands 
>to reason that there is a place for an intercepting proxy to 
>upgrade its legitimacy by letting the UA know it is there.  It 
>doesn't matter that some people wont want this feature.  Some 
>people will.  Passing moral judgements over the motives of 
>those who may wish their transparent proxies to remain 
>As to what happens once a UA learns of the existence of an 
>intercepting proxy that is happy to be "found out", that's yet 
>another stage.
>Resolving trust issues allows a proxy-auto-config type 
>benefit, but there are likely other benefits as well.  If you 
>are concerned about the trust issues, try re-working your 
>sample above with more meaningful IP addresses, like in sample 
>1 try using a private IP, and in 2 a public one.  Sure private 
>vs public IPs are crude forms of trust, but given the 
>un-routable nature of private IPs on the internet, it achieves 
>security in the same mechanism as you claim for DHCP.  More 
>trust can be obtained by other methods, say for instance SSL 
>with a certificate.  I think it's too soon to conclude that 
>trust cannot be achieved.

Start quote:
>And what are we here for, why do anything to HTTP at all if 
>it's not going to in the end benefit the users of HTTP?
End quote

So what is going to be the benefit of recognizing the intercepting HTTP
proxy?

>standards-based notification vs heuristic forensics) impinges 
>a lot on what can be done with that knowledge.
>
>Also, as other have said, there are currently some artefacts 
>left in communications by intercepting proxies.  Currently 
>there's no UA that I know of that lets users know they are 
>going through an intercepting proxy.  Another area where the 
>spec could acknowledge the existence of intercepting proxies 
>would be to provide guidelines to UA developers in
>
>b) whether or how detected presence of an intercepting proxy 
>should be highlighted to the users.
>
>So, there's 2 areas where I think contemplation of 
>intercepting proxies could provide tangible benefits.  There 
>are probably others.
>
>However, if everyone feels this is not something that is worth 
>pursuing, then fine, I'll carry on doing it my own way, as 
>will every other vendor of intercepting proxies, and maybe 
>someone that writes UAs will do something, and hopefully we 
>will all drift in the same direction, but probably not :)

I'm lacking to see the real benefits at this point. Which doesn't mean
that it can't be useful...

Still I believe the major technical issues, one way or another, is going
to be client support and compatibility. But this is currently not the
point.

- Joris Dobbelsteen

---

From: Adrien de Croy <[email-masked]>
Date: Mon, 18 Jun 2007 22:38:06 +1200

Hi

I see 3 main scenarios for how UA knowledge of intercepting proxies 
could provide benefits, and there are likely more that I haven't thought of.

1. intercepting proxy is also the LAN gateway

This is a common scenario.  The proxy will be on the same network as the 
client, and the client will generally consider the IP of the proxy to be 
trusted.  (i.e. can connect to it without using a default gateway).

* Proxy auto-config can be achieved without requirement for DHCP or DNS 
configuration.
* UA can show some UI demonstrating that the connection is going through 
a proxy.
* By effectively stopping intercepting proxy operation by forcing all 
clients to use explicit proxy semantics, proxy authentication issues are 
resolved.

2. Intercepting proxy at the ISP

This is also common in many countries.

* If there's no gateway proxy, and LAN clients are surfing directly, 
then same as 1, if the ISP proxy can be trusted.  If there is one, it 
would most likely be manually configured to use the ISP proxy if that 
was a requirement of ISP policy.
* a tag could be used to request non-intercepted connection, which could 
be evaluated at the ISP proxy based on policy, and honoured or 
rejected.  This would allow ISPs to provide automatic opt-out of 
intercepted connections, i.e. if a request is received by a client with 
a tag requesting non-intercepted connections, and the ISP proxy is 
configured to honour such requests, then it could action its opt-out 
mechanism for that client.

3. Malevolent intercepting proxy somewhere else

Who knows how common this is.

* UA should warn the user if it detects that a connection is being 
intercepted without explicit notification from a trusted IP range or 
with some other mechanism to establish trust.  If there is explicit 
notification, the warning could be different.

One could argue that there's nothing preventing UAs now from attempting 
to determine if connections are being intercepted transparently and warn 
users if they are being intercepted by agents on IPs outside the local 
LAN.  How to determine where the intercepting agent is heuristically is 
of course difficult and gets more difficult the further (in network 
hops) the IA is from the UA, however there is definitely some use in 
knowing if your ISP (or something beyond) is intercepting your connections.

If the UA is unable to determine that there is an intercepting agent in 
the path, then it's in the same boat it is now.

UAs can then take into account whether an IA (intercepting agent) is 
more or less bona fide depending on how forthcoming it is with 
information about itself, however I'd imagine supporting intercepting 
proxies (notified or not) not on the local network or ISP network is 
unlikely to be useful.

At the end of the day, I'm not proposing we embark on work to make the 
security of the protocol reliant on cooperation and goodwill of agents.  
An eventual goal of this proposal could be to enable enhanced operation 
where agents *can* be trusted to co-operate.  Where there is no trust 
between agents, then of course no proxy auto-config should be performed.

Another goal could be guidelines for UAs about security implications of 
intercepted communications.  UAs warn about everything else (often 
needless, or misguided), a warning about connnections being intercepted 
could arguably be more pertinent than many other things UAs warn / nag 
us about.

The spec continuing to ignore the existence of intercepting proxies I 
don't think does the issue justice, especially since HTTP is probably 
the most intercepted/abused protocol.

Regards

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 19 Jun 2007 01:04:11 +0200

sön 2007-06-17 klockan 19:29 -0700 skrev Travis Snoozy:

> de-facto a standard

Yes, on all the above..

>  -- but I can't comment on the interoperability of
> the implementations.

From my experience it's fairly OK provided the admin sets up both DHCP
and DNS properly. Which is not by any means hard to do. The instructions
needed for the average network admin fits on a single page. 

The main problem with WPAD is it's assumption that the client is a full
fledged browser capable of running JavaScript. There is also a need for
something simpler where the client can discover what proxy to use, not
to discover the location of a program/script which it can run to
calculate what proxy to use.

> So, yeah, the problem needs to be solved, but

I certainly would not mind having a HTTP response code where an
interception device could tell the client that "sorry, you need to use a
proxy in this network" asking the client to perform automatic proxy
discovery even if normally not enabled. But having that same message on
the intercepted connection tell the client which proxy to use is
absolutely wrong.

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 19 Jun 2007 01:14:55 +0200

> I'm talking about intercepting proxies, which squid is not.

Isn't it? That's news to me an the countless number of network admins
which uses it that way... (supported since 1998).

> are not very knowledgeable.

True, and they generally either install something prepackaged for them
with support, or have someone make the initial setup for them..

What exactly is the use case here? A network admin not knowing how their
network knows, installing a proxy, and not having any documentation on
the network requirements of installing a proxy? And expecting it to just
work without making any network changes?

> places to start looking, and that's where the support cost comes in.

So there is an opportunity for you to write a support tool telling the
customer why their WPAD setup fails.. and while at it add tests for some
of the other common network misconfigurations..

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 11:31:21 +1200

> which uses it that way... (supported since 1998).
sorry - my mistake.  You'd know :)

I was assuming that squid didn't have packet interception capabilities, 
and that therefore couldn't itself intercept TCP connections. 

But even if it doesn't, I guess the interception part can be provided by 
some other device.

> work without making any network changes?

actually yes.  Insane though it may seem.  Or even no network admin at all.

Any amount of documentation of any quality doesn't make admins read it.

Hotels are a classic case.  You should see the hoops companies like 
Cisco jump through to get hotel guests access through a hotel gateway 
without a human anywhere having to do anything (i.e. work whether or not 
DHCP enabled on the guest laptop by using ARP spoofing and VLANs, work 
whatever subnet the client is on, and whatever the DNS and default 
gateway settings are).

Then try ringing the front desk when you can't get your laptop working.  
They are generally completely clueless. 

> of the other common network misconfigurations..
Sure that's an option.  Actually one we've considered many times.

But in my experience, users are really lazy.  It's a bit of a sad 
indictment on net protocols in general that we still have the problems 
that we do, from ARP, DHCP, IP routing upwards.  The protocols are so 
old, but the people using them now are

a) less informed
b) less kindly towards being confronted with having to actually *learn* 
something

Anything that can be reliably automated is a help.

It's all very well explaining to people that if they wanted to fly a 
helicopter, or perform brain surgery they would need to learn something 
(train even!).  When it comes to using software, people have an 
expectation that they should be able to do anything without having to 
learn a thing.  Educating people is expensive.  Even more so when they 
resist it.  Manuals and documentation are simply not read, and utilities 
are not used.  So the level of automation becomes a competitive factor.

Regards

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 19 Jun 2007 01:51:05 +0200

> for making a suggestion as to "how" instead of sticking to the "why".

And I have only been trying to steer you away from "how", not "why".

Doing network configuration in HTTP is not the right path. Period. There
is other technologies much better suited for the task such as DHCP and
DNS.

> problem to bother doing anything about?  The what and the how comes after.

Yes.

> years is that it is significant for users of HTTP.

Yes.

> going to in the end benefit the users of HTTP?

Now you walk towards the "how" again..

> Anyway, let's not at this stage assume that this would be an impossible

Lets not at this sage consider it a problem of HTTP to be solved by
HTTP. Instead look at the actual problem:

- Admins seemingly lack a reasonable mechanism to get clients to use the
proxy.

In my eyes interception is just a band-aid solution to that problem. If
a real solution to the problem is to be looked for it should be how to
get the client to use the proxy right without having to resort to things
like interception.

The argument that interception is something we have to live with doesn't
hold up as a motivation for adding yet another thing trying to solve
this problem without getting rid of interception.

I would even say it's the use of intercepting proxies which is the cause
to proxy discovery not being fully standardized and interoperable today.
As most admins settle for interception no one demands a well defined
proxy discovery, and instead everyone invests in how to work around the
badness of interception.

> tell customers they are senseless.. you tend to lose their business.

And how is adding yet another HTTP extension any different?

> > appropriate place for that something to be? 

And I strongly maintain that HTTP over an intercepted connection is by
no means the appropriate place to communicate proxy settings. For proxy
settings you need a traceable chain of trust to at least the level of
DHCP, and that's not provided by an intercepted connection.

> another matter, but we shouldn't be prejudiced.

I don't agree this should go into HTTP.

But I am all for making a standardized proxy discovery mechanism if
there is industry interest for it. Main requirement for such task is
that the major browser and HTTP client library vendors is interested.

I am also for documenting the issues with interception, and commonly
deployed solutions to those issues, and the problems with those
solutions. But not as part of the HTTP specifications but rather as a
separate informal document.

> Connect, and WWW-auth tags) can only be resolved if the UA knows it's

Yes, and not only that it's going through a proxy but also that it can
trust that proxy.

> are known about by their clients).

Yes, and that's all the HTTP spec should care about imho.

> So it stands to reason that there is
> the UA know it is there.

As already said RFC2616 requires them to let the UA know it is there.
The standard can't be blamed for proxy vendors not implementing this.
It's already there, adding yet another way for the proxy to indicate
it's presence won't improve the situation, just make it worse.

> proxy that is happy to be "found out", that's yet another stage.

Is it?

> know of that lets users know they are going through an intercepting

That's an implementation detail. There is no UA interested in bothering
users with this technical detail which usually has little or no impact
on his experience.

> Another area where the spec could acknowledge the existence of

I argue that it's already there. Just a matter of making use of the
information already available.

> highlighted to the users.

I don't think you will find many in favor of alerting the user..

> something, and hopefully we will all drift in the same direction, but

I hereby call you to look into the problem of why interception is
needed, and to try to come up with ideas to address that problem. In the
long run a proper solution to that problem will benefit you and everyone
else.

Trying to solve this in HTTP is short-sighted and doomed for failure
imho.

Regards
Henrik

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 12:48:00 +1200

> long run a proper solution to that problem will benefit you and everyone
I can start with why we implemented an intercepting proxy.  Apart from 
commercial / competitive issues that is.

Basically the main thing we identified was that customers wanted it.  
Feedback was pretty explicit on this.

Why they wanted it, I guess is because proxy auto-detect or proxy 
operation wasn't doing it for them. 

Proxy auto-detect
------------------
There are plenty of our customers that use proxy auto-detect and are 
quite happy with it.

However, proxy auto detect can require some configuration.  Originally 
proxy auto-detect worked by doing a DNS lookup for the name "WPAD".  
WinGate's DNS server would respond to this with its own IP address.  The 
client would then make an HTTP request for a file called wpad.dat.  We 
would auto-generate this and send it to the client.  So, basically 
customers got proxy auto-detect for free.  I presume other vendors 
adopted a similar approach.

then someone at MS decided DHCP would be a better option for discovering 
proxy config, and option 252 was created whereby the DHCP client issues 
a request for option 252 as part of a DHCP_INFORM message.  DHCP_INFORM 
was a late addition to the DHCP spec, also option 252 is still listed by 
the IANA as "private".  So there are a bunch of DHCP servers out there 
that don't support DHCP_INFORM nor option 252.

So, we have some browsers that use DNS (which they can control, but 
which seems to be being deprecated), and others that rely on their OS 
platform using a private DHCP option that is not formally specified.  
Some client platforms don't provide a mechanism where applications (such 
as a browser) can obtain info from DHCP clients or servers (i.e. 
influence DHCP requests, and access response data).  Most notable OS 
that springs to mind is most if not all versions of windows.

finally, enabling or disabling proxy auto-detection is under user 
control.  Heaven forbid!

I'm not aware of any browsers that do a sanity check on the values that 
get returned from a WPAD DNS lookup, or the URL from DHCP option 252 
either.  We just presume that the DNS and DHCP server responses are 
trustworthy.

So in short, IMHO proxy auto detect is fraught with problems.

Proxy operation
----------------
There are zillions of applications out there that use HTTP for whatever 
purpose.  Many of these are cobbled together.  Many do not consider the 
existence of proxies at all, or can't do proxy auth, or any sort of HTTP 
auth.  Many do not present any UI, but operate as background service 
processes.  Thankfully with APIs like MS INETAPI on Windows, developers 
now have an easier way to write such things, and they gain support for 
things like chunking, auth etc.  But there are still a bunch of broken 
apps out there.

Couple this with the sys admin that still wants to force everything 
through the proxy anyway, and your only option is connection 
interception, where the proxy pretends to behave like a server (right 
down to the header level).

Many of these apps are very old, and aren't being maintained, but are 
still in use and relied on.

This isn't an argument for changes to the protocol actually since 
proposed changes would break all these apps.

> Trying to solve this in HTTP is short-sighted and doomed for failure
Actually I see the configuration of HTTP proxy as being an HTTP protocol 
configuration element, and therefore I don't see why it shouldn't be 
solved in HTTP.  It's not like we're setting a parameter unrelated to 
HTTP.  Pushing the config out to other services multiplies the number of 
applications that need to be maintained.  A sys admin doesn't need to 
keep just their proxy and clients up to date, but also their DHCP 
servers and other servers.  If these are all built into a hardware box, 
there aren't many options.

You might laugh, but one of our biggest support issues used to be people 
configuring their proxy config to use port 21 for FTP proxy in IE.  They 
"knew" that port 21 was the port to be used for FTP.  In the end, we had 
to write an HTTP handler in WinGate's FTP proxy to tell them to change 
their config when it detected an HTTP request for an FTP URL.

There are lots of people out there with a little knowledge, enough to 
make it dangerous to get them to do any sort of manual configuration.

If we could assume that all HTTP agents could work through a proxy, and 
do proxy auth, we should be looking at a more foolproof mechanism for 
proxy auto configuration.  The form that takes has many possibilities, 
you don't like the one proposed so far, that I can live with.  But there 
are potentially others that would solve concerns, and achieve the goals 
of minimum impact on other systems.  The fewer the links in the chain 
IMO the better.  So firstly I think such a mechanism should be

1. under network not individual control (so the network can set the 
requirement to use the proxy, not the client, and it can be managed 
centrally).
2. not reliant on DHCP.  Why must we force everyone on the planet to use 
DHCP just to get an auto proxy config.  Why must we force all OS vendors 
to open up their DHCP clients so that applications can obtain DHCP 
configuration data.

So maybe DNS SRV records are the way to go here.  Some customers lock 
down even DNS though, since the proxy does the DNS lookup for normal 
proxy operations.  Also most OS provided DNS resolver implementations 
(i.e. winsock) only support A record lookups, so unless you want browser 
authors to write their own DNS resolvers (and deal with the 
configuration issues around that), they won't have much luck on many 
OSes even getting a SRV record.

the one common factor that all browsers have is HTTP support.

In the end, the best solutions may be platform-specific, i.e. Active 
Directory policy based, or extensions to network logons.  But I'm 
struggling :)

Adrien

---

From: Lisa Dusseault <[email-masked]>
Date: Mon, 18 Jun 2007 18:30:53 -0700

> is other technologies much better suited for the task such as DHCP and

Not that I'm disagreeing with you but...

http://www.ietf.org/rfc/rfc4743.txt

lisa

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 13:33:56 +1200

PS, there are several types of intercepting proxy as well.

To date, discussions have been centred on those that sit on another 
machine (usually a gateway), and divert TCP SYN packets etc.

However, on Windows, there are other types, using either TDI filtering 
or Winsock2 Layered Service Provider approaches to "break into" the HTTP 
data stream.  The purpose of these is also commonly the same as 
filtering at a gateway - i.e. antivirus, or content control

most big-name AV companies install technology like this in their 
internet security suites, as well as many net nanny type apps.

These apps have other ways to make themselves known to the user of the 
machine, since they reside on it, but there may be considerations that 
could apply to them as well.

Cheers

Adrien

---

From: Mark Andrews <[email-masked]>
Date: Tue, 19 Jun 2007 11:35:56 +1000

> operation wasn't doing it for them. 

	To be perfectly honest.  Applications SHOULD have access to
	DHCP configuration results. 

> OSes even getting a SRV record.

	Most OS's support looking up any record you can think about.

	For Windows you need DNSQuery*().  Even the embedded versions
	of DNSQuery_W support SRV lookups.
	http://msdn2.microsoft.com/en-us/library/ms886653.aspx

	The problem with SRV is that you need to know the query name.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 13:46:31 +1200

> 	DHCP configuration results. 
I wish :)

They should also be able to instigate DHCP_INFORM requests to request 
specific parameters from the DHCP server as well.

I don't see it happening any time soon though

> 	http://msdn2.microsoft.com/en-us/library/ms886653.aspx

Interesting - thanks for that.

I can only find documentation on MSDN for this for CE and Windows Mobile 
though (trying to find out OS support).  We had no end of trouble trying 
to get a generic DNS resolver going for all versions of win32.  Even 
finding the DNS server to use is non-trivial on Win95.  That's the main 
reason WinGate has a spot where you can configure the DNS server to 
use.  I'd hate to see the day where browsers have to put up a UI for the 
DNS server to use for the SRV record lookup for proxy auto config :)

But thankfully win95 is mostly dead now anyway, except for in some less 
rich countries.

---

From: Travis Snoozy <[email-masked]>
Date: Mon, 18 Jun 2007 18:51:47 -0700

I'm going to try and be terse today. :)

On Tue, 19 Jun 2007 12:48:00 +1200, Adrien de Croy <[email-masked]>
wrote:

> However, proxy auto detect can require some configuration.
> Originally proxy auto-detect worked by doing a DNS lookup for the
> name "WPAD".
<snip>
> then someone at MS decided DHCP would be a better option for
> discovering proxy config, and option 252 was created whereby the DHCP
> client issues a request for option 252 as part of a DHCP_INFORM
> message.
<snip>
See the friendly notice at http://www.wpad.com/ for why this was a
REALLY bad idea. He could be hijacking a whole lotta browsers right
now, if he weren't so nice.

<snip>

http://www.wpad.com/draft-ietf-wrec-wpad-01.txt states that option 252
must be supported by clients. Yeah, yeah; not a formal spec, but if
you use part of it, you're in for the whole nine yards.

> Some client platforms don't provide a mechanism where applications
> (such as a browser) can obtain info from DHCP clients or servers
> (i.e. influence DHCP requests, and access response data).  Most
> notable OS that springs to mind is most if not all versions of
> windows.
<snip>

Ironic, ne?

> So in short, IMHO proxy auto detect is fraught with problems.

Totally agree.

> Proxy operation
> ----------------
> There are zillions of applications out there that use HTTP for
> whatever purpose.  Many of these are cobbled together.  Many do not
[follow the standard] ...
<snip>

Yep. And HTTP is a crummy standard at that. It's huge, it's
self-conflicting & ambiguous, it's hard to implement, and there isn't a
reference implementation.

> proposed changes would break all these apps.

Yeah, backwards compatibility is a bitch. It's one of the biggest
reasons for the current HTTP mess.

> If we could assume that all HTTP agents could work through a proxy,

We do.

> and do proxy auth,

We don't (it's a MAY level thing).

> we should be looking at a more foolproof mechanism  for proxy auto
> configuration.

Agreed.

(hit my length limit)

-- 
Travis

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 14:38:23 +1200

> I'm going to try and be terse today. :)
OK, me too.

> now, if he weren't so nice.
actually I only ever saw lookups for "wpad", none for "wpad.com".  I 
guess there was a time where DNS resolvers were pretty gung-ho about 
automatically appending local domain suffices to all lookups.  actually 
they still are.  But yeah, DNS is wide open to abuse like this, since 
you never know where the record will come back from.

And if that page is anything to go by, we can probably expect that a LOT 
of browsers are configured not to use proxy auto configuration.  
Typically upgrades to the browser honour previous settings like this, so 
we have a sizeable migration issue if we want to push WPAD out due to 
control being client-side.

> you use part of it, you're in for the whole nine yards.
It also suggests that browsers writers write minimal implementations of 
DHCP clients into themselves.  I see a few problems with this, firstly 
that many DHCP servers will only send replies back on port 68, which the 
browser may not be able to reliably bind to (since the OS DHCP client 
already is).  Sure, you can disable binding conflicts, but in my 
experience you don't reliably receive the data.

Plus then you get a zillion different implementations of DHCP clients in 
various states of bugginess.  Sure, the nice engineering solution to 
this is to require authors of DHCP clients to put in APIs to allow 
clients to get data.  Having written a DHCP server though, I can attest 
that it is probably the ugliest protocol, closely followed by DNS then 
probably IMAP (all of which I have written servers for, so have a small 
amount of experience with).

All the chain of WPAD query types must add up to a lot of timeouts :)  
If you take your laptop to a network that doesn't implement ANY WPAD 
option, you'll be waiting a while.

<snip>

I thought of one more reason for interception: To force use of the proxy.

If you're an admin and you have a gateway and firewall and proxy, what 
are your options if you want to force everyone through the proxy for say 
port 80?

You can block access to port 80 through the firewall except for the 
proxy.  You can enable option 252 in your DHCP server, enable WPAD in 
your DNS server etc etc.

However, the client can circumvent all of this.  By manually setting 
their browser not to use a proxy, and not to use WPAD, then they will 
try connecting directly out on port 80.  Ok, so you're blocking this, 
what does the client see?  Connection timeout.  Not very friendly.  
Results in calls to IT department.

In order for a sys admin to be able to send any meaningful response back 
to the user and UA, HTTP protocol must be used, and this can only be 
done if you intercept/divert the connection to something that will take 
the connection, and talk HTTP.

Even if the only purpose on the planet for an intercepting proxy was to 
send a page back to a client saying "smack! naughty you must use a 
proxy", then that is still a valid and useful purpose.   A slightly more 
helpful proxy could even provide a page documenting how to turn on proxy 
auto-config.  If you take that line to its logical conclusion, the 
client will automatically set the proxy to use.  This is what the 
customers will be asking for.  They will say "I don't care, just solve 
the security issues, why keep giving me more tasks to do when you can 
get the software to do it for me".  I personally believe that in certain 
cases, the security issues can be addressed.  I don't think anyone here 
is proposing that it's impossible in all cases to do this securely.

Adrien

---

From: Mark Andrews <[email-masked]>
Date: Tue, 19 Jun 2007 16:26:34 +1000

> I don't see it happening any time soon though

	Some OS already do this.  As a DHCP vendor (client and server)
	improving integration with the rest of the OS is on our roadmap.

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 19:09:39 +1200

I've just spent most of the day reading DHCP RFCs.  There are still 
numerous problems with DHCP, even apart from issues with current 
implementations.

The home page of the DHC WG outlines a number of issues, the one at the 
top of their list being "address security in DHCP". 

http://www.ietf.org/html.charters/dhc-charter.html

They state no-one has implemented RFC 3118 (DHCP-AUTH).  Given that RFC 
3118 has been out for 6 years, that may have something to do with trying 
to auth at too low a level in the stack.  Same reason IP auth options 
were chucked out.  Auth fundamentally requires sharing of secrets.  It's 
unmanageable to share secrets (esp on a large scale) without using 
networking protocols (i.e. you'd need to use sneakernet).  Therefore 
DHCP can't effectively or efficiently be authed, since it sets up the 
networking protocols that would be used for sharing secrets, and 
therefore "DHCP Auth" is a chicken-and-egg paradox.  You'd need an 
ethernet (non IP) level key management / auth subsystem to auth DHCP.  
One that can cross subnets.  Since most routers are IP routers, ethernet 
level is a non-starter as well.  You really need an IP level or higher 
protocol for auth.

So I don't see security issues with DHCP being solved any time soon.  
It's a hard problem.

Furthermore when they created the DHCPINFORM message, they neglected to 
make it usable by client applications other than the OS DHCP client 
(since they didn't relax the requirements for the destination port that 
the unicast DHCPACK would be sent to - it woulda been sooo easy to allow 
that to be sent back to the source port of the agent that sent the 
DHCPINFORM message, but no - port 68 is it).  That's only the start of 
the problems with the DHCPINFORM message as well - a DHCP server is even 
prohibited from making certain security decisions about whether it will 
reply (prohibited from verifying that the DHCP client is a known client).

The relevance here is that WPAD's primary method is based on the 
DHCPINFORM message.

In my experience, opportunity for reliable security improves as you move 
up the stack (*), and degrades as you go down it.

For instance, ARP is an extremely insecure protocol, and a rogue ARP 
responder can wreak absolute havoc on a network.  A rogue DHCP server 
can also wreak havoc.

But anyway this is the HTTP WG mailing list not the DHC one :)  And 
you've all heard more than enough from me today I'm sure :)

All I'm getting at is that people are making arguments about security of 
configuration, and making assertions about superior suitability of 
protocols such as DHCP and DNS in that same context. 

I would argue that HTTP has much more capability of being much more 
secure over a much broader realm than DNS or DHCP ever will have, and 
that therefore maybe HTTP is a more suitable protocol to be used for 
certain configuration tasks, especially more security-sensitive ones. 

Cheers

Adrien

(*) E.g. signing of emails, SSL certificates etc.  TLS is a special case 
- it's really an application level protocol, sitting as it does over 
TCP.  IPSEC would be the closest I guess to packet level security.  It's 
hideously over-burdened in terms of administration though, which I 
believe has hampered its widespread adoption in LAN operation (apart 
from the obvious performance issues).

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 19:14:51 +1200

Adrien de Croy wrote:
>
> They state no-one has implemented RFC 3118 (DHCP-AUTH).  Given that 
> RFC 3118 has been out for 6 years, that may have something to do with 
> trying to auth at too low a level in the stack.  Same reason IP auth 
> options were chucked out.  Auth fundamentally requires sharing of 
> secrets.  It's unmanageable to share secrets (esp on a large scale) 
> without using networking protocols (i.e. you'd need to use 
> sneakernet).  Therefore DHCP can't effectively or efficiently be 
> authed, since it sets up the networking protocols that would be used 
> for sharing secrets, and therefore "DHCP Auth" is a chicken-and-egg 
> paradox.  You'd need an ethernet (non IP) level key management / auth 
> subsystem to auth DHCP.  One that can cross subnets.  Since most 
> routers are IP routers, ethernet level is a non-starter as well.  You 
> really need an IP level or higher protocol for auth.
>

PS, even though obviously DHCP sits over UDP which sits over IP and 
therefore is a higher level protocol than IP, it configures IP, and IP 
is dependent on it, and therefore in this context it is effectively 
lower level - there's doubtless a better phrase to describe this.

---

From: Adrian Chadd <[email-masked]>
Date: Tue, 19 Jun 2007 15:25:55 +0800

> level is a non-starter as well.  You really need an IP level or higher 

Its not more difficult to setup than shared keys for WPA-PEAP IIRC.
Group Profiles/Active Directory has already solved this problem for
distributing authentication keys (at least in the Windows world.)

(Not that this is a workable solution for -everyone-, but certainly
in the corporate environments you're talking about..)

Adrian

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 19:30:52 +1200

Actually that proves my point.

this is an example of security problems inherent in low-level protocols 
being solved using high level protocols, e.g. SSL certificates, key 
exchange protocols etc.  All of which require the IP config to be 
working, which therefore already required DHCP to be working without 
auth.  So, it pretty much makes DHCP auth pointless.

Adrien

---

From: Adrian Chadd <[email-masked]>
Date: Tue, 19 Jun 2007 15:35:48 +0800

> auth.  So, it pretty much makes DHCP auth pointless.

.. and you can push out centralised HTTP proxy server settings via
the same mechanisms. Admittedly its only one platform and I'm not sure
when the AD/Group Policy support popped up, but its certainly doable.

Me, I'd prefer to see the proxy discovery draft properly worked into
an RFC as there are -plenty- of instances of WPAD being used in the
real world these days. It'd also be nice to have it extensible to proxy
other protocols, such as P2P client proxy discovery (when P2P caches
become all the rage, that is..)

Adrian

---

From: Adrien de Croy <[email-masked]>
Date: Tue, 19 Jun 2007 21:58:15 +1200

Yeah, this all points to some sort of generic config protocol.

I think there was a config protocol that Eudora used a while back - it 
didn't catch on.  You'd need something more like SNMP, where 
configuration points are registered under a global UID schema, or LDAP.  
Anything that can have a schema and data.

DHCP is just so restricted it's hard to see why so much effort is still 
being poured into it.  There are now more options than you can 
realistically fit into a 576 byte single UDP packet.  All to do config.  
A separate config protocol or framework would be much easier to grow.  
Use DHCP to discover the config server and go from there.  That's 
basically what AD does, uses DHCP to find the DNS server to do SRV 
lookups on to find the LDAP server.

As long as there is decent client-side API support so that UA developers 
can hook into it then the solution to auto-config issues is doable.

However, to quote Donald Rumsfeld, there are still the things we don't 
know that we don't know.  I'm still open to the possibility that 
considering intercepting proxies could have some benefits, even if one 
of those benefits doesn't turn out to be auto-config.  That requires 
contemplation of the issue.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 20 Jun 2007 00:35:00 +1200

>> notable OS that springs to mind is most if not all versions of
I'm going to have to take it back.  Looks like MS implemented a DHCP 
client API since Windows 2000.

http://msdn2.microsoft.com/en-us/library/aa363344.aspx

Adrien

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 19 Jun 2007 16:41:02 +0200

> operation wasn't doing it for them. 

That's my guess as well, and I propose fixing that instead of trying to
work around it by adding yet more questionable "hacks".

> However, proxy auto detect can require some configuration.

So will whatever HTTP extension signaling this on a transparently
intercepted connection. For a starter it's quite unlikely to be enabled
by default due to the security implications of not having a reasonable
chain of trust.

> Originally 

Yes. Well, it's using the normal DNS domain search list from what I can
gather.. But I suppose that if the clients is configured without a DNS
domain then you'll get lookups for just "wpad".

> that don't support DHCP_INFORM nor option 252.

It's also usable in the DHCPOFFER, and is where it's normally used.

DHCPINFORM is mainly for hosts not using DHCP for the IP configuration
but still needing the additional configuration details from DHCP. It's
DHCPREQUEST minus the IP allocation and lease checks.

> finally, enabling or disabling proxy auto-detection is under user 

Don't see it as a protocol problem. Administrators have loads of tools
to control this is they desire. Sure, requires administrators to
actually care about their environment and locking down the users
preventing them from doing (in his opinion) stupid things, but is not a
protocol issue.

Yes.

> So in short, IMHO proxy auto detect is fraught with problems.

On the local LAN yes, but it's confined there. And if the LAN is
upgraded with DHCP security or DNSSEC then WPAD also benefits from that.

> things like chunking, auth etc.  But there are still a bunch of broken 

And there will always be. Adding yet another extension for proxy
discovery won't help that at all.

> down to the header level).

It's not the only option. There is several ways of enforcing the use of
a proxy. Interception is the most common as it doesn't require much
thinking to deploy.

> still in use and relied on.

Yes, and will with an absolute certainty not be upgraded to support any
new extension either, so I don't get why you bring this up in the
context of extending the awareness of intercepting proxies.

> If we could assume that all HTTP agents could work through a proxy

You can, by the specs.

> and 
> do proxy auth

Optional, and unrelated.

> we should be looking at a more foolproof mechanism for 
> proxy auto configuration.

Yes.

> The form that takes has many possibilities, 
> you don't like the one proposed so far, that I can live with.

Good.

> But there 
> of minimum impact on other systems.

Sure. And what I think efforts should be focused on.

In my view the DHCP track is pretty good even if WPAD has it's
limitations. SLP is also reasonable but less common.

> The fewer the links in the chain 
> IMO the better.

Yes, but there needs to be a traceable chain.

>   So firstly I think such a mechanism should be
> 

Yes, which calls for DHCP as it's the closest to the network level..

> to open up their DHCP clients so that applications can obtain DHCP 

DHCP is the accepted standard for local LAN network configuration
distribution.

> proxy operations.

No problem to run a small internal DNS in such networks, and many do.

> Also most OS provided DNS resolver implementations 

Correct. For manual SRV lookups you need to use a DNS client, not an
abstract socket interface.

In Windows this is the DNS Functions..
http://msdn2.microsoft.com/en-us/library/ms682058.aspx

Regards
Henrik

---

From: Henrik Nordstrom <[email-masked]>
Date: Tue, 19 Jun 2007 16:51:55 +0200

> Yeah, this all points to some sort of generic config protocol.

Which is what DHCP, SRV and SLP all are in the discussed context, at
different levels..

> being poured into it.

Because it serves the purpose of distributing network local
configuration details quite well.

> There are now more options than you can 
> realistically fit into a 576 byte single UDP packet.

And is why DHCP allows for larger messages (up to 64KB), if the client
says it supports it..

> All to do config.  

SLP?

Regards
Henrik

---

From: Jamie Lokier <[email-masked]>
Date: Tue, 19 Jun 2007 20:30:45 +0100

> could apply to them as well.

I've seen a few Linux boxes doing local interception too, to provide a
HTTP cache which is shared among multiple users of the machine,
without the individual users having to configure their choice of
browser.

In these cases it was purely to improve the perceived speed of web
access and reduce bandwidth costs.  The users couldn't be expected to
configure anything themselves, and the admin couldn't be expected to
configure each user's environment as they used different browsers, as
well various non-browser HTTP applications unknown to the admin.

Locally served automatic proxy detection might have worked instead of
interception, but that would probably have been quite a lot more
difficult to implement, and still would not have worked with some HTTP
using programs.

-- Jamie

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 20 Jun 2007 12:12:18 +1200

> configuration details quite well.
I wouldn't say quite well.

* negotiated options to overload server->Client message length that 
doesn't even consider client->server message length, and means we have 
to upgrade not only servers and clients to cope with bigger packets, but 
also *network infrastructure* such as routers (relay agents), subscriber 
line modules and managed switches.  Good one.
And results in support burden for vendors and customers, as evidenced by 
http://support.microsoft.com/kb/321592
Even with negotiated size, max is still 64k.  Evokes memories of Bill 
Gates' comments about 640kb being more memory than anyone will ever need :)

* option splitting and concatenation to get around boundary issues of 
the 3 (2 of which are overloaded and must be signalled for use) storage 
areas.
* a single byte for the option id?  so only 255 options.  I guess that 
will be overloaded too.
* all options have been designed to save space in the packet. Many 
options don't have standardised implementations (i.e. relay info).

None of this comes across to me as a protocol particularly well suited 
to the scalability issues that confront it.  There is a large list of 
current I-Ds related to DHCP.  Everyone is trying to shoe-horn things 
into it.  It doesn't have the skeleton for the amount of flesh being put 
on it.  It kinda reminds me of our use of fossil fuels, everyone is 
rushing to get some while there's still any left :)

I think we should take the lead of the designers of the protocol.  They 
weren't dumb.  They only allowed a single byte for the option, because 
the protocol fundamentally was only designed to support a relatively 
small number of options - enough to get IP up and running.  That 
original design tenet has been since ignored, and we are now paying for 
ignoring it.

> And is why DHCP allows for larger messages (up to 64KB), if the client
I've not seen such a client advertisement yet (up to XPSP2, haven't 
checked Vista DHCP client yet).  I think the infrastructure issues are 
the biggest ones.  Even if those are solved, we will eventually run out 
even with 64k.  The sooner an extensible system is adopted, the less 
pain there will be down the track with people having to migrate to it.

>> A separate config protocol or framework would be much easier to grow.  
Thanks - I need to read more on it.  LDAP is the one that springs to my 
mind though, since isn't SLP only for service location, rather than 
dissemination of configuration data?

Anyway, I think it could be more useful for me to just sketch out a 
proposal about what the results of contemplation of intercepting proxies 
could be, and why.  We got hung up on the auto-config option, but I 
think some guidelines to implementors and operators of intercepting 
proxies, and UA developers could still be useful.

Adrien

---

From: Adrien de Croy <[email-masked]>
Date: Wed, 20 Jun 2007 13:09:58 +1200

interesting to see this on cnet.

http://news.com.com/2100-1002_3-6170229.html?part=rss&tag=2547-1_3-0-20&subj=news

seems coincidentally topical.