Subject: #385: HTTP2 Upgrade / Negotiation

From: Mark Nottingham <[email-masked]>
Date: Tue, 23 Oct 2012 13:03:35 +1100

<http://trac.tools.ietf.org/wg/httpbis/trac/ticket/385>

This issue is how we upgrade from 1.x to 2.x. I'd suggest we consider the issue in two parts, for HTTPS URIs and HTTP URIs. Furthermore, we need to figure out how we'll identify the protocol that's in use as we develop it.

HTTPS URIs
==========

Based on previous discussion and our charter, I think the path forward for negotiating HTTP/2.0 for https:// URIs is pretty clear; we need to make a request to the TLS Working Group, asking for a in-protocol negotiation mechanism for the "next" protocol to use.

The question for us right now is what requirements we want to place upon that work. Currently, I have:

---8<---
TLS Working Group Chairs,

This is a request from the HTTPbis Working Group for you to commence work upon a mechanism that allows clients and servers to negotiate the particular application protocol to use once the session is established.

Our use case is for HTTP/2.0 in conjunction with HTTP URIs; rather than defining a new port, which incurs both performance and deployment penalties, a negotiation mechanism would allow for better deployment of HTTP/2.0 for HTTPS URIs.

We would expect such a mechanism to allow the client and server to negotiate the use of one of potentially many such protocols (in our case, HTTP/1.x and HTTP/2.x), identified by tokens, and falling back to a default for the port in use (in our case, HTTP/1.x) when either side doesn't support negotiation, or an agreement can't be found.

We also note existing work in this area:
  http://tools.ietf.org/html/draft-agl-tls-nextprotoneg-04

The HTTPbis Working Group will be happy to coordinate schedules, review drafts and provide further input as required.

--->8---

Does anyone have further input? I'd like to get the request to the TLS WG very soon.

HTTP URIs
=========

For HTTP URIs, the field is much more open.

I think the basic requirement is that it be possible to negotiate use of:
  - HTTP/1.x
  - HTTP/2.x
  - HTTP/2.x over TLS, possibly with caveats around cert checking, etc.

for HTTP URIs. Note that we may or may not define the third "mode", but the negotiation mechanism needs to be able to accommodate it, as per our charter.

There are a *lot* of ways this could happen. 

One suggestion has been made by Gabriel and Willy:
  http://tools.ietf.org/html/draft-montenegro-httpbis-http2-negotiation-00
... which is basically a good explanation of how HTTP Upgrade was intended to work. Notably, it does not introduce a roundtrip. 

One of the major perceived issues with this approach is interference by broken proxies (i.e., those that don't honour the hop-by-hop status of Upgrade). 

I know that some experiments were done with this in the HYBI work, but I'd like to do some more, to get an idea if it's possible to mitigate some of this risk (e.g., by sending "Connection: Upgrade"). Based upon that, we can make an informed decision about this approach.

Who's willing to do some experimentation? Specifically, does anyone have access to the code that was used before (IIRC, people bought some ads and inserted some Java to probe the network)?

Other approaches suggested as alternates and/or compliments to this include:

1) Using a SRV (or other DNS) record. There seems to be a fair amount of interest in this from the browser implementers; the only resistance seems to be around how hard this would be to deploy, but if it's not the only mechanism for upgrading to HTTP/2, it shouldn't be a concern. 

Presumably, we'd define a default port for HTTP/2-only communication when used in conjunction with this, so that firewalls would know to open it, etc.

Does anyone object to us defining such a record (type TBD), as long as it's not the only way to get to HTTP/2 for HTTP URIs?

2) Using a response header to hint that HTTP/2 is available on another port. 

This approach hasn't been talked about in detail yet, but it apparently (as some have noted) has the disadvantage of not upgrading the first interaction, and of requiring a separate cache (and caching model) for this information.

Who wants to pursue this approach? If so, we need a small proposal written, preferably as an I-D, but an e-mail would do.

And, are there any other approaches we want to put on the table?

Identifying The Protocol in Development
===============================

Finally, I'd like to make sure we understand how we're going to identify this protocol as we move forward.

The NPN, Upgrade and Alternate-Protocol approaches all (should, if we go there) use a token to identify the protocol being upgraded to, meaning that we'll be able to do something like

  http2-draft-3
  http2-draft-5

when negotiating the protocol while we're developing it; as long as implementations deploy the latest version and perhaps the one before it, they should be able to interoperate, and fall back to HTTP/1 if not. The browsers have used this approach for SPDY, and it seemed to work well.

To me, this implies that if we define a DNS-based mechanism to hint upgrade, it should have a similar capability, so we can reuse those tokens and this approach there too. Make sense?

Concurrently, the SPDY draft uses 15 bits to identify the protocol in use. Provided we get the upgrade mechanism right, I'm less concerned about this, but it would be good if the messages were self-describing, so we should figure out what the appropriate values to put in here are too. And, maybe saving a few bits too.

Cheers,

--
Mark Nottingham   http://www.mnot.net/

---

From: William Chan (陈智昌) <[email-masked]>
Date: Mon, 22 Oct 2012 19:33:09 -0700

Overall, sounds good. I've included some clarifications/questions below.

> This issue is how we upgrade from 1.x to 2.x. I'd suggest we consider the issue in two parts, for HTTPS URIs and HTTP URIs. Furthermore, we need to figure out how we'll identify the protocol that's…

Do you mean the Chromium HTTP upgrade experiment agl referred to in
http://www.ietf.org/mail-archive/web/tls/current/msg05593.html?

> Does anyone object to us defining such a record (type TBD), as long as it's not the only way to get to HTTP/2 for HTTP URIs?

I'm not sure if my previous emails were taken to indicate "interest"
here. I forget what I said too :) As long as this is more of an
optional optimization than anything, I guess I'm OK with it. I'm very
much concerned about relying on it, due to experiments we have run
with TXT records that show us noticeably higher failure rates in
comparison to port 443, higher latency (we'd definitely have to race
this), and extra DNS queries.

> This approach hasn't been talked about in detail yet, but it apparently (as some have noted) has the disadvantage of not upgrading the first interaction, and of requiring a separate cache (and cach…

Just to be clear, SRV records also have the disadvantage of not
upgrading the first interaction, unless you block on the response,
which Chromium definitely is not going to do unless the environment
changes such that it doesn't kill performance.

---

From: Mark Nottingham <[email-masked]>
Date: Tue, 23 Oct 2012 13:50:46 +1100

> Overall, sounds good. I've included some clarifications/questions below.

[...]

> http://www.ietf.org/mail-archive/web/tls/current/msg05593.html?

No, IIRC there was also some broader experimentation using ads; I'll dig around a bit more. 

Of course, if Chrome (or any other browser) would be interested in running an experiment, we'd love to do that too -- provided we can have input into the design.

> this), and extra DNS queries.

Yes, I think relying on it would be a mistake, for a variety of reasons.

> changes such that it doesn't kill performance.

I'll leave it to the DNS experts to debate the capabilities and merits here.

---

From: James M Snell <[email-masked]>
Date: Tue, 23 Oct 2012 09:50:24 -0700

On Mon, Oct 22, 2012 at 7:03 PM, Mark Nottingham <[email-masked]> wrote:

> [snip]
>
> 1) Using a SRV (or other DNS) record. There seems to be a fair amount of
> interest in this from the browser implementers; the only resistance seems
> to be around how hard this would be to deploy, but if it's not the only
> mechanism for upgrading to HTTP/2, it shouldn't be a concern.
>
> Presumably, we'd define a default port for HTTP/2-only communication when
> used in conjunction with this, so that firewalls would know to open it, etc.
>
>
Such a dedicated default port definitely needs to be defined. There are
many scenarios where upgrade negotiation simply is not going to be required
at all.

> Does anyone object to us defining such a record (type TBD), as long as
> it's not the only way to get to HTTP/2 for HTTP URIs?
>

Defining a new dns record type would be unnecessary and would be the wrong
approach. SRV records were made specifically for this kind of use. They're
certainly not perfect but inventing a new record type specifically for
http2 stuff is even less so.

> This approach hasn't been talked about in detail yet, but it apparently
> (as some have noted) has the disadvantage of not upgrading the first
> interaction, and of requiring a separate cache (and caching model) for this
> information.
>
> Who wants to pursue this approach? If so, we need a small proposal
> written, preferably as an I-D, but an e-mail would do.
>
>
This is one of those ideas that sounds interesting in theory but I don't
see how it would be effective in practice.

> And, are there any other approaches we want to put on the table?
For browsers, upgrading on the first connection is obviously going to be
critical. The DNS option seems like the best approach for the most general
case, despite the various flaws in that approach.
montenegro-httpbis-http2-negotiation seems to provide a reasonable fallback
when we're talking primarily about GET/HEAD traffic, but it has obvious
issues when dealing with POST/PUT/PATCH type operations. I do not really
want to always have to send PUT requests as HTTP/1.1 when what I really
want is to use HTTP/2.0 for the entire flow. For payload-bearing requests,
upgrade needs to be negotiated before the initial request is sent.

One possible way to mitigate this would be to combine the Upgrade header
approach with a new Expect token. For instance...

  POST /default.htm HTTP/1.1
  Host: server.example.com
  Connection: Upgrade
  Upgrade: HTTP/2.0
  Expect: 101-upgrade

The addition of the Expect header does two things...

1. It makes the upgrade a requirement rather than optional...
2. The client would then wait as reasonable time for the 101 Switching
Protocols before it proceeds to send the http/2.0 encoded data...

When the client does start sending the data (after receiving the 101
response), it would do so by first sending a SYN_STREAM.. To http/1.1
servers, this would appear to be nothing more than regular POST payload
data. For http/2 traffic, this requires an initial round trip but it
addresses the upgrade on payload-bearing request issue... and is exactly
the kind of thing the Expect header was intended for. The scheme is
obviously far from perfect, however.

Alternatively, I firmly believe that a http2 URI scheme as a reasonable
fallback needs to be considered as an option here... particularly if we do
define a dedicated http/2 default port. Yes, I agree with the whole "we
want to do existing http:// and https:// traffic over http/2 transparently"
argument and fully understand the challenges of introducing a new URI
scheme. However, there will be instances where upgrade negotiation is
simply not going to be necessary and for which backwards compatibility with
existing infrastructure is going to be unnecessary. For those cases, the
ability to pass a URL like "http2://example.org" and have it just
automatically understood that it's "http/2 over port whatever" will be
extremely useful. A no brainer even. So much so that if the working group
here doesn't choose to define it, I'll likely just end up writing it up as
an individual submission. That way, when I write.. <form method="post"
action="http2://example.org">...</form> the notion that http/2 is being
used is explicit and no upgrade discovery/negotiation is required at all.
That's what url schemes are for in the first place, right?

- James

> The NPN, Upgrade and Alternate-Protocol approaches all (should, if we go
> there) use a token to identify the protocol being upgraded to, meaning that
> we'll be able to do something like
>
>   http2-draft-3
>   http2-draft-5
>
> when negotiating the protocol while we're developing it; as long as
> implementations deploy the latest version and perhaps the one before it,
> they should be able to interoperate, and fall back to HTTP/1 if not. The
> browsers have used this approach for SPDY, and it seemed to work well.
>
> To me, this implies that if we define a DNS-based mechanism to hint
> upgrade, it should have a similar capability, so we can reuse those tokens
> and this approach there too. Make sense?
>
> Concurrently, the SPDY draft uses 15 bits to identify the protocol in use.
> Provided we get the upgrade mechanism right, I'm less concerned about this,
> but it would be good if the messages were self-describing, so we should
> figure out what the appropriate values to put in here are too. And, maybe
> saving a few bits too.
>
> Cheers,
>
>
> --

---

From: Eliot Lear <[email-masked]>
Date: Tue, 23 Oct 2012 19:17:27 +0200

Mark,

> The HTTPbis Working Group will be happy to coordinate schedules, review drafts and provide further input as required.

It's a little odd having to have one working group liaise a request to
another working group, but alright.  Traditionally we do not name
individual drafts in liaisons, and I suggest we not do that in this
case, as there are posted alternatives.

Eliot

---

From: Patrick McManus <[email-masked]>
Date: Tue, 23 Oct 2012 14:54:06 -0400

> upgrading the first interaction, unless you block on the response,

also being clear - in cases where SRV wins the race there is an advantage
to be had without blocking. A high quality implementation can bundle the
srv response with the A as an additional record.. even if that isn't what
is happening today, supporting this mechanism provides a path for servers
to opt themselves into it without blocking on it. And of course sometimes
name resolution happens considerably before connect happens so there still
might be an opportunity to collect both the a* and the srv before going to
connect without blocking on it even if the SRV considerably lags the A. So
I think this is an important optimization.

but +1 on not building in a mandatory delay inducer like blocking on srv,
or an upgrade with an empty payload.

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 24 Oct 2012 14:23:23 +1300

> many scenarios where upgrade negotiation simply is not going to be 

My understanding of the charter was that defining a new scheme was out 
of bounds. A *default* port would mean a new scheme name was mandatory 
to identify default http://*:80 from default http://*:443 from default 
http2 port.

I find it highly undesirable either way. Lets make HTTP/2 the successor 
to merge both HTTP/1.1 port 80 and HTTP/1.1 port 443 safely with scheme 
http:// and default port 80. Note that this does not require it being 
ASCII text initial request. We can easily and safely prefix with a 
TLS-follows frame to determine HTTPS over port 80.

> http2 stuff is even less so.

+1 on that.

>>
>> 2) Using a response header to hint that HTTP/2 is available on 
>> another
>> port.
>>

Maybe yes maybe no. This seems to me to be a repetition of the Via: 
header with its path version hints. Equally valid in HTTP/1.a and 2.0 
and covers this use-case already.

> case, despite the various flaws in that approach.

*except* for all those cases where middleware exists. In which case DNS 
hinting will be the cause of problems, not the solution.

> upgrade needs to be negotiated before the initial request is sent.

That said, how prevalent is first-contact PUT/POST as opposed to a page 
GET followed by further interaction from scripts and forms?
  Is it just from badly designed sites using cross-domain PUT/POST?
  or from non-browser agents which should be able to know better anyway 
with less critical timing requirements?

> obviously far from perfect, however.

+1.

Also note that the new Expect token needs to be defined as hop-by-hop 
to prevent its relay.

This is also just as useful for Upgrade:HTTP/1.1 to mandate HTTP/1.1 
feature support on the connection.

> That's what url schemes are for in the first place, right?

Consider whether you are making HTTP/2 an second protocol to exist in 
parallel to HTTP/1 or a replacement version of the protocol.

A parallel protocol would have a separate scheme (eg. HTTP vs HTTPS 
today). And may as well be called HTTP2/1.1 or "HTTP/1.1 over port 1234" 
as HTTP/2.0.

A replacement would use the same scheme and let the agents negotiate 
the version level between themselves without the web developers getting 
involved with specific URL versions.

Also, how would one go about re-writing the entire legacy web page set 
to use http2:// URLs when the server (or even just its frontend proxy!) 
upgrades?

Amos

---

From: Roberto Peon <[email-masked]>
Date: Tue, 23 Oct 2012 19:29:26 -0700

On Oct 23, 2012 6:25 PM, "Amos Jeffries" <[email-masked]> wrote:
> My understanding of the charter was that defining a new scheme was out of
bounds. A *default* port would mean a new scheme name was mandatory to
identify default http://*:80 from default http://*:443 from default http2
port.
to merge both HTTP/1.1 port 80 and HTTP/1.1 port 443 safely with scheme
http:// and default port 80. Note that this does not require it being ASCII
text initial request. We can easily and safely prefix with a TLS-follows
frame to determine HTTPS over port 80.
> Maybe yes maybe no. This seems to me to be a repetition of the Via:
header with its path version hints. Equally valid in HTTP/1.a and 2.0 and
covers this use-case already.
> Also note that the new Expect token needs to be defined as hop-by-hop to
prevent its relay.
> A parallel protocol would have a separate scheme (eg. HTTP vs HTTPS
today). And may as well be called HTTP2/1.1 or "HTTP/1.1 over port 1234" as
HTTP/2.0.
>
> A replacement would use the same scheme and let the agents negotiate the
version level between themselves without the web developers getting
involved with specific URL versions.
>
> Also, how would one go about re-writing the entire legacy web page set to
use http2:// URLs when the server (or even just its frontend proxy!)
upgrades?

As I understand the suggestion, that want required. What I understood was
that a scheme of http2 meant one should assume you can use HTTP/2. An HTTP
scheme , however didn't mean only http/1.x-- it could also be http/2 after
negotiations.

Proxies make any of this moot, however, if they are used-- a negotiation
must happen if there is ever anything other than direct connectivity. If
the scheme is just used as a hint instead of an assertion (I.e. one can
always fall back to 1.1) it is technically useful in potentially , maybe,
saving an rtt... but ugly and it could end up costing more in the case
where a proxy doesn't understand http/2 and where the session must be
restarted...

-=R

 On Oct 23, 2012 6:25 PM, "Amos Jeffries" <[email-masked]> wrote:

> My understanding of the charter was that defining a new scheme was out of
> bounds. A *default* port would mean a new scheme name was mandatory to
> identify default http://*:80 from default http://*:443 from default http2
> port.
>
> I find it highly undesirable either way. Lets make HTTP/2 the successor to
> merge both HTTP/1.1 port 80 and HTTP/1.1 port 443 safely with scheme http://and default port 80. Note that this does not require it being ASCII text
> initial request. We can easily and safely prefix with a TLS-follows frame
> to determine HTTPS over port 80.
>
>
> Maybe yes maybe no. This seems to me to be a repetition of the Via: header
> with its path version hints. Equally valid in HTTP/1.a and 2.0 and covers
> this use-case already.
>
>
>  montenegro-httpbis-http2-**negotiation seems to provide a reasonable
>> fallback
> Also note that the new Expect token needs to be defined as hop-by-hop to
> prevent its relay.
>
>> action="http2://example.org">.**..</form> the notion that http/2 is being
> today). And may as well be called HTTP2/1.1 or "HTTP/1.1 over port 1234" as
> HTTP/2.0.
>
> A replacement would use the same scheme and let the agents negotiate the
> version level between themselves without the web developers getting
> Also, how would one go about re-writing the entire legacy web page set to
> use http2:// URLs when the server (or even just its frontend proxy!)
> upgrades?
>
>
> Amos
>
>

---

From: James M Snell <[email-masked]>
Date: Tue, 23 Oct 2012 19:56:37 -0700

> scheme , however didn't mean only http/1.x-- it could also be http/2 after
Yep. http2:// would mean skip the negotiation.

> The point of the http2 scheme is that backwards compatibility with
http/1.1 infrastructure is not always going to be requirement. I'm not
arguing that it should be used for all cases... just in the subset of cases
where explicitly skipping negotiation makes sense. For example, if I have a
media server running on my local home wireless network and I have a
dedicated client app for that on my mobile device connected to the same
wireless network. There is no proxy between the two devices, just a direct
connection between a client app and a server app. There's no reason at all
why upgrade negotiation would be required in this case. A client stack on
the mobile device could be given a url.. http2://10.0.0.2/movies and just
know that http/2 is to be used.

One thing that needs to be recognized is that there will not be a single
solution that's going to work for all cases.

- James

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 14:21:01 +1100

> case, as there are posted alternatives.

OK, I think they're aware of it anyway. :)

Cheers,

---

From: David Kendal <[email-masked]>
Date: Wed, 24 Oct 2012 04:31:24 +0100

On 23 Oct 2012, at 17:50, James M Snell <[email-masked]> wrote:

> define a dedicated http/2 default port.

Bear in mind that if this happens HTTP would become the only (well-
known, at least) many-to-many (ie. any client should be able to talk to
any server talking any version of the protocol, for public access by
strangers -- with eg. IMAP that is not a problem because it's not so
unreasonable to ask people to switch mail client to be able to retrieve
mail from their *own* server) protocol to be specified where different
versions exist on different ports and have different URI schemes.

This would significantly slow deployment. It also breaks the meaning of
URIs: consider the number of existing http:// links published already.
If someone clicks one of those in a new HTTP/2.0 era, we have two
choices: we can either just use HTTP/1.x, which will force servers to
support that protocol forever, which could involve leaving another port
open, and it will also probably neutralise the speed benefits of
HTTP/2.0 for some of those links (I'll bet that the cost of Upgrade/101
negotiation turns out to be more than the savings of HTTP/2.0; I concede
that for any resources linked to that page will likely get the benefits
of 2); or we can break the meaning of the URI, try an HTTP/2.0 request
first and if that fails, degrade to HTTP/1.1. In the latter case, why
bother maintaining the http URI scheme for years if browsers will just
treat it more-or-less like http2?

My preferred solution to this would be to ask clients to send an
HTTP/2.0 request to every server, with a old-fashioned http URI, on port
80. If the server doesn't understand, it will either not respond -- just
close the connection (some servers do this, I forget which) -- or send a
response (quite possibly a 400 or 505, but it could be anything, even
success) with an HTTP/1.x version indicated in the header. If that
happens, the client should try again with an appropriate HTTP/1.1
version. It will be slightly slower to get things from HTTP/1.x servers
than before, but no negotiation will be needed for HTTP/2.0.

Obviously this isn't a very "pure" solution, in that it involves
muddying the waters of HTTP versions. But it does get the job done.

Now, to get this right, a few rules are needed:

- an HTTP/2.0 client MUST retry with an HTTP/1.x request if the response
 from the server has a non-HTTP/2.x version in the header, regardless of
 response code, including success. (This is to ensure that any new
 headers introduced in 2.0 which could change the resulting response
 somehow are properly respected by the server.) It should close the
 connection to the server as soon as it receives the response-line.
 (This is to stop large files which might be included in the response
 from being downloaded, wasting bandwidth, before the retry.)
- an HTTP/2.0 client MUST retry with an HTTP/1.1 request if the server
 closes the connection without responding.
- an HTTP/x.y client (x > 2) MUST apply the same policy.
- an HTTP/1.x server which is aware of the existence of HTTP/2.0 but
 does not yet support it SHOULD reply with a 505 status code.
- an HTTP/2.x server MUST reply with an HTTP/2.x version number in the
 response-line.
- an HTTP/2.x server SHOULD also support the HTTP/1.x line, but in case
 this is not desired or needed (for instance, in a proprietary system
 where the capabilities of all clients that will access the server are
 known) it SHOULD respond with a 505 status code. It SHOULD in all cases
 be able to give an appropriate 101 response to an HTTP/1.x client which
 sends an Upgrade header which indicates 2.0 support.

> - James

— dpk.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 14:44:48 +1100

On 24/10/2012, at 12:23 PM, Amos Jeffries <[email-masked]> wrote:

> *except* for all those cases where middleware exists. In which case DNS hinting will be the cause of problems, not the solution.

How so?

For  accelerators / "reverse" proxies, the party in control of the middleware is in control of the DNS.

For configured proxies, the client is aware of the middleware and can figure out and remember its capabilities, via the upgrade mechanism.

It's only for unconfigured / "transparent" / intercepting proxies that this might be a problem. If the protocol is designed to fail fast in their presence, this too should be manageable -- there might be some latency on first request in these networks, at worst. Bonus points for putting the blame squarely where it lies somehow...

Cheers,

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 14:44:46 +1100

On 24/10/2012, at 3:50 AM, James M Snell <[email-masked]> wrote:

[...]
> Defining a new dns record type would be unnecessary and would be the wrong approach. SRV records were made specifically for this kind of use. They're certainly not perfect but inventing a new record type specifically for http2 stuff is even less so.

We're not talking about defining a new record type, only deciding whether DNS is an appropriate OPTIONAL upgrade path, and EVENTUALLY which existing record type to use.

Everyone, please focus on the questions at hand -- we don't need to have a round of +1s and resulting back-and-forth on an issue that's not even being discussed yet.

> This is one of those ideas that sounds interesting in theory but I don't see how it would be effective in practice.

Yep - that's why I asked for proposals. 

> For browsers, upgrading on the first connection is obviously going to be critical. The DNS option seems like the best approach for the most general case, despite the various flaws in that approach. montenegro-httpbis-http2-negotiation seems to provide a reasonable fallback when we're talking primarily about GET/HEAD traffic, but it has obvious issues when dealing with POST/PUT/PATCH type operations. I do not really want to always have to send PUT requests as HTTP/1.1 when what I really want is to use HTTP/2.0 for the entire flow. For payload-bearing requests, upgrade needs to be negotiated before the initial request is sent.
> 
> One possible way to mitigate this would be to combine the Upgrade header approach with a new Expect token. For instance...
> 
> 2. The client would then wait as reasonable time for the 101 Switching Protocols before it proceeds to send the http/2.0 encoded data...
> 
> When the client does start sending the data (after receiving the 101 response), it would do so by first sending a SYN_STREAM.. To http/1.1 servers, this would appear to be nothing more than regular POST payload data. For http/2 traffic, this requires an initial round trip but it addresses the upgrade on payload-bearing request issue... and is exactly the kind of thing the Expect header was intended for. The scheme is obviously far from perfect, however.

Hm. I'd be interested to hear from implementers on this. The only advantage that I see is that you'd be able to start multiplexing other requests on the same connection sooner. However, you could also just use SRV, or a different connection if it's not available. 

> Alternatively, I firmly believe that a http2 URI scheme as a reasonable fallback needs to be considered as an option here... particularly if we do define a dedicated http/2 default port. Yes, I agree with the whole "we want to do existing http:// and https:// traffic over http/2 transparently" argument and fully understand the challenges of introducing a new URI scheme. However, there will be instances where upgrade negotiation is simply not going to be necessary and for which backwards compatibility with existing infrastructure is going to be unnecessary. For those cases, the ability to pass a URL like "http2://example.org" and have it just automatically understood that it's "http/2 over port whatever" will be extremely useful. A no brainer even. So much so that if the working group here doesn't choose to define it, I'll likely just end up writing it up as an individual submission. That way, when I write.. <form method="post" action="http2://example.org">...</form> the notion that http/2 is being used is explicit and no upgrade discovery/negotiation is required at all. That's what url schemes are for in the first place, right?

I don't see any consensus to do this here; does anyone support James' position, or want to discuss it more?

(i.e., we've heard enough from folks challenging this approach, for now)

Regards,

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 23 Oct 2012 20:57:47 -0700

> Yep - that's why I asked for proposals.

It works in practice. Alternate-Protocol is deployed and live. I'm not
terribly excited about it, since I'd rather people use https:// with
whatever negotiation scheme the TLS WG comes up with.

> Hm. I'd be interested to hear from implementers on this. The only advantage that I see is that you'd be able to start multiplexing other requests on the same connection sooner. However, you could a…

I agree with Mark.

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 14:59:27 +1100

> whatever negotiation scheme the TLS WG comes up with.

Would you or someone else familiar with it mind writing up a small spec for it? 

Ideally an I-D, but an e-mail will suffice.

Thanks,

---

From: James M Snell <[email-masked]>
Date: Tue, 23 Oct 2012 21:04:09 -0700

On Tue, Oct 23, 2012 at 8:44 PM, Mark Nottingham <[email-masked]> wrote:

> > Defining a new dns record type would be unnecessary and would be the
> wrong approach. SRV records were made specifically for this kind of use.
> They're certainly not perfect but inventing a new record type specifically
> for http2 stuff is even less so.
>
> We're not talking about defining a new record type, only deciding whether
> DNS is an appropriate OPTIONAL upgrade path, and EVENTUALLY which existing
> record type to use.
>
> Everyone, please focus on the questions at hand -- we don't need to have a
> round of +1s and resulting back-and-forth on an issue that's not even being
> discussed yet.
>
>
Ok.. consider this a proposal then ;-)

> Hm. I'd be interested to hear from implementers on this. The only
> advantage that I see is that you'd be able to start multiplexing other
> requests on the same connection sooner. However, you could also just use
> SRV, or a different connection if it's not available.
>
>
Well.. the other advantage is that you upgrade before sending payload data,
which is a step up from montenegro-httpbis-http2-negotiation... It's
certainly not without it's problems, however. For my money, however, the
SRV route is preferable but certainly has it's own range of issues.

(and, so we're clear, who exactly do you classify as "implementors" here?)

> want to do existing http:// and https:// traffic over http/2
> transparently" argument and fully understand the challenges of introducing
> a new URI scheme. However, there will be instances where upgrade
> negotiation is simply not going to be necessary and for which backwards
> compatibility with existing infrastructure is going to be unnecessary. For
> those cases, the ability to pass a URL like "http2://example.org" and
> have it just automatically understood that it's "http/2 over port whatever"
> will be extremely useful. A no brainer even. So much so that if the working
> group here doesn't choose to define it, I'll likely just end up writing it
> up as an individual submission. That way, when I write.. <form
> method="post" action="http2://example.org">...</form> the notion that
> http/2 is being used is explicit and no upgrade discovery/negotiation is
> required at all. That's what url schemes are for in the first place, right?
>
You're the one who asked for proposals ;-) ... I believe you're also the
one who said, "we don't need to have a round of +1s and resulting
back-and-forth on an issue that's not even being discussed yet." ... I
brought this up as one of several possible proposals for discussion at some
point. I'm not going to argue this particular point of view further right
now but it does need to be recognized that
http2-without-upgrade-negotiation is definitely something that needs to be
considered in-scope.

- James

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 23 Oct 2012 21:12:56 -0700

> Of course, if Chrome (or any other browser) would be interested in running an experiment, we'd love to do that too -- provided we can have input into the design.

Is there anything you'd like to change about the design in
http://www.ietf.org/mail-archive/web/tls/current/msg05593.html? I'm in
general supportive of running experiments, I just don't want to waste
our time unless there was something deficient about the previous
experiment. Also, I suspect a non-Google experiment would carry more
weight :) External validation is good.

On that point, at Realtime Conf today, Arnout Kazemier provided a
bunch of data on issues with WebSockets deployment.
https://speakerdeck.com/3rdeden/realtimeconf-dot-oct-dot-2012. As he
says "tl;dl: Always use SSL".

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 15:19:35 +1100

> Ok.. consider this a proposal then ;-)

For what?  I asked a very clear question, and you jumped to the next phase of the discussion prematurely. 

<soapbox>
We need to start to control the discussion here; it was comparatively free-wheeling before we re-chartered, but if it's allowed to meander too much now, we'll get off track. Every message sent to this list is delivered to thousands of mailboxes, and needs to be read by (at least) hundreds of people. The more noise we have on the list, the more likely it is to scare away folks who have valuable input but not much time.

Please understand I'm not picking on you in particular; this is for everyone.
</soapbox>

> Well.. the other advantage is that you upgrade before sending payload data, which is a step up from montenegro-httpbis-http2-negotiation... It's certainly not without it's problems, however. For my money, however, the SRV route is preferable but certainly has it's own range of issues. 

Ack.

> (and, so we're clear, who exactly do you classify as "implementors" here?)

See the charter. 

In this case, it's the people who have to implement the upgrade mechanism, both on the client side and the server side. This includes everyone, but I'll be paying particular attention to the implementations that can "move the needle" and get market adoption. 

In other words, we want this to work for someone when they write a Perl library to do it, but we NEED it to work for that handful of implementations that define the lion's share of traffic on the Web -- again, both client and server.

> You're the one who asked for proposals ;-)

Yes, and you've made yours. If you'd like to add to it, that's fine, but I'm not sure what point you're trying make here.

> ... I believe you're also the one who said, "we don't need to have a round of +1s and resulting back-and-forth on an issue that's not even being discussed yet." ... I brought this up as one of several possible proposals for discussion at some point. I'm not going to argue this particular point of view further right now but it does need to be recognized that http2-without-upgrade-negotiation is definitely something that needs to be considered in-scope.

It's been brought up before, and I'd characterise the reaction of the WG so far as very much against it. This is why I asked for anyone who wanted to support it, to firm up that characterisation. 

Regards,

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 23 Oct 2012 21:25:06 -0700

On Tue, Oct 23, 2012 at 11:54 AM, Patrick McManus <[email-masked]> wrote:
>
>
> On Mon, Oct 22, 2012 at 10:33 PM, William Chan (陈智昌) <[email-masked]>
> wrote:
>>
>>
> also being clear - in cases where SRV wins the race there is an advantage to
> be had without blocking. A high quality implementation can bundle the srv
> response with the A as an additional record.. even if that isn't what is
> happening today, supporting this mechanism provides a path for servers to
> opt themselves into it without blocking on it. And of course sometimes name
> resolution happens considerably before connect happens so there still might
> be an opportunity to collect both the a* and the srv before going to connect
> without blocking on it even if the SRV considerably lags the A. So I think
> this is an important optimization.

Ah, I totally forgot that DNS prefetches might significantly mitigate
the penalty of an srv lookup. Good point, thanks.

>
> but +1 on not building in a mandatory delay inducer like blocking on srv, or
> an upgrade with an empty payload.
>

---

From: William Chan (陈智昌) <[email-masked]>
Date: Tue, 23 Oct 2012 21:27:14 -0700

Will http://dev.chromium.org/spdy/spdy-protocol/spdy-protocol-draft2#TOC-Server-Advertisement-of-SPDY-through-the-HTTP-Alternate-Protocol-header
do for now? I'd like to reiterate though that I'd rather people just
use the TLS WG mechanism, whatever it is :)

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 15:33:10 +1100

Thanks, that's fine for now.

Cheers,

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 24 Oct 2012 17:47:47 +1300

> says "tl;dl: Always use SSL".

The main deficiency I can see there is that it is aging data already and 
measures what the situation is in a market where middleware authors were 
given no incentive to support the feature.

The use case for Upgrade:HTTP/2.0 in particular is now providing that 
incentive for middleware implementations to support Upgrade better and 
indeed seems to have stronger support from middleware than end-software 
authors. Working with us rather than assuming we we never implement a 
needed feature will go a long way towards seeing that feature rolled 
out. The results of that single test could change dramatically in the 
coming years and are very likely to flow along with HTTP/2 rollout.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 15:51:31 +1100

On 24/10/2012, at 3:12 PM, William Chan (陈智昌) <[email-masked]> wrote:
> experiment.

Could you dig up the exact bytes-on-the-wire handshake that was used?

> Also, I suspect a non-Google experiment would carry more

Agreed.

> says "tl;dl: Always use SSL".

Thanks.

I'd like to validate an assumption that has been discussed in-person in a meeting (forget which one) but IIRC not yet on-list.

It's that if we can design an upgrade that fails fast and reliably for HTTP URLs, it's acceptable for there to be a (say) 70% success rate initially, with the idea that it will rise over time -- perhaps slowly, in terms of browser releases, but relatively quickly, in terms of the lifetime of HTTP.

Can we (everyone) agree upon that?

WRT WebSockets, keep in mind that the proxy/firewall/virus scanning vendors have had less than a year since it became an RFC, for a protocol that has (relatively) unproven demand and hard-to-nail-down semantics.

I suspect that HTTP2, once we finish, will be get more priority from them; the demand is already proven, and the semantics of the protocol are easier to fit into their existing threat models, etc.

Cheers,

---

From: James M Snell <[email-masked]>
Date: Tue, 23 Oct 2012 21:51:30 -0700

> with less critical timing requirements?
Likely significantly more prevalent in the "web applications" space than
within browser-based environments. For instance, POST a new entry to an
Atompub Collection does not first require a GET on the collection URI...

Other examples include...

  - Posting a new tweet (
https://dev.twitter.com/docs/api/1.1/post/statuses/update)
  - Publishing to facebook (
https://developers.facebook.com/docs/reference/api/#publishing)
  - Pushing content notifications (
http://pubsubhubbub.googlecode.com/svn/trunk/pubsubhubbub-core-0.3.html#publishing
)
  - https://developers.google.com/+/history/#writing_moments
  - http://developer.github.com/v3/issues/#create-an-issue
  - ...

While such apis generally can consist of a mixture of GET/POST/PUT/DELETE
operations, these can generally be spread out over separate sessions over
time and first-contact payload-bearing requests are not unusual.

Also consider (while all the cool kids are going to cringe and snicker in
the corner when I mention it...) there is still a significant amount of
WS-* traffic out there to deal with that tunnels all traffic over POST.

- James

---

From: Takeshi Yoshino <[email-masked]>
Date: Wed, 24 Oct 2012 14:12:44 +0900

On Tue, Oct 23, 2012 at 11:50 AM, Mark Nottingham <[email-masked]> wrote:

> >> Who's willing to do some experimentation? Specifically, does anyone
> have access to the code that was used before (IIRC, people bought some ads
> and inserted some Java to probe the network)?
> >
Maybe this work http://w2spconf.com/2011/papers/websocket.pdf which
resulted in introduction of masking to the WebSocket protocol.

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 24 Oct 2012 18:21:47 +1300

> It's only for unconfigured / "transparent" / intercepting proxies that this might be a problem. If the protocol is designed to fail fast in their presence, this too should be manageable -- there mi…

Yes they are the major hassle and they are not going away until that 
blocker problem of browsers not supporting TLS to proxies is resolved. 
Rolling out DNS record dependency, even if only as a first attempted 
connection in their presence is a guaranteed fail for that DNS mechanism.

For the other types of middleware the DNS is irrelevant as you say. So 
where is the benefit come in from doing it when middleware is involved? 
there is none. Yet the interceptors will cause problems. A very 
unbalanced situation leaning towards breakage.

I'm +0 on the DNS mechanism. Looks good, but only for a small segment of 
the market.

Amos

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Oct 2012 08:48:50 +0200

> I'm +0 on the DNS mechanism. Looks good, but only for a small segment of 

I'd go further, the problem with DNS is that by design it's used to
provide information for one record which corresponds to a host name.
In my opinion it's totally unsuited for use with HTTP the way it's
used today because I see no way it would cover the following cases :

  - multiple IP addresses advertised at different locations for the
    same record, possibly talking different protocol versions due to
    a progressive deployment. Eg :

	$ host -t a google.com
	google.com has address 74.125.230.197
	google.com has address 74.125.230.196
	google.com has address 74.125.230.201
	google.com has address 74.125.230.200
	google.com has address 74.125.230.192
	google.com has address 74.125.230.199
	google.com has address 74.125.230.195
	google.com has address 74.125.230.193
	google.com has address 74.125.230.198
	google.com has address 74.125.230.206
	google.com has address 74.125.230.194

  - anycast routing : the same name and IP address exist on two continents,
    but they don't necessarily speak the same version. Some CDNs right now
    use anycast, it would be better if we don't tell them "switch all your
    nodes at once".

  - react to quick changes : you're installing your shiny new gateway
    supposed to talk http/2. You advertise it in DNS. Then you realize
    it crashes. Oops, rollover to previous 1.1 gateway, but the bad was
    already made, the world knows you're on the new one and caches this
    for some time. Site remains randomly accessible for the time users
    cache the record ;

  - multiple service versions at the same IP address as is often encountered
    on internal networks. For example, you have dynamic on :80 and :443 in
    HTTP/1 and static on :8080 and :8443 using HTTP/2. And note that in
    developer environments, it's very common to have 100+ ports bound on
    a same host with different applications being developped. I wouldn't be
    one of these users if my browser randomly used either version to access
    the application I'm developing depending on DNS records...

Seriously, I think that using negociation for the standard scheme is the
only way to make something compatible with what exists. Having a new scheme
is fine to say 2.0 only but as someone (I don't remember whom) said a
few months ago, proxies will have no choice but to obey and they won't all
be 2.0 ready for some (long) time.

Willy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 17:52:52 +1100

> be 2.0 ready for some (long) time.

Willy, 

A fair number of folks has said they want this, and no one is saying that it's mandatory.

AFAICT most of your problems can be boiled down to:

1) Different servers may back up one DNS record - in which case, they don't have to use SRV

2) Clients assuming HTTP/2 may fail - in which case, we need to design things so they can recognise this and fall back to 1.1 seamlessly.

Make sense?

Regards,

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Oct 2012 09:10:46 +0200

> A fair number of folks has said they want this, and no one is saying that it's mandatory.

Not being mandatory still means that browsers will have to decide what to
do based on this record at one point, with lists of exceptions.

> 2) Clients assuming HTTP/2 may fail - in which case, we need to design things so they can recognise this and fall back to 1.1 seamlessly.

While this seems to be a prerequisite to me, it does not sound like the
solution to everything : failing fast is what we all hope for, but we
know well that there are environments where the failure will cause delays
etc... (you're well aware of the issues faced by pipelining for example).
So similarly we'll have some boxes drop packets or simply not forward data
when they discover junk after the request headers.

DNS records are just for global view of how the server is presented to the
net, but it does not take into account the whole path to the clients.

I'd suggest that the zero-roundtrip upgrade is made only on TLS using the
NPN extension that already works well with SPDY. The advantage is that it
says what protocol can be spoken on top of a connection and does not rely
on external components unaware of the path.

And for clear text we'd keep the good old HTTP upgrade which should be
optimal for the most common usages (GET/HEAD first) and require that
other methods would be handled as 1.1 during the first round trip. I
don't see this as a terrible tradeoff.

Willy

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 18:25:57 +1100

> do based on this record at one point, with lists of exceptions.

Yes -- but I'm hearing (so far) from the browser folks that they're willing to do this.

> on external components unaware of the path.

So, I'd put it this way:

The DNS-based upgrade is optimistic; it will be fast as long as there isn't an unknown middlebox present; if there is, it'll be slower to start. The question is how often that will happen, and how detectable / variable it is.

On the other hand, the Upgrade-based negotiation is pessimistic; it assume that something will go wrong until we prove that both ends speak the same language. It'll be fast but have some limitations at the start (as discussed).

Ideally, we'd have exactly one way to upgrade for HTTP URIs. However, that *may* not be possible -- if enough people's needs aren't met by the Upgrade path, other mechanisms for doing it will be developed, and if that's going to happen, I'd like it to be interoperable, and ideally part of the spec itself.

I do hear what you're saying, though. We'll need more data to make hard decisions; at this point I'm just trying to figure out what high-level approaches we're going to pursue in earnest. 

> don't see this as a terrible tradeoff.

At this (early) point it looks like we're going that way, yes. The question is whether we have an optimistic path as well.

Cheers,

---

From: Eliot Lear <[email-masked]>
Date: Wed, 24 Oct 2012 09:46:24 +0200

On 10/23/12 8:54 PM, Patrick McManus wrote:
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>
> also being clear - in cases where SRV wins the race there is an
> advantage to be had without blocking. A high quality implementation
> can bundle the srv response with the A as an additional record.. even
> if that isn't what is happening today, supporting this mechanism
> provides a path for servers to opt themselves into it without blocking
> on it. And of course sometimes name resolution happens considerably
> before connect happens so there still might be an opportunity to
> collect both the a* and the srv before going to connect without
> blocking on it even if the SRV considerably lags the A. So I think

I continue to be concerned about the use of SRV for any number of
reasons, but one big one is that what you describe above could induce
cache pollution due to different sources of authority.  I would prefer
to see a different record used that uses the same name, and therefore,
the same authority.

Eliot

---

From: James M Snell <[email-masked]>
Date: Wed, 24 Oct 2012 00:47:38 -0700

On Wed, Oct 24, 2012 at 12:25 AM, Mark Nottingham <[email-masked]> wrote:

> [snip[
>
> So, I'd put it this way:
>
> The DNS-based upgrade is optimistic; it will be fast as long as there
> isn't an unknown middlebox present; if there is, it'll be slower to start.
> The question is how often that will happen, and how detectable / variable
> it is.
>
> On the other hand, the Upgrade-based negotiation is pessimistic; it assume
> that something will go wrong until we prove that both ends speak the same
> language. It'll be fast but have some limitations at the start (as
> discussed).
>
> Ideally, we'd have exactly one way to upgrade for HTTP URIs. However, that
> *may* not be possible -- if enough people's needs aren't met by the Upgrade
> path, other mechanisms for doing it will be developed, and if that's going
> to happen, I'd like it to be interoperable, and ideally part of the spec
> itself.
>
>
An optimistic approach where DNS is not in the picture is going to be a
necessity at some point.

- James

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 24 Oct 2012 09:56:48 +0200

> Yes -- but I'm hearing (so far) from the browser folks that they're willing to do this.

I know, I'm just raising some points so that we don't forget anything.

(...)
> At this (early) point it looks like we're going that way, yes. The question is whether we have an optimistic path as well.

I really think that we could consider that https:// is the only fast path.
After all, the web is progressively moving towards more https, and even the
SPDY team wanted full TLS at the beginning. Why not have it this way ? It
would be another incentive for moving web sites to TLS, and would also make
https look faster than http in some carefully biased benchmarks :-)

That said for https to work we still need to fix browser-to-proxy
communications.

Regards,
Willy

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 24 Oct 2012 21:40:04 +1300

> Can we (everyone) agree upon that?

I thought we are working from a baseline of Upgrade: being *the* 
existing defined HTTP/1.1 method for negotiation. Those Chromium 
experimentd proving it had ~67% success rate. Now just looking for ways 
to use it more efficiently and alternative methods that would improve 
that success rate faster?

Amos

---

From: Eliot Lear <[email-masked]>
Date: Wed, 24 Oct 2012 10:41:27 +0200

On 10/24/12 9:48 AM, Mark Nottingham wrote:
> Can you expand upon that a bit? You mean where the successive DNS
> lookups come from different servers, or...?

The issue is additional information that Patrik is suggesting that we
use (to be fair I may have made the same suggestion without thinking it
through earlier).  So, take for instance the case where you want to look
up what the server on example.com is using.  You might have the
following SRV response:

_http2._tcp.example.com    IN    SRV 0 10 880 http2server.example.com

and additional information of

http2server.example.com    IN    A 192.0.2.1

The problem is that _http2._tcp.example.com may not be in the same zone
as http2server.example.com, and the querying resolver can't tell, simply
based on one query.  The nameserver for _http2._tcp.example.com doesn't
really have the right to make claims about anything outside its zone. 
There are common enterprise deployments in which this is in fact the
case.  Someone even asked me if it was possible NOT to have a zone cut
at _tcp....!!!

Now let's take a more nefarious example:

_http2._tcp.badguy.com    IN    SRV 0 880    mybank.com

and additional information of

mybank.com    IN    A    192.0.2.18 ;; where this address leads you to
the wrong site.

A solution to this issue is to use the same name.  That guarantees the
same authority.

Eliot

---

From: Amos Jeffries <[email-masked]>
Date: Wed, 24 Oct 2012 22:09:40 +1300

> Yes -- but I'm hearing (so far) from the browser folks that they're willing to do this.

They are very willing to implement NTLM as well.  Which you will notice 
... also works very well on the same single-hop connections as DNS SRV 
record.

The problems occur when a multi-hop path in introduced. Not so bad in 
the DNS feature but enough to make it a no-gain feature not worth 
looking up the DNS for configured proxies.

Are we also going to define this official DNS record for locating the 
local LAN proxy?
  It would only need to define what port + HTTP(S) version to connect 
with. Same details as an origin server domain. *That* would be far more 
useful than just an origin server record IMO. Since it goes a long way 
to eradicating those pesky WPAD issues as well as being re-usable for 
origin hosts.

> The DNS-based upgrade is optimistic; it will be fast as long as there isn't an unknown middlebox present; if there is, it'll be slower to start. The question is how often that will happen, and how…

I think it would be safe to assume most of those 33% failed Upgrade: 
attempts over port 80 the Chromium experiments were on proxied 
connections. A repeat experiment could be defined to clarify what 
portion were configured proxy, interception proxy, other failure.

Amos

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 24 Oct 2012 21:12:12 +1100

On 24/10/2012, at 7:40 PM, Amos Jeffries <[email-masked]> wrote:

> I thought we are working from a baseline of Upgrade: being *the* existing defined HTTP/1.1 method for negotiation. Those Chromium experimentd proving it had ~67% success rate. Now just looking for ways to use it more efficiently and alternative methods that would improve that success rate faster?

Yes, that appears to be the case right now.

---

From: Patrick McManus <[email-masked]>
Date: Wed, 24 Oct 2012 09:57:40 -0400

Hi,

I'm not an expert here, but yes, there are lots of ways to badly design
your DNS and incur delays - but there are also reasonable ones. This is not
substantially different than the 'out of baliwick' DNS issues from the 90's
right? but that also shows it is resolvable within the current record type
framework, as you say in this case by using the same name which is what I
think we're all envisioning.

something like

www.example.com 86400 IN A 10.10.10.100
_http2tls._tcp.www.example.com. 86400 IN SRV 0 5 443 www.example.com.

there is nothing really unworkable about using a different target name, it
just might incur delays (depending on cache state) that remove the value of
the scheme. So don't do that :)

-Patrick

[eliot, sorry for the double send to you - meant to send it to the list the
first time.]

On Wed, Oct 24, 2012 at 4:41 AM, Eliot Lear <[email-masked]> wrote:

Eliot

---

From: Eliot Lear <[email-masked]>
Date: Wed, 24 Oct 2012 20:10:31 +0200

Hi Patrick,

On 10/24/12 3:51 PM, Patrick McManus wrote:
> I'm not an expert here, but yes, there are lots of ways to badly
> design your DNS and incur delays - but there are also reasonable ones.
> This is not substantially different than the 'out of baliwick' DNS
> issues from the 90's right? but that also shows it is resolvable
> within the current record type framework, as you say in this case by
> using the same name which is what I think we're all envisioning.

You're right in that none of this is new.  The nefarious example I gave
wasn't under control of the administrator of www.example.com, but rather
the bad guy.  That's why there's a fair amount of code in many
implementations to avoid accepting additional information in any
authoritative way, but that may well cause additional lookups, defeating
the purpose of the shortcut.  It might be possible to race lookups in
some heuristic or predictive manner, but it may be just as well to
address this with a new record.

My suggestion is to ask the DNS directorate for a view.  They could say
that I'm in the weeds, but then I doubt it.  I think they would say that
we should carefully consider our requirements and available engineering
choices (buy/build).  Yes, SRV is there.  No, it's not that hard to do
another record, but it does take time to get out there, should we go
that route.

Eliot

---

From: William Chan (陈智昌) <[email-masked]>
Date: Wed, 24 Oct 2012 12:10:30 -0700

On Tue, Oct 23, 2012 at 9:51 PM, Mark Nottingham <[email-masked]> wrote:

> >> Of course, if Chrome (or any other browser) would be interested in
> running an experiment, we'd love to do that too -- provided we can have
> input into the design.
> >

Er, I'm not an expert here, but here's the experiment
http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/net/websocket_experiment/websocket_experiment_runner.cc?view=markup&pathrev=63020and
the handshake code (
http://src.chromium.org/viewvc/chrome/trunk/src/net/websockets/websocket_handshake.cc?view=markup&pathrev=63020
).

> I'd like to validate an assumption that has been discussed in-person in a
> meeting (forget which one) but IIRC not yet on-list.
>
> It's that if we can design an upgrade that fails fast and reliably for
> HTTP URLs, it's acceptable for there to be a (say) 70% success rate
> initially, with the idea that it will rise over time -- perhaps slowly, in
> terms of browser releases, but relatively quickly, in terms of the lifetime
> of HTTP.
>

Not trying to derail the thread, but <mumble>TLS</mumble> :)

>
> WRT WebSockets, keep in mind that the proxy/firewall/virus scanning
> vendors have had less than a year since it became an RFC, for a protocol
> that has (relatively) unproven demand and hard-to-nail-down semantics.
>
> I suspect that HTTP2, once we finish, will be get more priority from them;
> the demand is already proven, and the semantics of the protocol are easier
> to fit into their existing threat models, etc.

This is probably true, but I just wanted to point out that vendor support
!= actual deployments.

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 25 Oct 2012 14:33:44 +1100

On 25/10/2012, at 6:10 AM, William Chan (陈智昌) <[email-masked]> wrote:
> Er, I'm not an expert here, but here's the experiment http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/net/websocket_experiment/websocket_experiment_runner.cc?view=markup&pathrev=63020 and the handshake code (http://src.chromium.org/viewvc/chrome/trunk/src/net/websockets/websocket_handshake.cc?view=markup&pathrev=63020).

That's helpful, thanks.

---

From: Salvatore Loreto <[email-masked]>
Date: Thu, 25 Oct 2012 09:21:45 +0300

On 10/24/12 4:23 AM, Amos Jeffries wrote:
> I find it highly undesirable either way. Lets make HTTP/2 the 
> successor to merge both HTTP/1.1 port 80 and HTTP/1.1 port 443 safely 
> with scheme http:// and default port 80. Note that this does not 
> require it being ASCII text initial request. We can easily and safely 
> prefix with a TLS-follows frame to determine HTTPS over port 80.

I am not really convinced that defining a *new* default port for 
HTTP/2-only communication is a good idea.

>> wrong

+1 on SRV record

> *except* for all those cases where middleware exists. In which case 
> DNS hinting will be the cause of problems, not the solution.
>
> That said, how prevalent is first-contact PUT/POST as opposed to a 
> page GET followed by further interaction from scripts and forms?

+1 on the Expected token, it is really useful

/Salvatore

>> transparently"
> today). And may as well be called HTTP2/1.1 or "HTTP/1.1 over port 
> 1234" as HTTP/2.0.
>
> the version level between themselves without the web developers 
> getting involved with specific URL versions.
>
> to use http2:// URLs when the server (or even just its frontend 
> proxy!) upgrades?
>
>
> Amos

---

From: Gabriel Montenegro <[email-masked]>
Date: Thu, 25 Oct 2012 07:02:34 +0000

> language. It'll be fast but have some limitations at the start (as discussed).

I've been thinking about this, and think we may be able to make it more optimistic after the first exchange.

This is the scenario:
1. client has no prior knowledge whether this server supports http2 on this port in the clear.
2. client performs Upgrade (presumably, he'd be using NPN if over SSL)
3. server sends 101 back.
4. they now speak http2, and the client learns that this server speaks http2 at this port.
5. time passes, and this http2 connection goes away.
6. a subsequent message from client to server (without a pre-existing http2 connection) could bypass the Upgrade and send http2 frames directly.

This would require the server to be able to detect if any given message is either http 1.X or http 2. Given the binary framing of the latter, it seems possible to figure out how to make this unambiguous at the server.

Comments?

Gabriel

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 25 Oct 2012 09:03:59 +0200

Hi Takeshi,

> resulted in introduction of masking to the WebSocket protocol.

Yes I think this is the work Mark was talking about, but warning,
there were some issues with this test due to the inability to analyse
intermediaries behaviour. While very useful, we must keep in mind that
several of the issues detected were assigned a possible cause that
justified masking (eg: connection resets supposed to be caused by
interpretation of payload contents as requests), while a number of
other causes were much more likely true for a number of us implemntors
(eg: classic close upon response receipt). I don't have all the details
in mind anymore, this was discussed to such great lengths on the hybi
list :-)

Today I'd much rather trust real-world tests from websocket clients
running in browsers !

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 25 Oct 2012 09:29:42 +0200

Hi Gabriel,

On Thu, Oct 25, 2012 at 07:02:34AM +0000, Gabriel Montenegro wrote:
> This would require the server to be able to detect if any given message is
> either http 1.X or http 2. Given the binary framing of the latter, it seems
> possible to figure out how to make this unambiguous at the server.
> 
> Comments?

This cannot work in environment where transparent protocol analysers and
intercepting proxies are installed. The first success only means one thing,
which is that the scenario used along the whole path works, it does not
preclude that changing the opening handshake would too woke. Two examples :

- in March I informed the hybi WG that I got a report from a user that
  Trend Micro's OfficeScan installed on his machine was blocking WS
  upgrades because it was analysing exchanges between the browser and
  the outer world and did not like 101 responses. When this program
  is eventually fixed, it will still inspect contents and expect HTTP
  to be spoken there, which would not be the case on the second handshake ;

- I was aware of an ISP combining haproxy + squid to save on bandwidth costs.
  The principle was very simple, haproxy was set up as an intercepting proxy
  at the border, and balanced the load across a squid farm. Haproxy was
  configured to bypass squid for websocket since there was no added value
  in solliciting the caches for this. Haproxy doesn't understand websocket,
  it only understands HTTP/1.1 and its associated Upgrade mechanism. If you
  start sending non-HTTP/1.1 frames at it while it expects HTTP, you can be
  sure to get a 400/badreq quickly in return (at least once the protocols
  diverge).

So what we learn from the first successful connection is that we should
continue to use the same mechanism.

However I think we should experiment with the same principle as pipelining, ie
when a handshake has worked once, we can probably start sending HTTP/2 frames
after the initial client handshake without waiting for a response since we know
that the whole path supports the Upgrade well.

Regards,
Willy

---

From: Salvatore Loreto <[email-masked]>
Date: Thu, 25 Oct 2012 13:02:34 +0300

On 10/25/12 10:29 AM, Willy Tarreau wrote:
> intercepting proxies are installed.
just to clarify
the bad is not with the intercepting proxies in general,
but only with the one that would expect a connection always starting 
with an Upgrade
However if we will standardize the possibility that an HTTP connection 
can start directly
from HTTP/2.0 then the intercepting proxy should not stop it
and then the Gabriel proposal may or may not work for other reasons...

> The first success only means one thing,
that is WebSocket specific that only specifies and mandate the HTTP/1.1 
Upgrade handshake
so the Haproxy is acting correctly (and I had no doubt about it).

> that the whole path supports the Upgrade well.
that would work in a desktop,
but if you are using a mobile scenario you can not be so sure that whole 
path is going
to be use is exactly the old path that supported the Upgrade well.
or I am missing something here?

br
/Salvatore

-- 
Salvatore Loreto, PhD
www.sloreto.com

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 25 Oct 2012 12:44:45 +0200

Hi Sal,

On Thu, Oct 25, 2012 at 01:02:34PM +0300, Salvatore Loreto wrote:
> >>6. a subsequent message from client to server (without a pre-existing 
> >>http2 connection) could bypass the Upgrade and send http2 frames directly.
> >>

HTTP/1.1 intercepting proxies will block if there is no HTTP/1.1 header
prepended to the request. That's the point I'm making. And as I said in
an earlier mail, I don't really see the point in trying to optimize for
a handshake-less 2.0 connection over plain text since we already have
this possibility with TLS NPN. So I'd suggest that for clear text we
always have the Upgrade handshake prefix to remain compatible with the
HTTP/1.1 infrastructure in place, we could possibly optimize the handhsake
so that the client does not wait for the server's response once it knows
the path supports it, and for usages where we want to get rid of the
Upgrade handshake, then we use TLS where the equivalent of the Upgrade
handshake may be performed by the NPN extension.

> so the Haproxy is acting correctly (and I had no doubt about it).

It's not specific, websocket uses the standard HTTP/1.1 Upgrade handshake,
precisely to be compatible with deployed components. And it works when
components comply with the standard. And for the same reason we cannot
decide to send raw WS frames over port 80 to a server which is known to
support WS, we cannot send raw HTTP/2 frames to a server which is known
to support HTTP/2, precisely because all we know is that intermediaries
are 1.1 compliant but not necessarily 2.0 compliant.

> or I am missing something here?

That's the point I was making about the risk of changing the connection
method between the first and subsequent connections :-) If you reuse the
same mechanism for all connections, you have much less risk of failure
than if you change. I think we're saying the same thing in fact :-)

Cheers,
Willy

---

From: Takeshi Yoshino <[email-masked]>
Date: Thu, 25 Oct 2012 19:49:22 +0900

On Thu, Oct 25, 2012 at 12:33 PM, Mark Nottingham <[email-masked]> wrote:

> http://src.chromium.org/viewvc/chrome/trunk/src/chrome/browser/net/websocket_experiment/websocket_experiment_runner.cc?view=markup&pathrev=63020and the handshake code (
This experiment was done by using very old WebSocket protocol. Hixie 75 or
prior. Take a look at this section to see what's sent/received
http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-75#section-1.2in
handshake stage. The files Will pointed contains code for newer
protocol, but older one was used to get the stats that agl@ posted to
tls@ietf.

In the experiment, Chrome did

- Send basic HTTP request
- Receive basic HTTP response

to check that the port is not blocked for normal HTTP (this is baseline).
If it's successful, it then did

- Send WebSocket handshake (HTTP Upgrade)
- Receive WebSocket handshake response
- Send "0x00 Hello 0xff" over the Web Socket
- Receive "0x00 Hello 0xff" over the Web Socket
- Wait for 1 sec (making the connection idle) to see the idle connection is
not closed immediately by anyone
- Wait for a server push message and verify (expected message is "0x00 This
message must be echoed back to the
server.\nこのメッセージがサーバにエコーバックされなければなりません。\n 0xff") (the same sentence in
English and Japanese (to check UTF-8 encoded string is transferred
correctly))
- Echo back the message to the server
- Wait for "0x00 B y e 0xff"
- Close

If all of these steps are done without any error, the experiment session
was considered to be successful. agl@'s post listed the ratio (WebSocket
success) / (HTTP success).

Thanks

---

From: Patrick McManus <[email-masked]>
Date: Thu, 25 Oct 2012 09:01:01 -0400

I know this has been said before by the Chrome team, but I too am starting
to see websocket bugs pile up due to transparent proxies that don't speak
upgrade and explicit proxies that don't let you tunnel to port 80 as
configured. (bluecoat and MS ISA/TMG are the most common I hear about).
ws:// hasn't been very successful in those environments but wss:// has
been. Good thing that in the websockets world there aren't a lot of legacy
urls to worry about - not true for http :(

For http2, I don't think it is enough to just fail fast to http/1 when for
most cases we could get those users speaking HTTP/2 over tls on 443. A
mechanism like Alternate-Protocol accomplishes that and I think that is a
more important property than upgrading in band (which is admittedly nice!).

As for caching successful upgrades or successful A-P's they pretty much
have the same set of tradeoffs.

-P

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 25 Oct 2012 15:13:49 +0200

Hi Patrick,

> urls to worry about - not true for http :(

On the other hand, we could say that one big difference between WS and HTTP
is that with WS you don't have much possible fallback while with HTTP the
fallback is HTTP/1.1.

Regards,
Willy

---

From: Yoav Nir <[email-masked]>
Date: Thu, 25 Oct 2012 15:21:46 +0200

On Oct 25, 2012, at 3:01 PM, Patrick McManus wrote:

> I know this has been said before by the Chrome team, but I too am starting to see websocket bugs pile up due to transparent proxies that don't speak upgrade and explicit proxies that don't let you tunnel to port 80 as configured.

Transparent proxies should learn about Upgrade: and either support websockets/http2 or remove the header. Hopefully they will have done this by the time HTTP/2.0 over port 80 is actually deployed.

> (bluecoat and MS ISA/TMG are the most common I hear about). ws:// hasn't been very successful in those environments but wss:// has been. Good thing that in the websockets world there aren't a lot of legacy urls to worry about - not true for http :(
> 
> For http2, I don't think it is enough to just fail fast to http/1 when for most cases we could get those users speaking HTTP/2 over tls on 443. A mechanism like Alternate-Protocol accomplishes that and I think that is a more important property than upgrading in band (which is admittedly nice!).

So rather than go to HTTP/1.1 you'd prefer going to TLS with an anonymous ciphersuite?  I don't think that would help much, as the transparent proxies that also MitM SSL are getting ever more popular.

> As for caching successful upgrades or successful A-P's they pretty much have the same set of tradeoffs.

Yes, but I don't think it's all that bad. Browsers and operating systems have some feeling of location, which changes when any of your IP addresses change, or your routing table changes, or the configuration of the proxy or DNS servers changes. Browsers notice these things so that they can quickly re-establish connections, re-query the DNS, and reload pages that had some failure.  If you have this event also invalidate the cache, so that you go through the Upgrade handshake again, you should usually not have a case where a proxy has suddenly got in your path and won't let you use HTTP/2 over TCP port 80 any more.

Yoav

---

From: Patrick McManus <[email-masked]>
Date: Thu, 25 Oct 2012 09:41:14 -0400

On Thu, Oct 25, 2012 at 9:21 AM, Yoav Nir <[email-masked]> wrote:

>
>
> > For http2, I don't think it is enough to just fail fast to http/1 when
> for most cases we could get those users speaking HTTP/2 over tls on 443. A
> So rather than go to HTTP/1.1 you'd prefer going to TLS with an anonymous
> ciphersuite?  I don't think that would help much, as the transparent
> proxies that also MitM SSL are getting ever more popular.
>
>
The websockets results show that the TLS approach will work well with
legacy infrastructure and yes I'd rather have everyone that can use http2
actually use it before falling back to http/1 because there are significant
performance gains to be had.

no doubt that an unauth-tls-http2 would create growth in the transparent
ssl proxy market but that isn't really a big legacy compatibility isssue.
That's one place where I would expect to see compatibility with http2. For
the long tail of compat problems then sure, we're down to http/1 fallback.

> Yes, but I don't think it's all that bad

I concur with that.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 25 Oct 2012 15:54:16 +0200

> performance gains to be had.

I see a good point here : I expect that support for Upgrade and unfiltered TLS
will be very complementary. I mean :
  - in completely opened networks, both will work.
  - In corporate networks, port 443 will continue to work with the usual
    whitelists and will not be suited for HTTP/2, however these infrastructures
    are maintained and managed and the proxies will quickly be upgraded to
    support the Upgrade if not ready yet.
  - and at places where interception proxies are in place (eg: ISPs, mobile
    phone ops), it's common to find the opposite : interception proxies are
    a bit outdated and not well maintained, but port 443 is wide open.

The reason for this complementarity is simple : corporate networks need to
control what enters and leaves the network, while ISPs need to save bandwidth
and optimize only.

Thus I would not be surprized to see a success rate close to 100% with the
following sequence to sites accepting both ports when HTTP/2 is released :
  1) HTTP Upgrade on port 80
  2) fallback to TLS on port 443

And when both fail, clearly 1.1 is the only way to go.

Regards,
Willy

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 25 Oct 2012 21:34:01 +0000

------ Original Message ------
From: "Yoav Nir" <[email-masked]>
To: "Patrick McManus" <[email-masked]>
Cc: "Gabriel Montenegro" <[email-masked]>;"Mark 
Nottingham" <[email-masked]>;"Willy Tarreau" <[email-masked]>;"Amos Jeffries" 
<[email-masked]>;"[email-masked]" <[email-masked]>
Sent: 26/10/2012 2:21:46 a.m.
Subject: Re: #385: HTTP2 Upgrade / Negotiation
>Yes, but I don't think it's all that bad. Browsers and operating systems have some feeling of location, which changes when any of your IP addresses change, or your routing table changes, or the conf…

On this note, I think it would be really useful if there could be some 
machine-readable way to query a proxy for HTTP-related capabilities.  
If that were part of 2.0, then failure to respond appropriately to such 
a query would in itself be an indication of lack of support for 2.0.

The client could do this test whenever it became aware of using a new 
proxy, e.g. before anyone types a URI or clicks a shortcut.  So by the 
time the first user request is made, the client should know all it 
needs to know about the proxy.  We should at least be able to solve 
that part of the problem.

Adrien

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Thu, 25 Oct 2012 21:39:52 +0000

------ Original Message ------
From: "Willy Tarreau" <[email-masked]>
> 2) fallback to TLS on port 443
I think 2 could introduce a significant delay.  What proportion of 
websites support TLS on 443?

Or is this purely to get past intermediaries to a site you already know 
supports 2.0?  Or how do you know already that the site is available on 
443, and if someone clicked a http:// URL, is it valid to make a https 
connection?  Sometimes it's a different site on the different port.

In fact for that reason alone, you can't change the port that the URI 
specified from 80 to 443.  You can end up getting the wrong site.

I foresee a bunch of problems where 1 fails due to intercepting proxy 
not understanding Upgrade, and 2 fails because the site is http only on 
80 only.

Or did I misunderstand what you're getting at?

Regards

Adrien

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 25 Oct 2012 23:46:18 +0200

Hi Adrien,

> websites support TLS on 443?

it is increasing but some of them will still block 443 (mine used to
until not long ago).

> supports 2.0?

Could be a reasonable prerequisite.

> Or how do you know already that the site is available on 

Hmmm you're right, I had not thought about this one. It could even
cause a security issue if the port is relayed to someone else who
controls the TLS termination. Please forget this idea.

> Or did I misunderstand what you're getting at?

No you did not, and instead you proved that my suggestion was stupid,
so there's no point thinking about it anymore :-)

Thanks !
Willy

---

From: Amos Jeffries <[email-masked]>
Date: Fri, 26 Oct 2012 14:28:09 +1300

On 26/10/2012 10:34 a.m., Adrien W. de Croy wrote:
>>> I know this has been said before by the Chrome team, but I too am 
>>> starting to see websocket bugs pile up due to transparent proxies 
>>> that don't speak upgrade and explicit proxies that don't let you 
>>> tunnel to port 80 as configured.
>>>
>>
>>
>> Transparent proxies should learn about Upgrade: and either support 
>> websockets/http2 or remove the header. Hopefully they will have done 
>> this by the time HTTP/2.0 over port 80 is actually deployed.
>>
>>
>>>
>>> (bluecoat and MS ISA/TMG are the most common I hear about). ws:// 
>>> hasn't been very successful in those environments but wss:// has 
>>> been. Good thing that in the websockets world there aren't a lot of 
>>> legacy urls to worry about - not true for http :(
>>>
>>> For http2, I don't think it is enough to just fail fast to http/1 
>>> when for most cases we could get those users speaking HTTP/2 over 
>>> tls on 443. A mechanism like Alternate-Protocol accomplishes that 
>>> and I think that is a more important property than upgrading in band 
>>> (which is admittedly nice!).
>>>
>>
>>
>> So rather than go to HTTP/1.1 you'd prefer going to TLS with an 
>> anonymous ciphersuite?  I don't think that would help much, as the 
>> transparent proxies that also MitM SSL are getting ever more popular.
>>
>>
>>>
>> Yes, but I don't think it's all that bad. Browsers and operating 
>> systems have some feeling of location, which changes when any of your 
>> IP addresses change, or your routing table changes, or the 
>> configuration of the proxy or DNS servers changes. Browsers notice 
>> these things so that they can quickly re-establish connections, 
>> re-query the DNS, and reload pages that had some failure.  If you 
>> have this event also invalidate the cache, so that you go through the 
>> Upgrade handshake again, you should usually not have a case where a 
>> proxy has suddenly got in your path and won't let you use HTTP/2 over 
>> TCP port 80 any more.
>>
>
> If that were part of 2.0, then failure to respond appropriately to 
> such a query would in itself be an indication of lack of support for 2.0.
>

You mean "OPTIONS * HTTP/..." ?

Amos

---

From: "Adrien W. de Croy" <[email-masked]>
Date: Fri, 26 Oct 2012 01:47:18 +0000

------ Original Message ------
From: "Amos Jeffries" <[email-masked]>

>>
>>On this note, I think it would be really useful if there could be 
>>some machine-readable way to query a proxy for HTTP-related 
>>capabilities. If that were part of 2.0, then failure to respond 
>>appropriately to such a query would in itself be an indication of 
>>lack of support for 2.0. 
>>

that kind of thing, but need to be more specific about what and how to 
advertise things.  RFC2616 is fairly vague.  E.g. bodies on request or 
response aren't specified, but may be specified in extension etc, 
otherwise you get to send headers indicating optional features, but 
like what?  Accept-range, Allow, T-E?

maybe I need to do some more research for docs describing specific 
option advertisements.  Is there a list of such specs anywhere?

Adrien

---

From: Salvatore Loreto <[email-masked]>
Date: Sat, 27 Oct 2012 08:59:24 +0300

Hi Adrien

see in line

>>> The client could do this test whenever it became aware of using a 
>>> new proxy, e.g. before anyone types a URI or clicks a shortcut.

that is an interesting!
however it would work only and exclusively for explicit proxies, isn't it?
a lot of issues are with transparent/intercepting proxies

>>> So by the time the first user request is made, the client should 
>>> know all it needs to know about the proxy. We should at least be 
>>> able to solve that part of the problem. 
>>

moreover, if it is just for explicit proxies this kind of mechanism 
would only work with the one configured in the client.
However even if the configured proxy supports /2.0 you can not be sure 
that there is the support all the way long till the server;
there can be always another proxy in between the supported proxy and the 
server that is not 2.0 updated

/Sal

-- 
Salvatore Loreto, PhD
www.sloreto.com