Subject: Fwd: Re: [tcpm] FW: Call for Adoption: TCP Tuning for HTTP

From: Joe Touch <[email-masked]>
Date: Wed, 2 Mar 2016 14:34:38 -0800

Hi, all,

This doc was noted on the TCPM list.

See my observations below.

Joe

-------- Forwarded Message --------
Subject: Re: [tcpm] FW: Call for Adoption: TCP Tuning for HTTP
Date: Wed, 2 Mar 2016 12:25:07 -0800
From: Joe Touch <[email-masked]>
To: Scharf, Michael (Nokia - DE) <[email-masked]>,
[email-masked] Extensions <[email-masked]>
CC: [email-masked]

On 3/2/2016 1:39 AM, Scharf, Michael (Nokia - DE) wrote:
> I assume this could be of interest to the TCPM community.

I have doubts:

- it reads like a Linux manual page

	All Linux-specific references and commands would need to be
	moved to an appendix to be useful as an RFC.

- this repeats (sometimes correctly, sometimes in error) existing advice

J. Heidemann, K. Obraczka, J. Touch, “Modeling the Performance of HTTP
Over Several Transport Protocols,” IEEE/ACM Transactions on Networking,
V5, N5, Oct. 1997, pp.616-630.

T. Faber, J. Touch, and W. Yue, “The TIME-WAIT state in TCP and Its
Effect on Busy Servers,” in Proc. IEEE Infocom, 1999, pp. 1573-1583.

- it has significant errors

	TIME-WAIT issues apply to servers, not clients.

	Nagle has been known to perform poorly for multibyte
	interactive traffic for a very long time, including not
	only web traffic but also multi-byte character or keyboard
	signals.

	Disabling slow-start after idle is safe only with pacing.
	Without pacing, the resulting traffic can generate a burst
	that was never experienced and result in both poor performance
	for the current connection and potential impact to competing
	traffic.

(those are just a few)

Overall, I think a man page might be useful, but this summary isn't
useful for the IETF.

Joe

> -----Original Message-----
> From: Scharf, Michael (Nokia - DE) 
> Sent: Wednesday, March 02, 2016 10:37 AM
> To: 'Mark Nottingham'; HTTP WG
> Cc: [email-masked]; Daniel Stenberg
> Subject: RE: Call for Adoption: TCP Tuning for HTTP
> 
> The document refers to several TCPM RFCs with experimental status, e.g., in Section 3. That may have to be taken into account when heading towards BCP status.
> 
> Michael
> (TCPM co-chair)
> 
> 
> -----Original Message-----
> From: Mark Nottingham [mailto:[email-masked]] 
> Sent: Wednesday, March 02, 2016 6:47 AM
> To: HTTP WG
> Cc: [email-masked]; Daniel Stenberg
> Subject: Call for Adoption: TCP Tuning for HTTP
> 
> [ copying Alison as our Transport Tech Advisor ]
> 
> Daniel has kindly started a document about how HTTP uses TCP, both for /1 and /2:
>   <https://tools.ietf.org/html/draft-stenberg-httpbis-tcp>
> 
> We haven't explicitly discussed this at a meeting, but I have heard interest in this topic from a variety of folks.
> 
> What do people think about adopting this with a target of Best Current Practice?
> 
> Please comment on-list.
> 
> Regards,
> 
> --
> Mark Nottingham   https://www.mnot.net/
> 
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 3 Mar 2016 00:21:25 +0100

> 	TIME-WAIT issues apply to servers, not clients.

Sorry but no it's the opposite. A server has no issue with knowing that
a SYN belongs to a new session by seeing its ISN greater than the end
of the previous window. On the opposite, a client cannot know if the
remote server it wants to connect to is safe for reuse and will refrain
from establishing a new connection during the whole TIME_WAIT state,
effectively preventing itself from doing its job.

Regards,
Willy

---

From: Joe Touch <[email-masked]>
Date: Wed, 2 Mar 2016 15:34:42 -0800

On 3/2/2016 3:21 PM, Willy Tarreau wrote:
> Sorry but no it's the opposite. 

TIME-WAIT is a state caused by the side that closes the connection.

In the bulk of HTTP connections, the server closes the connection,
either to drop a persistent connection or to indicate "EOF" for a transfer.

Clients generally don't enter TIME-WAIT, so reducing the time they spend
in a state they don't enter has no effect.

> A server has no issue with knowing that
> of the previous window. 

That's exactly the reason the server keeps information in the TIME-WAIT
state.

> On the opposite, a client cannot know if the
> remote server it wants to connect to is safe for reuse 

TIME-WAIT isn't just for new connections; it's to protect against
injecting traffic from previous connections that is delayed into new
connections...

> and will refrain

If that's what it doesn, that's not TIME-WAIT - it's some new state in
the OS to avoid the possibility of hitting a TIME-WAIT at the server.
That's mislabeled at best, and defeats the entire purpose of the
TIME-WAIT at the server anyway.

Joe

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 3 Mar 2016 07:55:45 +0100

> TIME-WAIT is a state caused by the side that closes the connection.

... that closes the connection *first* (since both sides close it).
This point is important because it means some proxies often should
better wait for a passive close from a server than deciding to
close themselves.

> either to drop a persistent connection or to indicate "EOF" for a transfer.

Yes.

> in a state they don't enter has no effect.

They can if they close first and that's exactly the problem we absolutely
want to avoid. There are certain cases where we had to put warnings in
rfc7230/7540, especially in relation with proxies. The typical case is
when a client closes a connection to a proxy (eg: a CONNECT tunnel) and
the proxy is supposed to in turn close the connection to the server. In
this case the proxy is the connection initiator, and it can very quickly
condemn all of its source ports by accumulating TIME_WAITs there. But the
same problem exists with idle persistent connections that clients must
avoid to close themselves if there's any hope the server will close soon.

> injecting traffic from previous connections that is delayed into new

Yes I'm well aware of this :-)

> TIME-WAIT at the server anyway.

No I'm not saying any such thing, I'm saying that by all means the
server must close first to keep the TIME_WAIT on its side and never
on the client side. A TIME_WAIT on a server is very cheap (a few tens
of bytes of memory at worst) and can be recycled when a new valid SYN
arrives. A TIME_WAIT on the client is not recyclable. That's why
TIME_WAIT is a problem for the client and not for the server.

The problem is that in some cases it's suggested that the client
closes first and this causes such problems. The only workaround for
the client is to close with an RST by disabling lingering, but that's
really ugly and unreliable : if the RST is lost while the server is
in LAST_ACK (and chances are that it will happen if the ACK was lost
already), the new connection will not open until this connection
expires.

So by all means we must do whatever we can to avoid to cause
TIME_WAITs to be accumulated on the client side and that was the
point mentionned in the document since it's supposed to be used as
a reference for future protocol designs. For example in HTTP/2, the
GOAWAY frame makes this a bit easier to take care of, since we can
declare an intent to close that will cause the other side to close.

Also, there are people who face this issue and work around them using
some OS-specific tunables which allow to blindly recycle some of these
connections and these people don't understand the impacts of doing so.
The doc will have to be clear enough to discourage them from doing so,
and to adapt the client code instead.

Willy

---

From: Joe Touch <[email-masked]>
Date: Thu, 3 Mar 2016 10:00:12 -0800

> ... that closes the connection *first* (since both sides close it).

Of course.

> better wait for a passive close from a server than deciding to

Transparent proxies don't have that choice - they're governed by the
semantics of the connection (whether EOF == close or not).

Non-transparent proxies shouldn't be opening one connection per
transaction anyway; they ought to use one or more persistent connections
and leave them open while they are interacting with the proxy. If they
do this, there won't be an issue with who closes the connection because
the close frequency should be very low.

> want to avoid.

TW buildup has two effects:

	1) limits the number connection rate to a given IP address

	2) consumes memory space (and potentially CPU resources)

Neither is typically an issue for HCI-based clients. Servers have much
higher rate requirements for a given address when they act as a proxy
and consume more memory overall because they interact with a much larger
set of addresses.

> There are certain cases where we had to put warnings in
> condemn all of its source ports by accumulating TIME_WAITs there. 

That speaks to a mismanagement of port resources. If they are allocated
on a per-IP basis, they won't run out. The error is in treating the pool
of source ports as global across all IP addresses, which TW does not
require.

> But the

I agree - this is the same problem if it exists -- port mismanagement.

> No I'm not saying any such thing,

OK - glad to hear that..

> I'm saying that by all means the
> of bytes of memory at worst) 

It costs exactly the same on the client and the server when implemented
correctly.

> and can be recycled when a new valid SYN
> arrives.

The purpose of TW is to inhibit new SYNs involving the same port. When a
new SYN arrives on another port, that has no impact on existing TWs.

> A TIME_WAIT on the client is not recyclable. That's why

See above; TW is *never* recyclable.

> closes first and this causes such problems.

That actually helps the server (see our 99 Infocom paper).

> The only workaround for
> the client is to close with an RST by disabling lingering,

That's not what SO_LINGER does. See:
http://man7.org/linux/man-pages/man7/socket.7.html

> but that's

TCP has a significant error regarding RSTs; the side that throws a RST
on an existing connection should really go into TW - for all the same
reasons that TW exists in the first place, to protect new connections
from old data still in the network.

> a reference for future protocol designs. 

This is the error I mentioned, and it should not be recommended.

> For example in HTTP/2, the

Receiving a GOAWAY says "don't start new connections", not "shut this
one down now". That's the right behavior, because it says nothing about
the semantics of existing connections.

> connections and these people don't understand the impacts of doing so.

They really ought to read the literature. It's been out there so long it
can probably apply for a driver's license by now.

> and to adapt the client code instead.

We don't need a new doc to address this, especially (IMO) incorrectly.

Joe

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 3 Mar 2016 19:44:18 +0100

> the close frequency should be very low.

It not that black and white unfortunately, and in practice it's very
common to see proxies fail in field above 500 connections per second
because their TCP stack was not appropriately tuned, and with the
default 60s TIME_WAIT timeout of their OS, they exhaust the default
28k source ports. The first things admins do in this case is to
enable tcp_tw_recycle (which basically causes timewaits to be killed
when needed), and this appears to solve the situation while it makes
it even worse.

Among the solutions, we can count on :
  - putting back idle connections into pools hoping that they will be
    reusable. Connection reuse rate still remains low on average.
  - keeping a high enough keep-alive idle timeout on the proxy and a
    smaller one on the server (when the proxy is a gateway installed
    on the server side) hoping for the server to close first
  - appropriately add "connection: close" into outgoing requests to
    ask the server to close after the response.
  - disabling lingering before closing when the HTTP state indicates
    the proxy has received all data
  - doing whatever is imaginable to avoid closing first

These are just general principles and many derivatives may exist in
various contexts, but these ones are definitely important points that
HTTP implementors have to be aware of before falling into the same
traps as the ones having done so previously.

> 	1) limits the number connection rate to a given IP address

Exactly.

> 	2) consumes memory space (and potentially CPU resources)

This one is vey cheap. A typical TW connection is just a few tens of bytes.

> Neither is typically an issue for HCI-based clients.

I don't know what you call HCI here, I'm sorry.

> Servers have much

Servers are not penalized at all with the connection rate since it only
limits the *outgoing* connection rate and not the incoming one. There's
never any ambiguity when a SYN is received regarding the possibility that
the connection still exists on the other side, which is why TW connections
are recycled when receiving a new SYN. Regarding the memory usage, it
remains very low compared to the memory used by the application itself.
My personal record was at 5.5 million timewaits on a server at 90000
connections per second. It was only 300 MB of RAM on a server having
something like 64 GB. And not everyone needs 90k conns/s but everyone
needs more than 500/s nowadays in any infrastructure.

> on a per-IP basis, they won't run out.

Yes they do, that's the problem everyone running a load balancer faces!
The highest connection rate you can reach per server is around 1000 with
64k ports! That started not being enough 15 years ago!

> The error is in treating the pool

No, the problem is to keep a TW which blocks a precious resource which is
the source port that is only addressed on 16 bits!

> It costs exactly the same on the client and the server when implemented

It costs the same except that in one case it prevents a connection from
being established while in the othe case it does not. I've seen people
patch their kernels to lower the TIME_WAIT down to 2 seconds to address
such shortcomings! Quite frankly, this workaround *is* causing trouble!

> new SYN arrives on another port, that has no impact on existing TWs.

I'm always talking about the same port. On todays hardware and real world
workloads, source ports can be reused every second (60k conns/s). Only the
server with TIME_WAIT can tell whether or not an incoming SYN is a retransmit
or a new one. The client knows it's a new one but doesn't know if the server
is still in LAST_ACK or has really closed, and due to this uncertainty it
refrains from connecting.

> See above; TW is *never* recyclable.

Yes it definitely is on the server side, which is the point. When you
receive a SYN whose ISN is higher than the end of the current window,
it's a new one by definition (as indicated in RFC1122).

> That actually helps the server (see our 99 Infocom paper).

Sure since the server doesn't receive any more traffic from this client,
that definitely helps, but the point is to ensure traffic flows between
the two hosts, not that one of them refrains from connecting.

> http://man7.org/linux/man-pages/man7/socket.7.html

But in practice it's used for this. When you disable lingering before
closing, you purge any pending data which has the benefit that the data
you just received from the server that carried an ACK for data you don't
have anymore triggers a reset. Yes it's absolutely ugly but you have no
other option when you are a client and are forced to close first due to
the protocol. Don't forget that we're discussing a document whose outcome
should be that protocols are designed in the future to avoid such horrible
workarounds.

> from old data still in the network.

There are many other issues regarding RST. When you send an RST through
a firewall, you'd better cross fingers for it not to be lost between the
firewall and the destination, otherwise chances are that you won't get
a second chance. That's one of the reasons why I'd love to live in a
world where a client never has to close first.

> can probably apply for a driver's license by now.

When people see their production servers stall at 5% CPU because their LBs
or proxies can't open new connections while full of TIME_WAIT, what they
do is ask their preferred search engine which simply proposes them such
advices :
  - https://ihazem.wordpress.com/2012/02/07/reducing-time_wait-socket-connections-recyclereuse/
  - http://serverfault.com/questions/212093/how-to-reduce-number-of-sockets-in-time-wait
  - http://kaivanov.blogspot.fr/2010/09/linux-tcp-tuning.html
  - http://www.linuxbrigade.com/reduce-time_wait-socket-connections/
  - http://www.stolk.org/debian/timewait.html

Yes they all involve the wrong and nasty workarounds consisting in allowing
to recycle outgoing TIME_WAIT connections, which is the worst ever thing to
do (except the last one which explains how to modify the TW timeout in the
kernel).

This is a *real* problem in field, it has been for a while because some
protocols have been designed for lower loads without imagining that one
day source ports would be reused that often. While we have to deal with
this the best we can, it's important to ensure the same mistake is not
done again in the future.

Regards,
Willy

---

From: Joe Touch <[email-masked]>
Date: Thu, 3 Mar 2016 11:15:36 -0800

> when needed), and this appears to solve the situation while it makes

Those proxies are acting as servers. We've known about the server TW for
a long time.

>     on the server side) hoping for the server to close first

This just moves the problem to the server where it is much worse in
aggregate.

>     ask the server to close after the response.

Same thing.

>     the proxy has received all data

Linger has nothing to do with this issue.

> traps as the ones having done so previously.

The important issue is largely the buildup of TW at the server, and this
has been known for a long time, as have the mitigations.

> I don't know what you call HCI here, I'm sorry.

Human-computer interface, i.e., client whose traffic is driven by a
person clicking on things.

> limits the *outgoing* connection rate and not the incoming one.

When TW accumulates, new connections are rejected at the side that
accumulates the TW. That can happen on either side.

> There's
> the connection still exists on the other side, 

That's what the TW is intended to inhibit.

> which is why TW connections
> are recycled when receiving a new SYN. 

That's exactly the opposite of what TW does.

> Regarding the memory usage, it

Memory impact depends on the device.

> 64k ports! That started not being enough 15 years ago!

That rate is nearly never seen on clients (which is the context of this
part of the thread); it's certainly seen on servers and proxies.

> the source port that is only addressed on 16 bits!

The TW should be blocking based on the socket pair - i.e., source IP,
dest IP, source port, dest port. Only one of these is fixed for HTTP
(the dest port). At either the client or server, typically the other
side's IP address also varies more than enough to avoid this issue - the
primary exception being proxy interactions.

> being established while in the other case it does not.

TW prevents a connection in either direction if implemented according to
spec.

> I've seen people

Again, we've known that for nearly two decades.

> refrains from connecting.

If the server initiates the close, it never goes into LAST_ACK (the
client would). A node can end up in LAST_ACK only if the final ACK of
the closing three-way handshake is lost, which should not be the typical
case. However, it shouldn't matter that it refrains from connecting at
that point - the other side should be in TIME-WAIT anyway.

> it's a new one by definition (as indicated in RFC1122).

RFC1122's statement on TIME-WAIT has nothing to do with the ISN the
server receives; it has to do with the ISN the server assigns. You don't
"recycle" the TIME-WAIT; you effectively reopen the connection with the
same port pair. But that then requires keeping more state in the TW.

> the two hosts, not that one of them refrains from connecting.

Neither side "refrains from connecting"; TW is intended to block certain
connection attempts, not prevent them from being attempted.

> But in practice it's used for this.

That practice is incorrect and should never be recommended.

> When you disable lingering before
> have anymore triggers a reset. 

Linger is intended to let the other side keep sending you data after you
issue a close - that's entirely valid TCP behavior.

I.e., when you use the option as intended, it won't accomplish what you
want.

> Yes it's absolutely ugly but you have no

The doc should provide correct advice, and certainly ought to not
reinvent 17-year old wheels.

> a second chance. 

Agreed - RSTs ought to be used only in "emergencies" to abort
connections, not as a hack to jump around the protocol space.

> That's one of the reasons why I'd love to live in a

The RST is bad in either direction, and the world you want to live in
was what we had 17 years ago and prompted those other papers.

> do is ask their preferred search engine which simply proposes them such
...

They ought to do better research (and so should this doc, IMO).

...
> done again in the future.

Agreed. Again, please refer to those two papers, among others.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 3 Mar 2016 12:18:21 -0800

On 3/3/2016 11:15 AM, Joe Touch wrote:
...
> same port pair. But that then requires keeping more state in the TW.

And while we're at it, this rule is nearly completely defeated if ISNs
are randomized or treated as a single resource throughout a machine
(rather than being specific to a socket pair).

Joe

---

From: Patrick McManus <[email-masked]>
Date: Thu, 3 Mar 2016 15:43:38 -0500

Hi Wily, Joe,

This message is a bit of a diversion from the discussion so far. sorry bout
that.

> such shortcomings! Quite frankly, this workaround *is* causing trouble!

really? That's fascinating to me. Can you provide background or citations
on what kind of trouble has been attributed to this and the scenario where
it was done?

You don't need to go through the theoretical - I know what TW could
conceptually catch - but the assertion about the shorter timeout causing
field problems is something I'd love to understand better.

For TW to be useful protection it also has to be paired with
re-transmission and some application states that will be impacted by the
screwup which reduces its utility, particularly for HTTP. I read the above
statement as saying that TW is indeed useful in the field at the
application layer - but maybe it is referencing some side effect I'm not
thinking of rather than the vulnerability of not using it.

Those kinds of post mortem war stories where it is seen in the field are
pretty interesting and help inform the discussion about whether the cure is
worse than the disease. My inclination has generally been that TW doesn't
help a lot in practice and has some limitations and causes pain (as well
documented in this thread.). So it would be interesting to look at the
fallout of a situation it could have helped with.

This feels a bit like the musing over the subpar utility of the tcp
checksum on high bandwidth networks. For that one, the answer at least in
the http space is 'use https for integrity and sort out the rare error at
the application level'. I'm wondering if that's the right advice in the
time_wait space for http as well.. we're really still talking about
integrity. Go ahead and turn it off - just make sure you're running a
higher level protocol that won't confuse old data with new data.

(Fun paper: http://conferences2.sigcomm.org/imc/2015/papers/p303.pdf showed
that at the tail of a ping survey, 1 % of replies from 1% of addresses
needed >= 145 seconds to arrive. And that's just delay - not
retransmission. A truly protective TW is a very large number.)

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 3 Mar 2016 22:11:05 +0100

> Those proxies are acting as servers. We've known about the server TW for

No, I'm sorry if my explanation was not clear. I'm talking about the outgoing
connection from the proxy to the server. Ie most common case :

   internet  ----> reverse ----> server
   clients          proxy

The problem is not with the TW on the internet-facing side of the proxy
but with the server-facing side.

> This just moves the problem to the server where it is much worse in

No, they're cheap and that what *everyone* having to deal with more
than 1000 connections per second has to do.

> Linger has nothing to do with this issue.

I explained already, I'm sorry.

> has been known for a long time, as have the mitigations.

This comes with a very small cost. The main cost of the TW buildup on the
server is that netstat can take ages to list sockets. This compared to a
completely stopped web site is pretty minor.

> person clicking on things.

OK thanks. So a browser. This is indeed the case where you don't care
for now (except when bogus scripts run connections in loops but that
causes other issues as well).

> accumulates the TW. That can happen on either side.

No, no no. Please read what I explained twice. This is RFC1122. If this
is what you believe I understand why you're against accumulating them on
the server. But if that was try or any internet-connected host, the web
would be pretty sad nowadays...

Let me restate it :
  - the client cannot bypass the TW state because it has no way to know
    whether or not the server has closed after seeing the last ACK or
    is still waiting for it.

  - the server when it sees a SYN with an ISN larger than the end of the
    previous window, *knows* that the client has closed, otherwise the
    client would be in LAST_ACK and couldn't send a SYN in this state.
    This is why servers recycle connections, and only in this case.

> That's exactly the opposite of what TW does.

No I disagree and RFC1122 disagrees with you as well, just like any web
server you can connect to over the net.

> Memory impact depends on the device.

Absolutely. But 56 bytes (IPv4) or 84 bytes (IPv6) are already much smaller
than the minimum 536 bytes that a device is supposed to accept when no MSS
is negociated.

> part of the thread); it's certainly seen on servers and proxies.

Yes it is on proxies and proxies are clients. That's the point.
You find it as well on database clients by the way. Just search the
net for people reporting the inability to connect to mysql after
accumulating timewaits on the client. Just because the protocol was
misdesigned, the client says "QUIT", the server responds "OK", then
the client closes. BAM. One TW blocking this port for 60 seconds.
After 64k connections the application cannot connect to the server
anymore.

>From the beginning we're talking about connection initiation issues by
TW accumulated on the client side, particularly on proxies, but not only.
Client here refers to the TCP client since the document is about TCP tuning
and best protocol practises to scale better and to better resist attacks,
not the client in terms of the role of the device in the complete chain.

> Again, we've known that for nearly two decades.

Then why do you insist on enforcing this well-known problem while it's
absolutely not necessary ?

> client would).

Absolutely agree. I'm talking about the case where the client closes first.

> A node can end up in LAST_ACK only if the final ACK of
> case.

It happens with the same rate as other ones, and this state tends to last
more than other ones for a simple reason : some intermediary NAT devices
considerably reduce their timeout once they see the first FIN packet (some
at least take care of the second one), so that past this point the random
loss of the last ACK will become more visible because retransmits will not
reach the other side. That's quite common and a non-problem most often.

> However, it shouldn't matter that it refrains from connecting at

Definitely, that was the case I mentionned, where the client being
in TW must refrain from recycling since it doesn't know if the server
on the othe side is in LAST_ACK.

> server receives; it has to do with the ISN the server assigns.

But it's what is implemented everywhere. Just send a SYN in window to
any internet-facing host, you'll get an ACK in response. Send a SYN
above the window and you get a SYN-ACK. Most implementations even support
PAWS and then a higher TCP timestamp helps sorting out the new and the
old packets. That's what is needed for example when some firewalls break
the end-to-end transparency by randomizing sequence numbers that were
already properly randomized.

> You don't

Excuse-me for the terminology but that's exactly what I call "recycling"
since a TIME_WAIT socket gets destroyed and recreated in SYN-RECV state.

> connection attempts, not prevent them from being attempted.

Yes thus as seen from the network, the client refrains from connecting since
its internal stack blocks its application's connection attempts.

> That practice is incorrect and should never be recommended.

But there's no other option when you have to close the connection and you're
on the client side if you don't want to run out of ports in the same second.

> issue a close - that's entirely valid TCP behavior.

No, it lets your local TCP stack continue to send the data pending in
the window after the application closes. Some operating systems call
such connections "orphans". You can't receive data over a connection
after you've closed it, it immediately causes an RST to be emitted,
which is one of the issues caused with redirects on POST in HTTP, you
have to drain all the incoming data to be sure no RST is emitted or
you risk the other side never to get your response. I think you were
mixing this with a FIN_WAIT state where you have performed the
shutdown(WR) and are still receiving data, but that's pretty standard
TCP.

> reinvent 17-year old wheels.

What 17-year old wheels ? The only one I know about consists in patching
kernels to force shorter timewaits in order not to block outgoing
connections when the rate approaches 1000/s. Until we have 32 bits for
the source port, these are the only two options. At some point one must
not wonder why more and more the transport is migrating to userland :-/

> connections, not as a hack to jump around the protocol space.

The problem we have is that the protocol space has not grown in the
last 35 years while the average bandwidth might have grown by 1 or
10 millions. This is an amazing achievement for such a design by then
bu we still have to live with todays needs.

> was what we had 17 years ago and prompted those other papers.

No, I'm leaving in 2016 where load balancers have to distribute all the
traffic they receive to the servers that are locate behind them and
where connection rates above 10000/s per server are quite common. Multiply
this by the 60s default time_wait and you need 600k source ports. I'm sorry
but I cannot encode this on 16 bits, and the fix is implemented in every
operating system : have cheap time_wait states that are easily recycled
-pardon, closed then reopened- when receiving a valid new SYN.

> They ought to do better research (and so should this doc, IMO).

Joe, don't take me bad, but from the beginning I find you very aggressive
and very negative regarding this document, and you constantly say that
everyone around is doing everything wrong. Please simply tell us how to
support 100k connections per second from a load balancer to a server,
each using a single IP address, with a single port for the server,
without sending RSTs from the client and without cheating on the client's
time_wait timeout. The only solution you leave to us is to increase the
source port range to 23 bits. That's not compatible with the protocol we
have. All server implementations correctly deal with time_waits and quick
port rollover and the net today works fine thanks to all this nice work
that has accumulated over the last 35 years. Why would anyone drop all of
this to get back to the antique 1000 connections per second limit with
64k ports or even 250 connections per second on certain operating systems ?

Thanks,
Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 3 Mar 2016 23:01:08 +0100

Hi Pat,

> This message is a bit of a diversion from the discussion so far. sorry bout

No pb, anyway we're discussing the issues our various implementations have
to face in field.

> on what kind of trouble has been attributed to this and the scenario where

Yes that's very simple. Someone runs a load balancer that has to close
server connections itself and that does not abuse SO_LINGER to force an
RST to be emitted. At one point this person notices that the load starts
to look like spikes approaching 1000 connections per second, is very
irregular and that a lot of connection errors are reported to the clients. 
The person runs netstat and sees thousands of established connections
from hosts everywhere on the net, and very few established connections
to the servers. Looking closer, it appears that the total number of
connections per server is 64512, all state included. That number sounds
familiar to all those who already extend their source port range from
1024 to 65535. The problem is that almost all connections are in
TIME_WAIT state, very few are established on this side. The reason is
that the point where you create connections faster than they expire has
been reached. The person then searches on the net and finds that this
problem is well known and various people use different methods depending
on the OS. On Solaris it was common to reduce the time_wait timeout which
was configurable (I believe it was the same as the close_wait but I could
be wrong, it was 10 years ago at least). On Linux people often suggest to
enable tcp_recycle (which you must never do).

In both cases, that appears to solve the issue but then the person starts
noticing that the connection times are very erratic and that an abnormal
number of TCP resets are reported by firewalls. And worse, since the symptoms
are the same as at the beginning of the problem, the person further reduces
the timeouts.

What happens is that from time to time under high load you'll experience
lost packets, especially in networks involving many connection opening/
closing because you can't count on the flow control imposed by the congestion
control algorithm for connections that are independant on each other, so there
are some packet storms that can exceed some switch buffers or even the
destination NIC's buffers.

When the lost packet is the last one (the ACK that the side in FIN_WAIT
sends in response to the FIN before entering the TIME_WAIT state), the
other one will retransmit the FIN. Please keep in mind we were considering
that the TCP client closed first so this TIME_WAIT side is the client. At
low enough loads, the first bad actor is a firewall between the LB and the
server (more and more often this firewall is implemented on the server
itself or worse, on the underlying hypervisor which nobody controls nor
monitors). Many firewalls are tuned with aggressive FIN_WAIT/TIME_WAIT
timeouts which cause their session to expire before the other side dares
to retransmit. The server then remains for a long time in LAST_ACK state,
resending this last ACK packet for some time before giving up.

But by this time, our nice client has already used all other ports and
needs to reuse this port. Since its TIME_WAIT timeout was reduced to
something lower than the server's LAST_ACK, it believes the port is
free and reuses it. It sends a SYN which passes through the firewall,
this SYN reaches the server which disagrees and sends an RST back
(when the client picked a new SYN above the end of previous window)
or an ACK which will generally be blocked by the firewall, or if the
firewall accepts it, will be transmitted to the client which will then
send an RST, wait one second and send the SYN again. As you can imagine,
this dance is quite counter-productive for the performance since you
convert hundreds of microseconds to multiples of seconds to establish
certain connections.

And above a certain connection rate, you don't need a firewall anymore
to mangnify the problem. If the client wraps its ports once a second
or so, often it's not possible anymore to set a TIME_WAIT timeout to
a lower value (due to the resolution of the settings), so they have
to use tcp_recycle to bypass the TIME_WAIT. You get the same effect as
above, with some connections being responded to with an RST and other
ones with an ACK instead of a SYN-ACK, to which the client sends a RST,
waits one second or so and tries again.

And guess what ? I've seen people rush haproxy in production with
"option nolinger" set on both sides to send RSTs to both sides just to
work around this, without even realizing that they were only hiding the
problem, transforming long connect times into truncated responses :-(

> field problems is something I'd love to understand better.

The TW is a very important state if you want to guarantee correctness
on your sessions in a situation where ports wrap quickly (and that's
the only case we're interested in BTW).

> thinking of rather than the vulnerability of not using it.

No, the application even doesn't know about it (until it fails to connect
because of it of course). However when the application is aware of how it
works, what protection it provides and what issues it can cause, then it
can try to do its best not to have to cheat on it nor to be impacted by
it. For example, I remember on Solaris 8 you had 32768 source ports and
240s time_wait timeout by default. That's a limit of only 136 connections
per second before being blocked if the client closes first. I remember it
even caused an Apache benchmark to fail at a customer's, they said that
"Apache couldn't go beyond 100 connections per second and it had a bug
because it couldn't use all the available CPU"!

> worse than the disease.

Definitely. And I remember your sentence "pcap or it didn't happen" :-)

> My inclination has generally been that TW doesn't

The real issues we have is that TCP stacks continue to use large timeouts
by default, and these ones multiplied by a number of retries quickly end
up being fairly large. It still makes sense over congested links to wait
a few seconds, but we just want to wait for a few *retransmits* not seconds.
Mobile operating systems seem to retransmit very fast. I've seen delays
around 150 ms between two SYNs on a smartphone in 2010, and the exponential
back-off didn't immediately start. That participate to the RAN congestion
of couse.

> higher level protocol that won't confuse old data with new data.

It doesn't work well for the connection setup as explained above. I could
agree with you that regarding integrity, multiple solutions are offered.
We even have MD5 checksums in standard if needed. But you need to be able
to quickly recover from an unclear situation, and TIME_WAIT helps you
never being in that unclear situation.

> retransmission. A truly protective TW is a very large number.)

Not surprized. I've seen the same order of magnitude in 2010 on a mobile
operator's RAN. The issue comes from deep buffers. Since the network link
can transparently handover between different speeds and the bandwidth is
huge, they need to store megabytes of data, literally. Well, at some point
you have to wait for these megabytes to reach the LAN over a shared link...
And there TCP's retransmits can hurt because at the output of the pipe,
you see a lot of duplicates, sometimes indicating that 20-50% of the data
were retransmits already.

Due to all this I'm not surprized at all about the ongoing works to port
the transport to userlad that we've seen last summer in Münster! There's
an amazing room for improvement that's left unexploited and that cannot
realistically be exploited by just tuning TCP without breaking
compatibility with everything already deployed!

Cheers,
Willy

---

From: Joe Touch <[email-masked]>
Date: Fri, 4 Mar 2016 10:21:41 -0800

Popping back up to the main point:

- there is very little in this doc that is specific to HTTP

	there is a lot of good advice in the literature
	about how to implement big servers (incl. TCP-based),
	and there is no good reason to try to condense that into
	an RFC

- the advice in this doc is often at odds with existing TCP advise

	Advice on changing/configuring TCP ought to happen on TCPM
	or TSVWG, not here.

IMO, the doc in its current form is no better than the kind of advice
that can be found on Google.

Joe

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 4 Mar 2016 19:38:39 +0100

> that can be found on Google.

The fact that it was written by one of the members of this WG, and
the fact that there are several supporters for its adoption tend to
indicate that some people here find more value in this document than
on random links found on Google. And don't please forget that the
links I sent you recommending to solve issues the absolute worst way
were the first ones reported by Google.

Willy

---

From: Joe Touch <[email-masked]>
Date: Fri, 4 Mar 2016 10:45:44 -0800

On 3/4/2016 10:38 AM, Willy Tarreau wrote:
> on random links found on Google. 

The value would be in exploring the literature on the subject of
designing large servers.

And don't forget that some people also find value in those links, so
community interest isn't necessarily a gauge of quality.

Joe

---

From: Patrick McManus <[email-masked]>
Date: Fri, 4 Mar 2016 16:16:22 -0500

To the extent this document can help promote efficient interop, rather than
just application tuning (and I acknowledge that is a blurry line), I think
it has promise as an IETF work and support adoption. I was skeptical of it
being able to play that role, but I think Wily has raised an interesting
point along those lines below. The document is clearly an -01 which needs
more exploration but it was nice of Daniel to put something forward.
(Daniel is afk for a bit - I'm sure we'll hear from him in the coming days.)

> resending this last ACK packet for some time before giving up.

Wily, thanks for the giant wall of text :). I have tried to distill it to
this, which I think is the essence of the discussion. I think the firewall
issue is one worth documenting for interop purposes - it can't know the end
state of receivers on either side so at least from a forwarding perspective
it should be configured permissively in an environment that anticipates low
TW. That seems like a valuable point to capture.

> convert hundreds of microseconds to multiples of seconds to establish

This is interesting, and a bit different than the integrity issues an
established frame could inject normally associated with TW (which https can
protect against - at least in a deterministic fast fail sense).

on the one hand, we've gone from the client having a high rate of  fast
0RTT fails (blocked from initiating by TW).. to a situation where 3 things
are going on

1] an unquantified "large" fraction of quick successes. (no packet loss
impact, no state machine out of sync and no blockage by a timer)
2] a number of cases of fast 1RTT fail where a RST is received by the client
3] a fraction that may succeed or fail, but much more slowly due to retry
behavior with its set of lovely constants and backoffs

If I've got that (at least vaguely) right, there seem to be situational
tradeoffs as to whether that is 'better' or not. Sounds like good
discussion material in a tuning doc :)

Within the scope of "TCP for HTTP" would you say something different? (and
sure, I agree legacy TCP might not be the fun thing here.. but its the
topic at hand.)

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 4 Mar 2016 23:03:47 +0100

Hi Pat,

On Fri, Mar 04, 2016 at 04:16:22PM -0500, Patrick McManus wrote:
(...)
> behavior with its set of lovely constants and backoffs

Yes that's exactly it.

> discussion material in a tuning doc :)

By definition, tuning has to be performed according to a specific
situation, otherwise it becomes the standard way of doing things.

> sure, I agree legacy TCP might not be the fun thing here.. but its the

My goal as well is not to dictate what should be done, but to enumerate
known impacts of various choices on various actors so that we can always
refer to that in the future for various parts of protocol designs (eg:
we've done this already when documenting the fact that we must drain a
request body before closing or that there's a DoS risk for proxies
passing a close received from the client). In the end it should help us
design more robust and more infrastructure-friendly protocols. And
that's exactly what Daniel has begun with this nice document.

Thanks for your support ;-)
Willy

---

From: Mark Nottingham <[email-masked]>
Date: Sat, 5 Mar 2016 09:56:01 +1100

Everyone,

A significant amount of IETF time and effort has been spent recently on trying to make sure that transport and application work is well-aligned; especially, that transport decisions are made with input from actual applications folks and vice-versa.

I've had encouraging discussions in the background with the transport ADs and others about coordinating to assure that this work -- if it gets taken on -- will help in those efforts. Right now we're figuring out exactly how that will occur, but suffice it to say that we won't be moving forward on this document until after B-A.

In the meantime, I'd encourage everyone to view this as an opportunity to learn more about how TCP is used by applications, and to learn more about how applications can get the most out of TCP. The discussion so far has been illuminating, please continue.

With regard to where the work is done -- that's certainly one of the things that we'll discuss, but I'll observe that while discussion about changing TCP should certainly take place in TCP-related fora, it's not reasonable to say that discussion about how applications  *use* TCP can only happen there.

With regard to the advice found on Google -- I think the point here is to provide an authoritative source to counter the misinformation out in the community. The document we're considering may not be in a suitable state to do that, but I'm confident we can get it there with appropriate participation.

Joe, you also said:

> The value would be in exploring the literature on the subject of designing large servers.

That is certainly *a* place we could find value. However, it is not the only place, and we're lucky enough to have a community of people on this list who develop and deploy some of the largest HTTP implementations (client, server and proxy), so I suggest it'd be best to consider them a source of value too.

Cheers,

> On 5 Mar 2016, at 5:21 AM, Joe Touch <[email-masked]> wrote:
>

---

From: Tim Wicinski <[email-masked]>
Date: Fri, 4 Mar 2016 20:05:31 -0500

DNSOP recently went through the process of updating RFC5966,
	"DNS Transport over TCP - Implementation Requirements"

and it was published just this week as RFC 7766
	https://datatracker.ietf.org/doc/rfc7766/

There was a lot of effort put in this document to reflect current 
technology in TCP, like Fast Open.

I support adoption, but I also think the document needs work.  I'll be 
happy to assist and make it look less like a list of linux kernel tunings,

tim

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 5 Mar 2016 08:38:13 +0100

> technology in TCP, like Fast Open.

This is a very interesting document, thanks for sharing. This is the
type of work we hope to see for HTTP, explaining the tradeoffs of
various solutions.

> happy to assist and make it look less like a list of linux kernel tunings,

The tuning is an important point, but only once all the explanation are
given for the trouble the tuning aims to fix. And ultimately it should
cover multiple operating systems.

Regards,
Willy

---

From: Ben Niven-Jenkins <[email-masked]>
Date: Sat, 5 Mar 2016 12:54:08 +0000

Willy, 

> On 5 Mar 2016, at 07:38, Willy Tarreau <[email-masked]> wrote:
> 

Does the doc need to cover operating systems at all?

Is it sufficient to have the explanations and describe what to tune and leave it to the reader to work out how to set tunable foo on their operating system?

Ben

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sat, 5 Mar 2016 05:27:34 -0800

Hi Willy,

>     This is why servers recycle connections, and only in this case.

Hmm, If there are multiple clients behind a NAT, the server cannot know if
it´s coming from the same one or difference one.
Also each client uses its own ISN (also timestamp). So, in this case, it is
possible that the server will see a SYN from different client with an ISN
smaller than the previous window or staled timestamp.
In this case, I guess server will send back old FIN ACK or reset, which
causes the termination of the connection anyway.
If clients go into TIME_STATE, I think we can avoid this.

Regards,
--
Yoshi

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 5 Mar 2016 19:03:40 +0100

On Sat, Mar 05, 2016 at 12:54:08PM +0000, Ben Niven-Jenkins wrote:
> Is it sufficient to have the explanations and describe what to tune and leave
> it to the reader to work out how to set tunable foo on their operating
> system?

I don't have a strong opinion on this, and I'm not the author, Daniel
did this work. I think that suggesting examples of what settings are
available in certain operating systems to achieve some of the explained
concepts is useful, especially to counter the mis-information or bad
advices found on many blogs. But sending that as strict rules is not
a good thing considering that RFCs are supposed to serve as a reference
for a long time while operating systems change.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Sat, 5 Mar 2016 19:07:44 +0100

> smaller than the previous window or staled timestamp.

Yes except that as long as the load is not too high, the firewall will
pick a different port. And if the load is high enough to make the firewall
reuse existing ports, then the TW issue on the client side (here the
firewall being the client) persists.

Yes.

> If clients go into TIME_STATE, I think we can avoid this.

That's why I said in this thread that tuning is a matter of tradeoffs
depending on the side you're looking at. Server people often want to
move the burden to the client to have an infinite scale, client people
prefer to move the burden to the server to enable even smaller clients
to work, and intermediaries people have to compose with the best of both
worlds since they're both clients and servers, and see aggregated traffic
that often reaches protocol limits much earlier.

Regards,
Willy

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sat, 5 Mar 2016 12:10:33 -0800

Hi Willy,

On Sat, Mar 5, 2016 at 10:07 AM, Willy Tarreau <[email-masked]> wrote:

> > it愀 coming from the same one or difference one.

Got it. Thanks for the clarification and that's basically what I felt when
I read the draft.
I would like to support the adoption of the draft, but at the same time, I
might want to see pros and cons for the techniques in the draft.
Then, I think it can be much more beneficial than some random info on
google search results.

Thanks,
--
Yoshi