Subject: Re: [tcpm] TCP Tuning for HTTP - update

From: Joe Touch <[email-masked]>
Date: Tue, 16 Aug 2016 21:21:06 -0700

Hi, Mark, et al.,

I posted a review of this document to both to TCPM and HTTP WGs.

This update fails to address the issues I raised - notably that many of
the issues therein are known *and published*.

So first, can we discuss the issue of PLAGIARISM?

Not only of two of my works, but of many others that pointed out most of
the information summarized in this doc.

Second, the step of "adoption" needs to wait until there's something new
here that wasn't known 20 years ago and the issue of plagiarism is
addressed.

Joe

On 8/16/2016 7:02 PM, Mark Nottingham wrote:
> Hi TCPM,
>
> Just a quick note; Daniel and Tim have made an update to the TCP Tuning for HTTP draft:
>   https://tools.ietf.org/html/draft-stenberg-httpbis-tcp
>
> We've had a Call for Adoption open for this draft for a while, and will likely adopt it soon. However, we'd like to get feedback from this community first -- both about the latest version of the input document, and to see if there's interest in helping out.
>
> You can give feedback on the HTTP WG mailing list <https://lists.w3.org/Archives/Public/ietf-http-wg/>, or  by responding to this e-mail (Please leave the CC line; Patrick and I will try to summarise the feedback to the WG).
>
> Cheers,
>
> --
> Mark Nottingham   https://www.mnot.net/
>
>
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

Subject: Fwd: Re: [tcpm] FW: Call for Adoption: TCP Tuning for HTTP
From: Joe Touch <[email-masked]>
To: [email-masked]
Cc: [email-masked]
Date: Wed, 2 Mar 2016 14:34:38 -0800

Hi, all,

This doc was noted on the TCPM list.

See my observations below.

Joe

-------- Forwarded Message --------
Subject: Re: [tcpm] FW: Call for Adoption: TCP Tuning for HTTP
Date: Wed, 2 Mar 2016 12:25:07 -0800
From: Joe Touch <[email-masked]>
To: Scharf, Michael (Nokia - DE) <[email-masked]>,
[email-masked] Extensions <[email-masked]>
CC: [email-masked]

On 3/2/2016 1:39 AM, Scharf, Michael (Nokia - DE) wrote:
> I assume this could be of interest to the TCPM community.

I have doubts:

- it reads like a Linux manual page

	All Linux-specific references and commands would need to be
	moved to an appendix to be useful as an RFC.

- this repeats (sometimes correctly, sometimes in error) existing advice

J. Heidemann, K. Obraczka, J. Touch, “Modeling the Performance of HTTP
Over Several Transport Protocols,” IEEE/ACM Transactions on Networking,
V5, N5, Oct. 1997, pp.616-630.

T. Faber, J. Touch, and W. Yue, “The TIME-WAIT state in TCP and Its
Effect on Busy Servers,” in Proc. IEEE Infocom, 1999, pp. 1573-1583.

- it has significant errors

	TIME-WAIT issues apply to servers, not clients.

	Nagle has been known to perform poorly for multibyte
	interactive traffic for a very long time, including not
	only web traffic but also multi-byte character or keyboard
	signals.

	Disabling slow-start after idle is safe only with pacing.
	Without pacing, the resulting traffic can generate a burst
	that was never experienced and result in both poor performance
	for the current connection and potential impact to competing
	traffic.

(those are just a few)

Overall, I think a man page might be useful, but this summary isn't
useful for the IETF.

Joe

> -----Original Message-----
> From: Scharf, Michael (Nokia - DE) 
> Sent: Wednesday, March 02, 2016 10:37 AM
> To: 'Mark Nottingham'; HTTP WG
> Cc: [email-masked]; Daniel Stenberg
> Subject: RE: Call for Adoption: TCP Tuning for HTTP
> 
> The document refers to several TCPM RFCs with experimental status, e.g., in Section 3. That may have to be taken into account when heading towards BCP status.
> 
> Michael
> (TCPM co-chair)
> 
> 
> -----Original Message-----
> From: Mark Nottingham [mailto:[email-masked]] 
> Sent: Wednesday, March 02, 2016 6:47 AM
> To: HTTP WG
> Cc: [email-masked]; Daniel Stenberg
> Subject: Call for Adoption: TCP Tuning for HTTP
> 
> [ copying Alison as our Transport Tech Advisor ]
> 
> Daniel has kindly started a document about how HTTP uses TCP, both for /1 and /2:
>   <https://tools.ietf.org/html/draft-stenberg-httpbis-tcp>
> 
> We haven't explicitly discussed this at a meeting, but I have heard interest in this topic from a variety of folks.
> 
> What do people think about adopting this with a target of Best Current Practice?
> 
> Please comment on-list.
> 
> Regards,
> 
> --
> Mark Nottingham   https://www.mnot.net/
> 
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
> 

Subject: Re: [tcpm] FW: Call for Adoption: TCP Tuning for HTTP
From: Joe Touch <[email-masked]>
To: "Scharf, Michael (Nokia - DE)" <[email-masked]>, "[email-masked] Extensions" <[email-masked]>
Cc: [email-masked]
Date: Wed, 2 Mar 2016 12:25:07 -0800

On 3/2/2016 1:39 AM, Scharf, Michael (Nokia - DE) wrote:
> I assume this could be of interest to the TCPM community.

I have doubts:

- it reads like a Linux manual page

	All Linux-specific references and commands would need to be
	moved to an appendix to be useful as an RFC.

- this repeats (sometimes correctly, sometimes in error) existing advice

J. Heidemann, K. Obraczka, J. Touch, “Modeling the Performance of HTTP
Over Several Transport Protocols,” IEEE/ACM Transactions on Networking,
V5, N5, Oct. 1997, pp.616-630.

T. Faber, J. Touch, and W. Yue, “The TIME-WAIT state in TCP and Its
Effect on Busy Servers,” in Proc. IEEE Infocom, 1999, pp. 1573-1583.

- it has significant errors

	TIME-WAIT issues apply to servers, not clients.

	Nagle has been known to perform poorly for multibyte
	interactive traffic for a very long time, including not
	only web traffic but also multi-byte character or keyboard
	signals.

	Disabling slow-start after idle is safe only with pacing.
	Without pacing, the resulting traffic can generate a burst
	that was never experienced and result in both poor performance
	for the current connection and potential impact to competing
	traffic.

(those are just a few)

Overall, I think a man page might be useful, but this summary isn't
useful for the IETF.

Joe

> -----Original Message-----
> From: Scharf, Michael (Nokia - DE) 
> Sent: Wednesday, March 02, 2016 10:37 AM
> To: 'Mark Nottingham'; HTTP WG
> Cc: [email-masked]; Daniel Stenberg
> Subject: RE: Call for Adoption: TCP Tuning for HTTP
> 
> The document refers to several TCPM RFCs with experimental status, e.g., in Section 3. That may have to be taken into account when heading towards BCP status.
> 
> Michael
> (TCPM co-chair)
> 
> 
> -----Original Message-----
> From: Mark Nottingham [mailto:[email-masked]] 
> Sent: Wednesday, March 02, 2016 6:47 AM
> To: HTTP WG
> Cc: [email-masked]; Daniel Stenberg
> Subject: Call for Adoption: TCP Tuning for HTTP
> 
> [ copying Alison as our Transport Tech Advisor ]
> 
> Daniel has kindly started a document about how HTTP uses TCP, both for /1 and /2:
>   <https://tools.ietf.org/html/draft-stenberg-httpbis-tcp>
> 
> We haven't explicitly discussed this at a meeting, but I have heard interest in this topic from a variety of folks.
> 
> What do people think about adopting this with a target of Best Current Practice?
> 
> Please comment on-list.
> 
> Regards,
> 
> --
> Mark Nottingham   https://www.mnot.net/
> 
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 17 Aug 2016 14:43:15 +1000

Joe,

> On 17 Aug 2016, at 2:21 PM, Joe Touch <[email-masked]> wrote:
> 
> Hi, Mark, et al.,
> 

I'm assuming you're talking about <https://lists.w3.org/Archives/Public/ietf-http-wg/2016JanMar/0314.html>.

> So first, can we discuss the issue of PLAGIARISM?

Sure, let's discuss it. That's a very serious accusation. Are you saying that your material has been intentionally used without proper acknowledgement?

Personally, I doubt that. What may have happened is that the text brushes up against things that you've written about in the past, and you feel that you're not adequately acknowledged. 

If that's the case, I'd observe that the IETF isn't an academic publisher, and acknowledging all prior work in an area is neither practical, nor required, nor current practice.

On the other hand, if it turns out that directing readers toward other documents (including yours) adds value, a reference might make sense.

> here that wasn't known 20 years ago and the issue of plagiarism is

Other people in the HTTP *and* TCP communities have commented that such a document would be very useful, whether or not it's something "new that wasn't known 20 years ago". 

Cheers,

> <Attached Message.eml><Attached Message.eml>

---

From: Joe Touch <[email-masked]>
Date: Tue, 16 Aug 2016 22:23:01 -0700

Hi, Mark,

> I'm assuming you're talking about <https://lists.w3.org/Archives/Public/ietf-http-wg/2016JanMar/0314.html>.

Yes.
> Sure, let's discuss it. That's a very serious accusation. Are you saying that your material has been intentionally used without proper acknowledgement?

Yes.

> Personally, I doubt that. What may have happened is that the text brushes up against things that you've written about in the past, and you feel that you're not adequately acknowledged. 
Plagiarism requires only that the material was published elsewhere
before. Intent has no bearing.

In addition, I informed the author - and both lists - about this over 5
months ago. You might claim that the first two versions were issued out
of ignorance, but you cannot claim that of the update.

> If that's the case, I'd observe that the IETF isn't an academic publisher, and acknowledging all prior work in an area is neither practical, nor required, nor current practice.
Plagiarism isn't an issue limited to academic environments. Publication
of a document on the web is still publication.

> On the other hand, if it turns out that directing readers toward other documents (including yours) adds value, a reference might make sense.

Those docs explain the issues more correctly and in more detail. That
should add enough value.

The real question is whether this draft adds value to those - which are
*already published*.

> Other people in the HTTP *and* TCP communities have commented that such a document would be very useful, whether or not it's something "new that wasn't known 20 years ago". 

We don't need to issue new documents for people who don't read old ones.

Joe

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 17 Aug 2016 16:42:53 +1000

Joe,

I've pinged some ADs to weigh in on this.

In the meantime -

> On 17 Aug 2016, at 3:23 PM, Joe Touch <[email-masked]> wrote:
> 

So to paraphrase, you're accusing Daniel of failure to cite a relevant source out of ignorance in the first case... 

> of ignorance, but you cannot claim that of the update.

... and then continuing to do so after being notified.

Correct?

Looking over <http://www.isi.edu/touch/pubs/ton97.pdf> and <http://www.isi.edu/touch/pubs/infocomm99/infocomm99-web/>, I'm having trouble identifying passages that are plagiarised; can you show us?

In particular, the latter paper proposes TIME-WAIT negotiation using a TCP option, sending a RST from clients, and a HTTP extension, whereas Daniel's draft only says this about TIME_WAIT:

"""
2.5.  Lower the TCP FIN timeout

   High connection completion rates will consume ephemeral ports
   quickly.  Lower the time during which connections are in FIN-WAIT-2/
   TIME_WAIT states so that they can be purged faster and thus maintain
   a maximal number of available sockets.  The primitives for the
   assignment of these values were described in [RFC0793], however
   significantly lower values are commonly used.

2.6.  Reuse sockets in TIME_WAIT state

   When running backend servers on a managed, low latency network you
   might allow the reuse of sockets in TIME_WAIT state for new
   connections when a protocol complete termination has occurred.  There
   is no RFC that covers this behaviour.
"""

So, where is the plagiarism?

Also, how do you reconcile that with the statement in your original e-mail that the draft repeats your material "sometimes correctly, sometimes in error"? Have you considered that what you consider to be an error in his plagiarism might in fact be a different idea (no matter whose is ultimately correct)?

> of a document on the web is still publication.

Sure. It also isn't a legal issue in this form (unless you're asserting copyright?). Effectively, it's a cultural norm. Again, I will point out that in the culture of the IETF, we historically have not cited the complete provenance of every idea, both because it's impractical and because it doesn't benefit the reader. 

As far as I know, the IETF does not have a stated position about what you regard as PLAGIARISM. Hopefully we can get some clarity about that from the ADs, as well as some definitive evidence of what you're asserting.

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 17 Aug 2016 08:45:45 +0200

Hi Joe,

On Tue, Aug 16, 2016 at 10:23:01PM -0700, Joe Touch wrote:
> > Other people in the HTTP *and* TCP communities have commented that such a
> > document would be very useful, whether or not it's something "new that
> > wasn't known 20 years ago". 
> 

I've just checked the two documents you referenced. They seem to be very
well detailed but they are *scientific* research. What Daniel created are
configuration advises for people who need to configure their servers. Yes
as you mentionned they look like man pages precisely because the purpose
is to ensure they're easily understood by people who are just seeking some
help to improve their configuration. You cannot expect a server admin to
read scientific papers explaining some TCP models with some formulas to
know what to do on their servers.

Also, I don't know if there have been any update, but these documents use
SunOS 4.1.3 running on a sparc 20 as a reference. While I used to love
working on such systems 20 years ago, they predate the web era and systems
have evolved a lot since to deal with high traffic. I remember by then I
used to be impressed when seeing a server sustain 50 connections per second.
Now when someone tells me he's limited to 100000 connections per second on
a single server, I ask how they bound their interrupts to see if there's
some headroom to go further.

So you need to expect that only researchers and maybe TCP stack developers
will find your work useful these days, server admins can hardly use this
anymore. However it is very possible that some TCP stacks have taken benefit
of your work to reach the level of performance they achieve right now, I
don't know. Thus I think that Daniel's work completes quite well what you've
done in that it directly addresses people's concerns without requiring the
scientific background.

Regards,
Willy

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 08:08:05 -0700

> So to paraphrase, you're accusing Daniel of failure to cite a relevant source out of ignorance in the first case... 
I'm being generous in assuming ignorance, but yes.

Yes.

> Looking over <http://www.isi.edu/touch/pubs/ton97.pdf> and <http://www.isi.edu/touch/pubs/infocomm99/infocomm99-web/>, I'm having trouble identifying passages that are plagiarised; can you show us?
It's the content - turning off Nagle, slow-start restart after idle, etc.

> In particular, the latter paper proposes TIME-WAIT negotiation using a TCP option, sending a RST from clients, and a HTTP extension,

It also outlines the whole problem for the first time and explains the
choice and impact of lowering TW timeout.

>  ...
That might be a reasonable conclusion if there were discussion
explaining why the "different idea" was correct in comparison.

But that's missing. Saying to lower the timer for TW without discussing
the potential impact isn't a new idea. It's an incomplete discussion of
an old idea already discussed in detail elsewhere.

> Sure. It also isn't a legal issue in this form (unless you're asserting copyright?). Effectively, it's a cultural norm. Again, I will point out that in the culture of the IETF, we historically have…
Although that's true in the smallest cases, the IETF does have two
concepts that support this norm: an author list and a set of references.

Can you explain how it helps the reader to not cite two documents that
are both squarely in the same area as this doc (interaction between HTTP
and TCP and the impact of running many small connections closed at the
client as for HTTP)?

> As far as I know, the IETF does not have a stated position about what you regard as PLAGIARISM. Hopefully we can get some clarity about that from the ADs, as well as some definitive evidence of wha…

You can if you want, but my primary point here is to have this work
corrected - and to stop the myth that "it doesn't matter" whether
*reasonable* citations are included.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 08:14:08 -0700

> know what to do on their servers.
Actually, my docs are the tip of the iceberg, addressing the reasons why
some of these configuration recommendations make sense.

There are many other sites - and books - that already indicate how to
configure systems efficiently.

So if your argument is that a man page summary is needed, sure - but
again, is a new one needed? And why is this then needed as an RFC?

> have evolved a lot since to deal with high traffic. ...
Yes, and discussing those issues would be useful - but not in this
document either.

> done in that it directly addresses people's concerns without requiring the
Let me see if I get your complete argument:

    - the appropriate refs are 20 years old
    - server admins need a doc

What exactly do server admins need regarding Nagle (which is configured
inside the app already), socket sizing (configured inside the app), etc?

I.e., at the most this is a man page (specific to an OS). At the least,
this isn't useful at all.

Joe

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 17 Aug 2016 16:28:45 +0100

Joe,

>>> On 17 Aug 2016, at 3:23 PM, Joe Touch <[email-masked]> wrote:
  [snip]
> and TCP and the impact of running many small connections closed at the
Instead of starting your discussion with words like "plagiarism", you 
could have just asked for information to be clarified and a 
citation/acknowledgement added? With your current introduction you 
pissed off lots of people.
> *reasonable* citations are included.
Noted.

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 08:53:54 -0700

> On Aug 17, 2016, at 8:28 AM, Alexey Melnikov <[email-masked]> wrote:
> 
> Joe,
> 
> Instead of starting your discussion with words like "plagiarism", you could have just asked for information to be clarified and a citation/acknowledgement added?
...

I did that 5 months ago, as noted in the attachments I included yesterday.

Joe

---

From: Eliot Lear <[email-masked]>
Date: Wed, 17 Aug 2016 18:26:58 +0200

Perhaps we can agree that the reasonable course of action here is for
Joe to (re)-recommend a compact set of citations to the authors, perhaps
even in some easily consumable form to them (kramdown-2629 or XML)?

Eliot

On 8/17/16 5:28 PM, Alexey Melnikov wrote:
>>>>> If that's the case, I'd observe that the IETF isn't an academic
>>>>> publisher, and acknowledging all prior work in an area is neither
>>>>> practical, nor required, nor current practice.
>>> Sure. It also isn't a legal issue in this form (unless you're
>>> asserting copyright?). Effectively, it's a cultural norm. Again, I
>>> will point out that in the culture of the IETF, we historically have
>>> not cited the complete provenance of every idea, both because it's
>>> impractical and because it doesn't benefit the reader.
>>> As far as I know, the IETF does not have a stated position about
>>> what you regard as PLAGIARISM. Hopefully we can get some clarity
>>> about that from the ADs, as well as some definitive evidence of what
>>> you're asserting.

---

From: Tim Wicinski <[email-masked]>
Date: Wed, 17 Aug 2016 13:00:12 -0400

That sounds like a fine idea.  I'll be glad to go through those.

tim

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 10:16:13 -0700

All,

It sounds like the job of an author or a search engine (or both).

I've given you a place to start.

Joe

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 17 Aug 2016 20:08:02 +0200

> again, is a new one needed? And why is this then needed as an RFC?

The difference is that a man page is OS-specific while an RFC gets a
unique number and serves as a reference. It can be cited in new RFCs
to justify certain choices. It can receive errata. You would probably
say that the current document is very linux-centric for now but I
understood it as a beginning of something more generic where advices
are given based on principles before sysctls and that the resulting
sysctls are just examples of applications of the principles in the
document.

> Yes, and discussing those issues would be useful - but not in this

Why ? Lots of admins don't understand why the time_wait timeout remains
at 240 seconds on Solaris with people saying "if you want to be conservative
don't touch it but if you want to be modern simply shrink it to 30 seconds
or so". People need to understand why advices have changed over 3 decades.

> inside the app already), socket sizing (configured inside the app), etc?

Lots of things : 
  - time_wait tuning (which everybody gives different advices on, I've
    even seen firewall vendors recommend to shrink it to one second because
    it allowed their product to perform better in benchmarks)

  - TCP timestamps: what they provide, what are the risks (some people in
    banking environments refuse to enable them so that they cannot be used
    as an oracle to help in timing attacks).

  - window scaling : how much is needed.

  - socket sizing : contrary to what you write, there's a lot of tuning
    on the web where people set the default buffer sizes to 16MB without
    understanding the impacts when dealing with many sockets

  - SACK : why it's better. DSACK what it adds on top of SACK.

  - ECN : is it needed ? does it really work ? where does it cause issues ?

  - SYN cookies : benefits, risks

  - TCP reuse/recycling : benefits, risks

  - dealing with buffer bloat : tradeoffs between NIC-based acceleration
    and pacing

  - what are orphans and why you should care about them in HTTP close mode

  - TCP fastopen : how does it work, what type of workload is improved,
    what are the risks (ie: do not enable socket creation without cookie
    by default just because you find it reduces your server load)

  - whether to choose a short or a large SYN backlog depending on your
    workload (ie: do you prefer to process everything even if the dequeuing
    is expensive or to drop early in order to recover fast).

... and probably many other that don't immediately come to my mind. None
of these ones was a real issue 20 years ago. All of them became issues for
many web server admins who just copy-paste random settings from various
blogs found on the net who just copy the same stupidities over and over
resulting in the same trouble being caused to each of their reader.

> this isn't useful at all.

As you can see above, nothing I cited was OS-specific but only workload
specific. That's why I think that an informational RFC is much more suited
to this than an OS-specific man page. The OS man page may rely on the RFC
to propose various tuning profiles for different workloads however.

Regards,
Willy

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 11:31:33 -0700

On 8/17/2016 11:08 AM, Willy Tarreau wrote:
> unique number and serves as a reference. 
That means the OS-specific stuff here needs to be removed...

> It can be cited in new RFCs
> to justify certain choices. 
Hmm. Like the refs I gave could be cited in this doc to justify *its*
choices? :-)
...
> or so". People need to understand why advices have changed over 3 decades.

The advice hasn't really changed - the advice was given in the 99 ref,
which includes some cases where it can still be appropriate to decrease
that timer.

Again, though - the 99 ref explains the implications of both decisions -
why it's 240 seconds and what happens when it's lowered.

>     it allowed their product to perform better in benchmarks)
Again, the 99 ref gives that detail.
>     as an oracle to help in timing attacks).
That's already covered in the security considerations of RFC 7323. How
is HTTP different, if at all, from any other app?
>   - window scaling : how much is needed.
Same issue here, same ref - how is HTTP different?

>     understanding the impacts when dealing with many sockets
There's a whole book that encompasses that and some related issues:
http://www.onlamp.com/pub/a/onlamp/2005/11/17/tcp_tuning.html

Some advice is also given in Sec 6.3.3 of this:
J. Sterbenz, J. Touch, /High Speed Networking: A Systematic Approach to
High-Bandwidth Low-Latency Communication/
<http://www.wiley.com/WileyCDA/WileyTitle/productCd-0471330361.html>,
Wiley, May 2001.

>   - SACK : why it's better. DSACK what it adds on top of SACK.
That's in the SACK docs, which aren't cited. Again, how is HTTP
different from any app?
>   - ECN : is it needed ? does it really work ? where does it cause issues ?
That's in the ECN docs, which aren't cited. Again, how is HTTP different
from any app?
>   - SYN cookies : benefits, risks
That's in the RFC 4987, which at least IS cited. Again, how is HTTP
different from any app?

>   - TCP reuse/recycling : benefits, risks
Not sure what you mean here. There are a lot of docs on the issues with
persistent-HTTP vs per-connection HTTP.

>   - dealing with buffer bloat : tradeoffs between NIC-based acceleration
Bufferbloat typically involves large *uplink* transfers and how they
interact with other uplink connections. Neither TCP nor HTTP is involved
in this really.

>   - what are orphans and why you should care about them in HTTP close mode
Orphaned TCP connections or orphaned HTTP processes?

>     by default just because you find it reduces your server load)

Another doc that exists.

> of these ones was a real issue 20 years ago.

See above. Many were known around that time, but weren't documented in
detail (it took a while for a proper ref to SYN cookies, and the book I
wrote with Sterbenz came about because we'd seen wheels being
rediscovered for 15 years).

>  All of them became issues for

This doc is all over the place.

If you want a doc to advise web admins, do so. But most of the items
above aren't under admin control; they're buried in app and OS
implementations, and most have already evolved do to the right thing.

I agree that a summary of a *focused set* of these might be useful *as a
review* (note that review discussions usually include lots of refs). The
key question is "what is the focus":

    - HTTP/TCP interactions
    - server administration advice
    - ?

IMO, RFCs should focus on issues specific to the protocols and their
deployment - not general knowledge that exists in courses and textbooks.

> to propose various tuning profiles for different workloads however.

You have a good point that this is general info, but OS issues are not
in the scope of the IETF and there are courses and books that already
provide good advice on efficiently running web (and other) servers.

Joe

---

From: Willy Tarreau <[email-masked]>
Date: Wed, 17 Aug 2016 23:13:17 +0200

> Hmm. Like the refs I gave could be cited in this doc to justify *its*

I think it would be nice that this is cited, but to be clear on one
point, I've never heard about your papers before you advertised them
here in this thread, and yet I've been dealing with timewait issues
for 15 years like many people facing moderate to large web sites
nowadays. It just happens that you probably identified these issues
very early at low connection rates, but today anyone dealing with
more than 500-1000 connections per second on a server or worse on
a gateway quickly discovers that he has to make a choice.

> which includes some cases where it can still be appropriate to decrease

Most people see it the other way around : they see no valid case to *increase*
it beyond a few seconds, because for them the default value should be extremely
low (ie this firewall vendor several years ago trying to insist on one second).
Yes that's really sad but that's reality. And you can tell them to read 6191
they won't care.

> is HTTP different, if at all, from any other app?

HTTP is special in that it is fairly common to have to deal with tens of
thousands of connections per second between one client and one server when
you are on the server side, because you place a number of gateways (also
called reverse-proxies) which combine all of the possible issues you can
think of at a single place. Timestamps are one way to improve fast connection
recycling between the client and the server without having to cheat on
timewaits. But since they consume 12 bytes per packet, it's often advised
to disable them in benchmarks to get the highest throughput...

> Same issue here, same ref - how is HTTP different?

Same as above.

> http://www.onlamp.com/pub/a/onlamp/2005/11/17/tcp_tuning.html

Looks fine, could be added to the list of references.

> That's in the RFC 4987, which at least IS cited. Again, how is HTTP

So probably you're starting to see the benefit of having a single doc
to concentrate all this. You provided at least 3 different articles
to read and 2 or 3 different RFCs in addition to the original ones,
of course. A hosting provider whose web sites are down due to a lack
of tuning doesn't start to read many very long articles and even less
the most scientific ones, they need to find quick responses that they
can apply immediately (matter of minutes). So they launch google, they
type "web site dead, time-wait overflow" and they get plenty of
responses on stackoverflow and serverfault, many from people having
done the same in the past and repeating the same mistakes over and over.

A document validated by several people and giving links for further
reading can help improve this situation.

> persistent-HTTP vs per-connection HTTP.

Often on a gateway you cannot completely chose. You have a mix of both.

> interact with other uplink connections. Neither TCP nor HTTP is involved

Maybe in the end Daniel is not the only one not to read all articles
published on the web :-)

     http://www.cringely.com/2012/03/25/linux-3-3-finally-a-little-good-news-for-bufferbloat/
     https://www.ietf.org/proceedings/86/slides/slides-86-iccrg-0.pdf
     https://lwn.net/Articles/564978/

In short, by sending 64kB segments to your NIC and counting on it to
cut them into small pieces for you and sending the resulting packets
very close together, you increase the risk of losses on many network
equipments which run with not-that-large buffers. It's easy to observe
even on some 10Gbps switches. When you switch mixes 10G and 40G, it's
horrible, really.

> Orphaned TCP connections or orphaned HTTP processes?

TCP connections. The server sends a response, performs a close() on the
socket, the data remain in the kernel buffers for the time it takes to
deliver these data to the client and to get them ACKed. From this point
the socket is called orphaned on the system because it doesn't belong to
any process anymore. But immediately after this, a process which runs with
a limited connection count can accept a new connection, which will in turn
be fed with large chunks of data and closed. And this runs over and over,
eating a huge amount of socket memory despite a small limit imposed on the
server's connection concurrency. At some point the kernel doesn't have
enough socket buffers anymore (especially after admins have set them to
16MB as instructed on $random tuning guide) and starts to kill some orphaned
connections to get some memory back. The not-so-nice effect that the admin
cannot detect is that the client gets truncated responses. Only a small part
of the tail is missing but in the logs, it's said that everything was sent.
But sent by the process means sent to the system only.

> rediscovered for 15 years).

People rediscover wheels because it's hard to find simple and accurate
information on the net. Basically you have the choice :
  - either uneducated blog posts saying "how I saved my web site using 2
    sysctls"
  - or academic papers which are only understandable by scientific people
    having enough time

At least the first ones have the merit of being easy to test, and since
they appear to work they are viral.

> If you want a doc to advise web admins, do so.

That's *exactly* what Daniel started to do when you told him he shouldn't
do it.

> But most of the items

That's not true anymore. There was a time where Solaris had one of the
most configurable TCP stack, you could do ndd /dev/tcp with a lot of
things. Nowadays all modern operating systems let you fiddle with a ton
of stuff. And even for the missing parts you can very easily find patches
all over the web because opensource has changed this considerably.

> review* (note that review discussions usually include lots of refs).

The purpose as I understood it was precisely to gather knowledge from
people here operating such systems and adding some elements found from
various sources. I also expressed my interest in sharing such experience
which some will find valid and others not, which is perfect because it
means there's a choice to make depending on a use case.

> The key question is "what is the focus":
> 
>     - HTTP/TCP interactions

Those two go hand-in-hand nowadays. You probably know that the difficulty
for HTTP implementors to find a properly tuned TCP stack is what makes them
consider UDP-based alternatives, so that they can totally control the
behaviour from user-space and provide quick updates for their protocol.
Want an example ?

    https://www.chromium.org/quic

If it was possible to always have a perfectly tuned TCP stack I'm pretty
sure we wouldn't even hear about such initiatives. And this is not something
new, I happened to discuss about this subject with some people at IETF83 in
2012 already. By then I thought "they have no idea what they're trying to
reinvent" and now I'm thinking "well, they have their reasons and they
might be right after all given all the resistance they may be facing on
the TCP side to get some default timers changed".

> deployment - not general knowledge that exists in courses and textbooks.

Courses and textbooks are totally outdated when they're not simply wrong.
I've first been tought that it was normal to fork a process after performing
an accept(), resulting in my very first proxy working this way. Then I've
been taught that it was mandatory to send an empty ACK after a SYN-ACK to
validate a connection (which is totally wrong otherwise it would not allow
this ACK to be lost). I've been taught that in order to accept an incoming
connection you had to have your socket in listen mode exclusively. This is
false as well, two clients can connect together during their connect()
phase, it even has some security implications that are often overlooked.
Like it or not, the HTTP protocol has brought TCP to an area it was not
initially intended for and I find it fantastic to see that this protocol
still scales so well. But we need to consider modern usages of this protocol
for the web, and not just academic research and e-mail.

> provide good advice on efficiently running web (and other) servers.

Well, you want to run a quick test ? Google "tcp tuning for http server".
Skip the first two responses which are Daniel's document, take the next one :

    https://gist.github.com/kgriffs/4027835

Read a bit... Not bad, could have been much worse. OK found : 5 seconds
time_wait timeout for conntrack. Dig a little bit more... suggests 2M
time_wait sockets. At 5s per socket, that's an expected 400k connections
per second limit. This with buffers that can be as large as 16MB on both
sides (one could ask why to put 16MB buffers on the Rx path for a web
server). So basically the guy probably thinks that there's a need to
fill up to 6.4 TB/s and despite this the max-orphans was not set according
to the time-wait value, so users of his doc will start to cause some data
truncation before they run into connection failures, thus only their visitors
will know there are problems.

And this is the highest ranked doc on google after Daniel's. And overall
it's really not bad, much better than what we easily find everywhere.

The fact that Daniel's doc found before is a good hope that this sad state
of affairs can reach an end. That's why I think we should encourage him to
continue and give him all the reference he needs to have an undiscutably
good reference doc.

Regards,
Willy

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 15:23:10 -0700

On 8/17/2016 2:13 PM, Willy Tarreau wrote:
> here in this thread, 

A search engine on the terms "TCP HTTP interaction" would have popped
them up rather quickly.

> and yet I've been dealing with timewait issues
> nowadays. 

"timewait issues" and we're the 5th hit in Google.

> It just happens that you probably identified these issues

The issue was very common when the doc was written in 99, when even at
that time there were two issues - running out of the number space and
running out of kernel memory.

The number space issue of running out of ports was the basis of the IETF
port names doc in 2006
(https://tools.ietf.org/html/draft-touch-tcp-portnames-00) that became
the current proposal for a TCP "service number option" in 2013 (which
has been discussed at various IETFs in TCPM since then).

> Yes that's really sad but that's reality. And you can tell them to read 6191

Most people's servers don't need to run fast enough to care (note that
nearly everyone runs some sort of web server on nearly every device,
whether for control or configuration). The only issue are high-volume
servers (the kind sysadmins deal with), and those people tend to already
know what the tradeoffs are and accept the risks.

> think of at a single place.

There are lots of services that have that many transactions - DNS
servers (even local ones), remote databases, etc.

The point is that HTTP doesn't make the problem different, so this isn't
an HTTP issue. It's a high rate server issue.

>  Timestamps are one way to improve fast connection
> to concentrate all this.
The same reason it's useful to have this all in one place is the reason
we already do - there are books and courses on this.

>  You provided at least 3 different articles

These people don't read RFCs to fix problems. They take online courses
or read "how to" books - which do already exist in this space.

> reading can help improve this situation.
Those are the books and courses I'm talking about already.

> Often on a gateway you cannot completely chose. You have a mix of both.

Sure... but again, that's not in this doc either.

> equipments which run with not-that-large buffers.

Bufferbloat describes what happens when the buffers are too large, not
too small.

The problem you're describing is the interaction between burstiness and
tail-drop, which is addressed by ECN.

>  It's easy to observe
RFCs aren't typically used as context in switch design...

> But sent by the process means sent to the system only.

That's an implementation issue of the OS (or a bug, depending on whether
you consider TCP a reliable transport or not, IMO).

> information on the net.
Nobody looks to RFCs to solve that problem...

>  Basically you have the choice :

... that's what net FAQs are for, as well as courses and books.

> That's *exactly* what Daniel started to do when you told him he shouldn't

I didn't say a doc to advise web admins wasn't useful. I said it wasn't
an RFC.

It's a web FAQ, a book, etc.

> all over the web because opensource has changed this considerably.
There are no kernel configs to tell apps to open more than one
connection at a time. You don't need a kernel config to tell Firefox to
disable Nagle, use a reasonable socket size, etc - yes, there are OS
defaults, but most web servers and clients build in the correct
overrides already.

> consider UDP-based alternatives, 
They want something different for a variety of reasons - the same kind
of airtight logic by which TBL developed HTTP instead of using FTP (he
said that you'd only typically need one file from a location, so why
open 2 connections? now we're stuck trying to mux control and data
rather than having a proper solution that already existed at the time -
it took nearly a decade for HTTP servers to catch up to the performance
of FTP).

> so that they can totally control the

There are many thousands of monkeys typing everywhere - look at the
Linux source if you want even better examples.

> Courses and textbooks are totally outdated when they're not simply wrong.
I'm now confused.

You don't want sysadmins to read books or take courses because they're
outdated and thus wrong, but you want to issue an immutable RFC (which
will likely be outdated by the time it's issued)?
> phase, it even has some security implications that are often overlooked.
I'm the first to admit there are bad courses, certainly.

> initially intended for 
HTTP makes mistakes that people blame on TCP (like HOL blocking), and
TCP is based on assumptions that are no longer true (not just for HTTP,
but for many other app protocols, e.g., the issue of burst after idle is
based on the outdated assumption that most transfers are roughly symmetric).

> and I find it fantastic to see that this protocol
> still scales so well. 
> But we need to consider modern usages of this protocol
You might consider that TCPM and TSVWG don't exist for just "academic
research and e-mail". What do you think we've been doing for the past 40
years?

>     https://gist.github.com/kgriffs/4027835

Like I said about Linux... ;-)

> continue and give him all the reference he needs to have an undiscutably

You might try some other terms in your searches. Like just "TCP tuning",
or "TCP HTTP interactions".

Joe

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 17 Aug 2016 22:51:18 +0000

------ Original Message ------
From: "Joe Touch" <[email-masked]>

>it took nearly a decade for HTTP servers to catch up to the performance
Whilst I've been finding this discussion very informative and 
interesting, I have to raise an objection on this point.

FTP was never going to be suitable for the web, and a very simple RTT 
analysis shows that.

Apart from initial 3 way TCP handshake and close, which is the same for 
both, with http you have a request and a response, whereas FTP requires 
you to wait for the server welcome, log in, negotiate another port, set 
up a data connection in addition to retrieving the file

So it's at minimum 5 round trips more.

Then try adding all the firewall issues due to transmitting data 
connection endpoint information over the control connection and it's no 
surprise FTP is not favoured for downloads.
So FTP was never going to be a "proper solution" for the web without a 
complete re-architecture.

Adrien

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 16:01:57 -0700

This is a bit of a side track, but...

On 8/17/2016 3:51 PM, Adrien de Croy wrote:
> Apart from initial 3 way TCP handshake and close, which is the same
> for both, with http you have a request and a response, whereas FTP
> requires you to wait for the server welcome, log in, negotiate another
> port, set up a data connection in addition to retrieving the file

That's only the first time you go somewhere new. You don't need to close
both ports so quickly; the control channel can stay open and you thus
avoid HOL blocking between data and control (and thus the need to
chunk-and-mux within persistent HTTP), which increases other delays for
HTTP.

Neither protocol matches exactly what is really needed for a true
transaction-oriented protocol.

> connection endpoint information over the control connection and it's
> no surprise FTP is not favoured for downloads.

FTP had a passive mode even back then that avoids this issue. It also
had suspend/resume, compression, and format conversion.

Joe

---

From: "Adrien de Croy" <[email-masked]>
Date: Wed, 17 Aug 2016 23:17:13 +0000

------ Original Message ------
From: "Joe Touch" <[email-masked]>
To: "Adrien de Croy" <[email-masked]>; "Willy Tarreau" <[email-masked]>
Cc: "Mark Nottingham" <[email-masked]>; "[email-masked]" <[email-masked]>; 
"HTTP Working Group" <[email-masked]>; "Patrick McManus" 
<[email-masked]>; "Daniel Stenberg" <[email-masked]>
Sent: 18/08/2016 11:01:57 AM
Subject: Re: [tcpm] TCP Tuning for HTTP - update

>chunk-and-mux within persistent HTTP), which increases other delays for
You still need to send another PORT/PASV and wait for the response 
before making another TCP connection for data, since you can't re-use 
this one.

So subsequent requests to the same server will be quicker, still at 
least 3 round-trips more than a subsequent http request on a persistent 
connection.

>transaction-oriented protocol.
We are probably in agreement here.

>had suspend/resume, compression, and format conversion.

Those all take more round trips to negotiate.  As for format 
conversion..... bad idea, never should have been the server's job.

We've seen a lot of server-side firewall problems with PASV as well.

Adrien

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 16:20:54 -0700

> conversion..... bad idea, never should have been the server's job.
Sure, but they can be negotiated once for a connection.

FTP also supports wildcard retrieval (e.g., "get *"), which can help
with anticipation.

My point is that FTP could have been much better place to start towards
a transaction protocol.

Joe

---

From: Matthew Kerwin <[email-masked]>
Date: Thu, 18 Aug 2016 09:35:13 +1000

Hi folks, I'm stepping in here on just a couple of points. I'll snip the
bits I can't or won't talk to.

On 18 August 2016 at 08:23, Joe Touch <[email-masked]> wrote:

> "
> ​​
> timewait issues" and we're the 5th hit in Google.
>
> ​
Google, unless it's changed again recently, tailors search results for the
user. My first page of hits for that query are all serverfault.com,
superuser.com, serverframework.com, stackoverflow.com, etc.

Guess where I get most of my advice.
​

> know what the tradeoffs are and accept the risks.
​Your sysadmins are not like my sysadmins. But these are generalisations
and anecdotes.​

> an HTTP issue. It's a high rate server issue.
​What makes HTTP different is that I expect most high-rate applications
would exist in a context where the people running the servers and
applications have some amount of specialist experience and knowledge, or at
least an expectation that they're doing something that requires such
knowledge, in high-rate throughput.

HTTP is ubiquitous, and resides all along the traffic scale, from my
website (~no bits per second) to google.com (all the bits); and the slide
-- or sudden jump -- up that scale doesn't always correspond with acquiring
expertise in TCP stack tuning.​

> we already do - there are books and courses on this.
​My users aren't getting my content right at the time my site is booming.
I've just been slashdotted/hackernewsed/whatever. Do I enroll in a course?
Buy a textbook and swot up (which I haven't done since I finished my IT
degree 15+ years ago)? Or do I hit up serverfault.com and bash the keyboard
until the fires go out?

Summary information is really important. Having it published by the same
people who published the protocol spec adds some serious cred, at least in
my eyes.​

> or read "how to" books - which do already exist in this space.
​Which people? I tend to google the error and see if there's some sort of
consensus on stackoverflow. And then, as often as not, I have to advise my
sysadmins of a course of action because they know as much as me, or they
don't want to deal with my situation, or some other reason.​

> ​​
​Here's the crux of the issue. What do you think an RFC is, that we
(apparently) don't? Why is an informational RFC not allowed to present the
same sort of information as an FAQ? (Isn't that what a BCP is?)

Personally I'd be happy if it was written up exactly like an FAQ, and
published as an informational RFC; because that tells me that this FAQ was
published by the IETF. It's not some random dude's unreliable blog full of
cargo-cult advice and anecdotes; it met the consensus of the organisation
that published the HTTP protocol spec. It's legitimate and reliable. And
one day, when it's out of date, it'll be updated or obsoleted by the new
consensus wisdom of the IETF.

That, as far as I know, doesn't defy the official definition of what an RFC
is*, nor does it devalue any existing (or future) RFCs.

And Google indexes RFCs. If this ends up being a really useful document
(which I imagine it would), with lots of inbound links and references,
people won't need to "look to RFCs to solve [their] problem", they'll do
what they already do -- look to Google, and Google will point them to this
RFC.

* heh​

> research and e-mail". What do you think we've been doing for the past 40
​Dunno, I was only born 35-odd years ago. ;) Shall we talk about generation
gaps? Greybeards vs millennials? (Of which I'm neither, BTW. Not yet, at
least.)

Where we come from, how we find information to solve our problems, the way
we view the IETF and its RFCs are all different. If this argument is just
that this stuff doesn't belong in an RFC, that's a cultural issue and not
one I think we can resolve in this one technical working group.

Cheers
-- 
  Matthew Kerwin
  http://matthew.kerwin.net.au/

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 17:21:21 -0700

Popping up to the key point...

On 8/17/2016 4:35 PM, Matthew Kerwin wrote:
> Hi folks, I'm stepping in here on just a couple of points. I'll snip
> the bits I can't or won't talk to.
...
> Where we come from, how we find information to solve our problems, the
> way we view the IETF and its RFCs are all different. If this argument
> is just that this stuff doesn't belong in an RFC, that's a cultural
> issue and not one I think we can resolve in this one technical working
> group.

IMO, RFCs deal with *protocols* and protocol deployment issues.

Here that would mean for HTTP in specific
    - turn Nagle off
    - watch out for ACK compression effects (turn it off in favor of ABC
if you can)
    - deal with slow-start restart (there are a variety of approaches here)

You also want to be generally efficient with TCP, which isn't related to
HTTP but still good advice for all apps:
    - be generally efficient and robust (use SACK, ECN, good window
scaling, etc.)
    - watch out for resource overloads (use SYN cookies/fast open,
time-wait buildup, etc.)

That advice is useful for all TCP for transactions, too.

Here's what it definitely does not mean:
    - setting socket buffer sizes (that's an OS-ism, and may be useful
for app designers to know, but isn''t directly part of TCP or HTTP)
    - running an efficient server for small connections (that's a pure
implementation performance issue, and depends on OS support for things
like threads, process/thread pools, etc.)
 
I.e., OS-specific issues in running servers are not in scope IMO.

Keep in mind who reads RFCs - mostly protocol people. Managers take
training courses to get certification, and that's where they learn their
operational issues.

Joe

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 18 Aug 2016 10:51:39 +1000

The AD has spoken, and I think we're rapidly approaching the point of being off-topic here. I'll respond to your question below, and of course if you change your mind and do want to follow up with a helpful list of suggested references, you're most welcome.

> On 18 Aug 2016, at 1:08 AM, Joe Touch <[email-masked]> wrote:
> 

Oh, it can be very helpful; I don't think anyone disputes that appropriate references are helpful. 

Whether or not specific documents -- such as yours -- should be referenced is a matter for editorial discretion and eventually WG consensus.  

I note that no one else has supported your claims, but others have said (e.g., [1]) that your work probably isn't appropriate for the intended audience (HTTP implementers and administrators).

Directing Daniel to add citations to mollify you at this point would be entirely inappropriate, because it would give an incentive to others who want their works cited without full review to make similar claims -- whether that be due to a genuine desire to help, a desire to increase their h-index, or to bolster an IPR claim against a technology.

Or, of course, Daniel could use his editorial discretion to decide to cite your work before the WG adopts it; however, I'd very much understand if he didn't want to do so at this point.

Regards,

1. http://www.w3.org/mid/[email-masked]

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 17:59:30 -0700

Mark,

My point is simple: this work is useless if it fails to cite the
relevant prior work that explains the rationale behind its recommendations.

Mine isn't the only work not being cited here - there's all the work in
TCP tuning from PSC, many web workshops from the 90s, and numerous RFCs
and books. I've pointed out just a few.

I don't care to do the author's homework, but as AD I'm disappointed you
don't seem to care about that either.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 18:09:45 -0700

On 8/17/2016 5:59 PM, Joe Touch wrote:
> Mark,
>
> ...
> I don't care to do the author's homework, but as AD ...
WG chair.

Sorry for the continued confusion on that.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 18:15:56 -0700

One other point...

On 8/17/2016 5:51 PM, Mark Nottingham wrote:
> The AD has spoken,
The only post from the AD so far was to ask why I didn't just point out
the relevant citations before.

Which, as I already replied, I had - many months ago.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 19:29:58 -0700

On the point of references:

> I note that no one else has supported your claims, but others have said (e.g., [1]) that your work probably isn't appropriate for the intended audience (HTTP implementers and administrators).
Others have claimed that the papers I cited are not a sufficient
replacement for this document, but the only claim made to not cite them
at all is based on "RFCs don't need to cite sources or detail" - which
nearly ever RFC published disproves. Citations serve many purposes, esp.
informational ones - to provide detailed background information and
context among them.

Note that if this doc is limited to *protocol* recommendations (i.e.,
the scope of the IETF), I agree that it remains useful. My primary
objection to the document itself is it includes far too many OS
operational considerations that are out of scope for the IETF.

> Directing Daniel to add citations to mollify you at this point would be entirely inappropriate, because it would give an incentive to others who want their works cited without full review to make s…
I agree that directing anyone to cite someone for mollification would be
inappropriate; I never asked for that.

There are plenty of ways that citation considerations avoid the issues
you raise:
    - does the document provide informational background?
        are you claiming that the docs I cite do not?
    - is it the original or most complete reference (it can be useful to
cite surveys rather than original literature where the surveys cite the
originals)
        have you found an earlier or more complete reference?

These considerations prevent people from merely claiming that their work
should be gratuitously cited.

I made my claim in the original post back in March - the bulk of the
actual TCP interactions are discussed in more detail with rationale in
the one document, and the same is true for the TIMEWAIT issue for the
second document.

The only argument I've seen put forth is that "RFCs don't need to cite
things", which is false by nearly every RFC published.

Joe

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 18 Aug 2016 12:43:21 +1000

> On 18 Aug 2016, at 12:29 PM, Joe Touch <[email-masked]> wrote:

> should be gratuitously cited.

Sure, and if the WG adopts the document, we can have those discussions. Adopting the document does not mean that we're going to rubber-stamp its content; it's just being taken as a starting point.

In the meantime, let's discuss the scope of work being proposed and whether it's appropriate, what modifications might be needed, etc., so as to inform the decision to adopt.

I'd especially like to hear from other people in the TCP community; we've heard from Joe and Michael; do others share their opinions, or have another view?

> things", which is false by nearly every RFC published.

Now you're misrepresenting what others have said.  Please stop.

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Aug 2016 20:19:45 -0700

> Now you're misrepresenting what others have said.  Please stop.
Here's a list of everyone I can find from the list who has commented on
the issue of citation.

I agree that the quote above misrepresents your position, and I
apologize for that.

I leave the sum of everyone's words - hopefully quoted with enough
context to remain accurate - as context for how to proceed.

Joe

--------

You:

Again, I will point out that in the culture of the IETF, we historically have not cited the complete provenance of every idea, both because it's impractical and because it doesn't benefit the reader.

{and in a later post:)

{NB: you cite Willy, below)

---

Eliot Lear:

--

Tim Wicinsk (in ref to Eliot's post):

--
Alexy Melnikov:

Instead of starting your discussion with words like "plagiarism", you 
could have just asked for information to be clarified and a 
citation/acknowledgement added?

--

Willy Tarreau:

I've just checked the two documents you referenced. They seem to be very
well detailed but they are *scientific* research. ...
...

... Thus I think that Daniel's work completes quite well what you've
done in that it directly addresses people's concerns without requiring the
scientific background.

{and later, addressing the references I asked to be cited)

I think it would be nice that this is cited, ...

...
A document validated by several people and giving links for further
reading can help improve this situation.
...
That's why I think we should encourage him to
continue and give him all the reference he needs to have an undisputably
good reference doc.

--

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 18 Aug 2016 07:38:37 +0200

> Here that would mean for HTTP in specific

All applications have been doing this for almost two decades.

>     - watch out for ACK compression effects (turn it off in favor of ABC

It does not happen that much with HTTP. Many connections on the server side
see only one, sometimes two requests, and most responses are small (about
20kB on average, with favicon fitting in a single segment). Note, I'm talking
about observations on average web sites.

>     - deal with slow-start restart (there are a variety of approaches here)

Due to the above this doesn't even happen most of the time, though with H/2
it will be more common.

> That advice is useful for all TCP for transactions, too.

Absolutely, it's just that HTTP tends to magnify those effects because you
can get a lot of traffic concentrated on one or a few machines, and there's
a huge pressure on the admins as they know that there are tens of thousands
of visitors witnessing their performance issues. Performance is constantly
monitored by customers as well, trying to shrink the last millisecond to rank
better on search engines, which further adds to the pressure.

> I.e., OS-specific issues in running servers are not in scope IMO.

So what you have cited above are implementation details which are either
basic to any implementer or covered by the application itself. Admins
have already dealt with this in the default tuning and the application
has already done what is supposed to be done. Yet the admin is left with
a web site that is down under load, and they need to understand what
sysctls to change and what their effects can be on the short term (ie:
put the site back online) and the mid term (ie: limit the amount of
problems caused to random visitors). Lots of people for example enable
time-wait recycling because this is done by some people in benchmarks,
it's a single sysctl and they don't realize that they cause a lot of
issues even under moderate load.

> training courses to get certification, and that's where they learn their

As Matthew said, admins tend to trust better the people who write the
protocols. Having an RFC authored by various protocol people will
definitely help make admins do the right thing.

Willy

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Aug 2016 08:07:31 -0700

> definitely help make admins do the right thing.
Admins don't know who protocol people are.

I do think that this doc needs to figure out whom it is speaking to,
what advice they actually need, etc.

If the result is a set of recommendations that involve the word
"sysctl", I remain skeptical it is appropriate as an RFC.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Aug 2016 10:50:52 -0700

Hi, Willy,

On the issue of ACK compression effects:

> about observations on average web sites.
The effect is more pronounced for smaller responses, for any response
using an odd number of packets. The last odd packet will be stalled
because the client needs to timeout before it will send an ACK for a
single segment (it's waiting for the second segment).

It doesn't matter how many requests you have, but the impact can be
complicated on persistent connections.

Joe

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 18 Aug 2016 21:59:48 +0200

> single segment (it's waiting for the second segment).

With short keep-alive responses yes but not short-lived connections
since the server sends the FIN immediately afterwards, which hints
the client not to wait for anything else and to ACK immediately.

> complicated on persistent connections.

Yes I agree. Please note that usually the last segment of a message
carries a push which precisely speeds up delivery to the application
and acking (as most OSes ack on push but apparently not all), so it
is not often emphasized that much. But I've met some 3G networks
where ACK compression was causing excessive retransmits from the
sender, resulting in excess retransmits of ACKs in turn, maintaining
the ACK link congested. It generally gets worse with many parallel
connections than with a single one, and in this regard HTTP/2 has
improved things a lot.

Willy

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Aug 2016 14:52:31 -0700

> the client not to wait for anything else and to ACK immediately.
If you have connection-per-transfer, but not if you have persistent
connections (the server wouldn't issue a close()).

> is not often emphasized that much.
PSH is optional.

The PSH bit forces data not to be aggregated by the sending TCP (i.e.,
to gather multiple send() calls) and forces the receiver to do the same,
but I don't see anywhere that PSH forces ACK compression to be
circumvented. If you have a pointer that'd be useful (I'm speaking of an
RFC). I looked for "delayed ACK" (the term used in 4.2.3.2 of RFC 1122)
and could not find any indication of a relation to PSH there, in fact
the delayed ACK discussion has no exceptions at all.

Additionally, that were true, setting the PSH bit constantly could cause
TCPs to open their slow-start windows much faster (2x per RTT, rather
than 1.5x as currently).

> But I've met some 3G networks

Most TCP variant assume that loss=congestion; that's often the wrong
interpretation for wireless.

(exceptions include TCP variants tuned for wireless)

Joe

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 19 Aug 2016 00:05:57 +0200

On Thu, Aug 18, 2016 at 02:52:31PM -0700, Joe Touch wrote:
> circumvented.

No that's not what I was saying, just that it was often avoiding the
extra 200ms delay before sending the ACK for the odd segment number
which further amplifies the ACK compression issue.

> If you have a pointer that'd be useful (I'm speaking of an

On some systems you can force to have delayed ACKs even on PSH, I do it
on linux (disable quick-ack). That's very useful to merge the response
with the ACK for the request. But these are tricky must not be abused.

> TCPs to open their slow-start windows much faster (2x per RTT, rather

I wasn't aware.

> (exceptions include TCP variants tuned for wireless)

Yep definitely. And it's even worse with deep buffers at the intersection
of fast and slow networks like the RAN for some mobile operators because
the stacks here tend to be tuned to assume that a loss is a loss and cause
fast retransmit which further fill the RAN's buffers. Sometimes the uplink
is filled with ACKs and I even found it could be efficient to kill one every
two ACKs in such a case. But I think this asymmetry is progressively going
away in favor of more balanced links.

Willy

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Aug 2016 16:03:08 -0700

Hi, Willy,

Yes - this is yet another example of TCP's "twisty maze of passages, all
alike" ;-)

Joe

---

From: Mark Nottingham <[email-masked]>
Date: Fri, 19 Aug 2016 13:49:21 +1000

> On 19 Aug 2016, at 1:07 AM, Joe Touch <[email-masked]> wrote:
> 
> I do think that this doc needs to figure out whom it is speaking to, what advice they actually need, etc.
> 
> If the result is a set of recommendations that involve the word "sysctl", I remain skeptical it is appropriate as an RFC

I think there's broad agreement on both of these points.

I'm wondering if it makes sense to aim it primarily at HTTP implementers rather than administrators, with the notion that it would inform:

- Their implementation decisions
- The configuration choices they offer to administrators / users
- Their documentation (e.g., advice to their administrators when the implementation can't change the appropriate parts of the OS)

Would that help? 

If so, it might make sense to organise it into sections for clients and servers (and intermediaries, if there's anything that isn't covered by the combination of the first two). Although IIRC Daniel was already talking about doing that.

Cheers,

---

From: Jeremy Harris <[email-masked]>
Date: Fri, 19 Aug 2016 11:50:26 +0100

> single segment (it's waiting for the second segment).

Should we be pushing for the delayed-ack timer to be autotuned?
-- 
Cheers,
  Jeremy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 19 Aug 2016 13:14:40 +0200

> Should we be pushing for the delayed-ack timer to be autotuned?

I personally don't think so. It is very important to ack quickly, especially
nowadays where some mobile phone's stacks are tuned to retransmit very
aggressively and RTTs are already getting pretty close to the ACK delays
we might reach. If we try to increase the delayed-ack timer, we'll risk
to get more retransmits (eg: twice the same request from the client, using
the smaller uplink). If we shrink them, we lose the opportunity to merge
segments. In my experience, the 200ms delayed ack and the 40ms the stack
gives me when I set MSG_MORE are pretty fine to cover all use cases I've
met in field.

Regards,
Willy

---

From: Eliot Lear <[email-masked]>
Date: Fri, 19 Aug 2016 18:52:10 +0200

Just to be the stick in the mud for a moment...

> If so, it might make sense to organise it into sections for clients and servers (and intermediaries, if there's anything that isn't covered by the combination of the first two). Although IIRC Danie…

If there are very specific settings for Linux and they are well vetted,
I see no reason not to continue to include them in an appendix, as they
are now.  Cheat sheets are nice, and there is nothing wrong with a cheat
sheet being in an RFC (and I wouoldn't mind a few being RFCs).

Eliot

---

From: Joe Touch <[email-masked]>
Date: Fri, 19 Aug 2016 10:27:26 -0700

Mark,

> - Their documentation (e.g., advice to their administrators when the implementation can't change the appropriate parts of the OS)

I think so. I also think it would be very important to differentiate the
issues that are unique to HTTP (if any) vs. those that are generic to
any transaction system.

> If so, it might make sense to organise it into sections for clients and servers (and intermediaries, if there's anything that isn't covered by the combination of the first two). Although IIRC Danie…

Clients, servers, proxies, and (even though a violation of so many parts
of the Internet) transparent proxies.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 19 Aug 2016 10:38:26 -0700

On 8/19/2016 4:14 AM, Willy Tarreau wrote:
> I personally don't think so. 
+1

Joe