Subject: TCP Tuning for HTTP

From: Daniel Stenberg <[email-masked]>
Date: Fri, 6 Nov 2015 11:33:02 +0100 (CET)

Hi friends!

I've just submitted the 00-version of the document "TCP Tuning for HTTP"[1] 
and I will appreciate any and all sorts of feedback and additional good advice 
you can provide. The good ideas we need to tell our fellow HTTP implementers 
on how we poke on the TCP stack to make it dance for us.

It is still a bit thin and there are some white spots in it. I'm counting on 
some help there! =)

[1] = https://www.ietf.org/id/draft-stenberg-httpbis-tcp-00.txt

-- 

  / daniel.haxx.se

---

From: Cory Benfield <[email-masked]>
Date: Fri, 6 Nov 2015 11:06:24 +0000

> On 6 Nov 2015, at 10:33, Daniel Stenberg <[email-masked]> wrote:
> 
> Hi friends!
> 
> I've just submitted the 00-version of the document "TCP Tuning for HTTP"[1] and I will appreciate any and all sorts of feedback and additional good advice you can provide. The good ideas we need to tell our fellow HTTP implementers on how we poke on the TCP stack to make it dance for us.
> 
> It is still a bit thin and there are some white spots in it. I'm counting on some help there! =)
> 

Daniel,

The first draft looks great! This is a really good start. I have a few small stylistic changes that I’ll propose as PRs to your repo, but wanted to add some more detail for stuff that seems missing or needs elaboration.

1. net.core.somaxconn

I did some digging here because I was interested in how this interacted with the ‘backlog’ parameter to listen(). It seems that, on Linux, net.core.somaxconn represents the maximum value. This means that the maximum backlog on a listening socket is min(backlog, somaxconn). It would probably help to elaborate this section, but elaborating that section probably doesn’t make sense without also talking about the backlog argument to listen() itself. I’m willing to submit some text for that if you’re interested.

Your description of somaxconn (“the number of incoming TCP SYNs allowed to backlog”) is not actually accurate, I don’t think, because of the next section.

2. tcp_max_syn_backlog

You don’t mention the setting related to net.core.somaxconn, which is net.ipv4.tcp_max_syn_backlog. somaxconn and the listen backlog on Linux  apply to acknowledged connections (that is, the handshake is completed). tcp_max_syn_backlog applies to connections that have not yet been acknowledged by the client. I can’t find a corresponding IPv6 setting, so we might need to do some more digging here to see if this is still used: does anyone else on this list know?

3. Separate into client, server, and both optimisations.

Some of the optimisations and tweaks suggested here only make sense for servers (e.g. somaxconn), but some are good for both clients and servers. It would be good if we could call this out a bit more.

Otherwise, I’m really looking forward to seeing this develop!

Cory

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 6 Nov 2015 17:15:42 +0000

Daniel, all,

I'd suggest to CC the tcpm list when discussing this draft. Basically all major TCP implementers monitor tcpm, and you should find useful expertise there. However, I am not sure whether all tcpm contributors love dancing ;)

Thanks

Michael

> -----Original Message-----
> From: Cory Benfield [mailto:[email-masked]]
> Sent: Friday, November 06, 2015 12:06 PM
> To: Daniel Stenberg
> Cc: HTTP Working Group
> Subject: Re: TCP Tuning for HTTP
> 
> 
> > I've just submitted the 00-version of the document "TCP Tuning for
> HTTP"[1] and I will appreciate any and all sorts of feedback and
> additional good advice you can provide. The good ideas we need to tell
> our fellow HTTP implementers on how we poke on the TCP stack to make it
> dance for us.
> >
> The first draft looks great! This is a really good start. I have a few
> small stylistic changes that I’ll propose as PRs to your repo, but
> wanted to add some more detail for stuff that seems missing or needs
> elaboration.
> 
> 1. net.core.somaxconn
> 
> I did some digging here because I was interested in how this interacted
> with the ‘backlog’ parameter to listen(). It seems that, on Linux,
> net.core.somaxconn represents the maximum value. This means that the
> maximum backlog on a listening socket is min(backlog, somaxconn). It
> would probably help to elaborate this section, but elaborating that
> section probably doesn’t make sense without also talking about the
> backlog argument to listen() itself. I’m willing to submit some text
> for that if you’re interested.
> 
> Your description of somaxconn (“the number of incoming TCP SYNs allowed
> to backlog”) is not actually accurate, I don’t think, because of the
> next section.
> 
> 2. tcp_max_syn_backlog
> 
> You don’t mention the setting related to net.core.somaxconn, which is
> net.ipv4.tcp_max_syn_backlog. somaxconn and the listen backlog on Linux
> apply to acknowledged connections (that is, the handshake is
> completed). tcp_max_syn_backlog applies to connections that have not
> yet been acknowledged by the client. I can’t find a corresponding IPv6
> setting, so we might need to do some more digging here to see if this
> is still used: does anyone else on this list know?
> 
> Some of the optimisations and tweaks suggested here only make sense for
> servers (e.g. somaxconn), but some are good for both clients and
> servers. It would be good if we could call this out a bit more.
>

---

From: Daniel Stenberg <[email-masked]>
Date: Sat, 7 Nov 2015 15:46:28 +0100 (CET)

On Fri, 6 Nov 2015, Cory Benfield wrote:

> The first draft looks great! This is a really good start. I have a few small 
> stylistic changes that I’ll propose as PRs to your repo, but wanted to add 
> some more detail for stuff that seems missing or needs elaboration.

The draft is on github and can be found here:
    https://github.com/bagder/I-D/blob/gh-pages/httpbis-tcp/draft.md

> 1. net.core.somaxconn
>
> I did some digging here because I was interested in how this interacted with 
> the ‘backlog’ parameter to listen(). It seems that, on Linux, 
> net.core.somaxconn represents the maximum value. This means that the maximum 
> backlog on a listening socket is min(backlog, somaxconn). It would probably 
> help to elaborate this section, but elaborating that section probably 
> doesn’t make sense without also talking about the backlog argument to 
> listen() itself. I’m willing to submit some text for that if you’re 
> interested.

I'm interested! Please suggest improvements to whatever section you think you 
can improve. I'm here to help producing a document to help current and future 
implementers and I willingly take all the help I can get.

> Your description of somaxconn (“the number of incoming TCP SYNs allowed to 
> backlog”) is not actually accurate, I don’t think, because of the next 
> section.

Nah, it is a bit too simplistic I guess. I also didn't want to make a document 
that only lists lots of Linux-specific details but rather wanted to mention 
that there can be limits "in this style". I'm sure that limit will work 
slightly different in other operating systems.

I also try to avoid exact numbers in the document for that reason. Also 
because fixed limits tend to grow old very fast while general advice may live 
on longer.

> net.ipv4.tcp_max_syn_backlog.

Correct or not, I was thinking the specific SYN (flood) handling would be 
better referred to RFC 4987. You think this option is worth mentioning 
explicitly in this document?

> 3. Separate into client, server, and both optimisations.

I've struggled back and forth on how to group the different suggestions and I 
don't think I've found the perfect layout just yet. I'm not sure it will be 
that easy to make separate sections for client, server or both and have the 
text flow naturally. Maybe it is enough to just mention the different target 
use cases in the text when they are clearly distinct?

It also most likely varies a lot depending what kind of servers or clients 
we're talking about.

Thanks a lot for your feedback and help!

-- 

  / daniel.haxx.se

---

From: "c^" <[email-masked]>
Date: Sun, 8 Nov 2015 13:19:37 +0900

I think the server client split is probably a good one, although it's
probably more common/server specific...

I have something in my head to submit with regards to planning and
derivation of the TCP window scaling values. Unless someone beats me to it,
you'll have something in the next two days...

--
c%5e
On 7 Nov 2015 14:51, "Daniel Stenberg" <[email-masked]> wrote:

>> small stylistic changes that I’ll propose as PRs to your repo, but wanted
>> to add some more detail for stuff that seems missing or needs elaboration.
>>
>
>> maximum backlog on a listening socket is min(backlog, somaxconn). It would
>> probably help to elaborate this section, but elaborating that section
>> probably doesn’t make sense without also talking about the backlog argument
>> to listen() itself. I’m willing to submit some text for that if you’re
>> interested.
>>
>
> I'm interested! Please suggest improvements to whatever section you think
> you can improve. I'm here to help producing a document to help current and
> future implementers and I willingly take all the help I can get.
>
> Nah, it is a bit too simplistic I guess. I also didn't want to make a
> document that only lists lots of Linux-specific details but rather wanted
> to mention that there can be limits "in this style". I'm sure that limit
> will work slightly different in other operating systems.
>
> because fixed limits tend to grow old very fast while general advice may
> live on longer.
>
> 2. tcp_max_syn_backlog
>>
> I've struggled back and forth on how to group the different suggestions
> and I don't think I've found the perfect layout just yet. I'm not sure it
> will be that easy to make separate sections for client, server or both and
> have the text flow naturally. Maybe it is enough to just mention the
> different target use cases in the text when they are clearly distinct?
>