Subject: [TLS] TLS Record Size Limitation

From: Software Engineer 979 <[email-masked]>
Date: Mon, 7 Dec 2015 15:00:54 -0600

Hello,

I'm currently developing an data transfer application using OpenSSL. The
application is required to securely transfer large amounts of data over a
low latency/high bandwidth network. The data being transferred lives in a
3rd part application that uses 1 MB buffer to transfer data to my
application. When I hook OpenSSL into my application I notice an
appreciable decline in network throughput. I've traced the issue the
default TLS record size of 16K. The smaller record size causes the 3rd
party application's buffer to be segmented into 4 16K buffers per write and
the resulting overhead considerably slows things down. I've since modified
the version of OpenSSL that I'm using to support an arbitrary TLS record
size allowing OpenSSL to scale up to 1MB or larger TLS record size. Since
this change, my network throughput has dramatically increased (187%
degradation down to 33%).

I subsequently checked the TLS RFC to determine why a 16K record size was
being used, and all could find was the following:

length
      The length (in bytes) of the following TLSCompressed.fragment.

      The length MUST NOT exceed 2^14 + 1024.

The language here is pretty explicit stating that the length must not
exceed 16K (+ some change).Does anyone know the reason for this? Is there a
cryptographic reason why we shouldn't exceed this message size? Based on my
limited experiment, it would appear that a larger record size would benefit
low latency/high bandwidth networks.

---

From: Dave Garrett <[email-masked]>
Date: Tue, 8 Dec 2015 03:49:59 -0500

> low latency/high bandwidth networks.

Well, the length field is 16-bit, so 2^16 is the hard upper limit for record format compatibility. (I don't personally know why it's less than that here, offhand) A TLS extension to negotiate max length might be viable. The handshake would have to keep to the old limits for backwards compatibility, but after that records could use a larger field (or add a multiplier to use 16-bit to cover larger sizes with padding as-needed).

Dave

---

From: Yoav Nir <[email-masked]>
Date: Tue, 8 Dec 2015 11:32:13 +0200

> On 7 Dec 2015, at 11:00 PM, Software Engineer 979 <[email-masked]> wrote:
> 
> 
>> Hello,
>> 
>> I'm currently developing an data transfer application using OpenSSL. The application is required to securely transfer large amounts of data over a low latency/high bandwidth network. The data being transferred lives in a 3rd part application that uses 1 MB buffer to transfer data to my application. When I hook OpenSSL into my application I notice an appreciable decline in network throughput. I've traced the issue the default TLS record size of 16K. The smaller record size causes the 3rd party application's buffer to be segmented into 4 16K buffers per write and the resulting overhead considerably slows things down. I've since modified the version of OpenSSL that I'm using to support an arbitrary TLS record size allowing OpenSSL to scale up to 1MB or larger TLS record size. Since this change, my network throughput has dramatically increased (187% degradation down to 33%). 
>> 
>> I subsequently checked the TLS RFC to determine why a 16K record size was being used, and all could find was the following:
>> 
>> length
>> The language here is pretty explicit stating that the length must not exceed 16K (+ some change).Does anyone know the reason for this? Is there a cryptographic reason why we shouldn't exceed this message size? Based on my limited experiment, it would appear that a larger record size would benefit low latency/high bandwidth networks.

This is only a guess, but you usually want to fetch the entire record off the network before you begin decrypting and checking MAC (unfortunately, it that order), so the intention was to limit that amount of buffering the receiver is required to do. Perhaps also allow for more parallel network fetching and decryption.  In either case, these are decisions made almost 20 years ago in SSLv3, and they may or may not make sense today. Although with all the IoT devices, it might make even more sense today to reduce buffering requirements.

Yoav

---

From: Peter Gutmann <[email-masked]>
Date: Tue, 8 Dec 2015 10:11:52 +0000

Dave Garrett <[email-masked]> writes:

>A TLS extension to negotiate max length might be viable.

I think a better starting point would be to look at the implementation that's
causing the problem.  There's nothing magical about a 16K max segment size
that causes poor performance, TCP typically has an MSS of 1400-1500 bytes, one
tenth of the TLS segment size, without there being a 187% loss in throughput
so it looks like the problem is in the implementation, not the protocol.  I
don't see any reason why you couldn't get close to wire speeds, or at least
min( crypto speed, wire speed ) for TLS for a properly-done implementation.

Peter.

---

From: "Yngve N. Pettersen" <[email-masked]>
Date: Tue, 08 Dec 2015 12:20:39 +0100

> min( crypto speed, wire speed ) for TLS for a properly-done  

Based on my past experience, a possible reason is that the code does the  
following:

read_event_handler:
   read data from socket
   decrypt data
   handle application data
   return

rather than:

read event handler:
   read data from socket
   queue data for decryption
   signal decryption handler
   return

decryption handler:
   decrypt data
   handle application data

Another factor I have seen influencing speed is the size of the buffer  
reading from the socket, bigger buffer gives better speed. Within reason,  
of course; IIRC the benefit of increasing the buffer stops at around 5-10%  
of the connection speed. A good rule of thumb is that the buffer should be  
larger than the largest block that you will ever receive over the  
computer's local connection, provided you read the arriving data when you  
are notified about the data.

I recently saw 10x+ speed increase by changing an application's handling  
of these two aspects (on Windows). In that case 300KB buffer was  
sufficient, but I prefer a 1 MB buffer.

-- 
Sincerely,
Yngve N. Pettersen

---

From: [email-masked] (Martin Rex)
Date: Wed, 9 Dec 2015 17:13:06 +0100 (CET)

> degradation down to 33%).

I have strong doubts that the TLS record size is the problem here.

But I've seen higher layer design flaws before, which can cause
such problems.  The OpenSSL membio-interface seems to exhibit 
quite sub-optimal behaviour, because it moves data around like crazy
if you fill it with MBytes of data.

-Martin

---

From: Software Engineer 979 <[email-masked]>
Date: Tue, 8 Dec 2015 07:21:34 -0600

Thanks for replies everyone. I also posted the same question the OpenSSL
mailing list. One interesting response stated that the size was limited to
prevent DOS attacks due to resource exhaustion (in the case that data was
being injected).

"The peer is required to buffer the entire record before processing it, at
at that point the data could be from an untrusted party/attacker.  So the
limit is for protection against denial-of-service via resource exhaustion."

Not sure If I totally buy this as you have the same issue (on a smaller
scale) w/16K buffers. Also, you underlying application implementation would
have alot to due with as well.

On Tue, Dec 8, 2015 at 5:20 AM, Yngve N. Pettersen <[email-masked]>
wrote:

> On Tue, 08 Dec 2015 11:11:52 +0100, Peter Gutmann <
> [email-masked]> wrote:
>