Subject: [tcpm] a question for TCPLS

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 23 Mar 2022 20:34:50 -0700

Hi,
I tried to ask a question for TCPLS during the presentation, but gave up
due to time constraints. So, I just sent it here before my memory expires...

In my understanding, the advantage of MPTCP is it's in the TCP stack so
that it can have fine-grade packet scheduling and path management.
But, in case of TCPLS, I'm not sure how it can control multiple TCP
connections effectively.

For example, I am wondering how TCPLS finds the failure of connections or
which packets have been considered lost at the TCP layer without delay.
Can you provide some more info about this point?

Thanks,
--
Yoshi

---

From: Maxime Piraux <[email-masked]>
Date: Fri, 25 Mar 2022 11:51:07 +0100

Hi Yoshi,

Thank you for this question. It is very interesting as it touches on a 
fundamental choice of TCPLS.

Indeed, TCPLS does not act at the TCP segment level. Instead, TCPLS 
makes decisions at the TLS record level. This decision is driven by 
several factors. First, when TLS is used atop TCP, TCP segments of a TLS 
record become bound together, e.g. the receiver cannot decrypt their 
data before the whole record is received. The loss of one segment, no 
matter if it is in order or not, can block the whole record data from 
being decrypted and processed, remaining in the receiver buffers until 
it becomes complete.

When TLS is used atop MPTCP, this effect can be further amplified by the 
scheduling. Spreading the segments of a single TLS record over two paths 
can create an increase in transmission time, as segments arriving on the 
fast path are of no use before segments sent on the slow path are 
received. In this case, splitting the application data in two 
independent records could make more sense to the application. In 
TLS+MPTCP, one could split this data into two TLS records, but as MPTCP 
maintains a single bytestream, they will be reordered before decryption 
at the receiver. So to effectively benefit from this finer scheduling, 
MPTCP has to be aware of the TLS segmentation happening in the TCP 
bytestream, with the limitation of the receiver reordering sometimes 
going against scheduling decisions.

As far as I'm aware, there is no easy way to implement this currently. 
So far, most MPTCP schedulers have been taking decisions based on inputs 
from the network and not from the application. Conceding this level of 
granularity enables TCPLS to be implemented purely atop TCP, and we 
believe this tradeoff is of value.

Observing packet losses, dupacks, RTOs and RTT variations could still be 
achieved, for instance by using tcp_info metrics. It can then be used to 
drive the TLS record scheduling.

At any point of a TCPLS session, the sender is aware of the TLS records 
that were received, decrypted and processed by the receiver over each 
TCP connection thanks to the ACK frame. These ACKs can be sent on any 
TCP connection of the session. TCPLS can take the decision to reinject 
the content of a TLS record on another TCP connection at any point, but 
it delegates handling TCP segment loss to the TCP connection. The 
decision of reinjecting the content of a TLS record can be made based on 
many inputs, for instance a timer on the receipt of the ACK frame, 
metrics from tcp_info indicating that the connection is performing badly 
and of course a timeout of the TCP connection.

We also defined a Connection Reset frame for the receiver to notify the 
sender that it received a TCP RST on a TCP connection, this can shorten 
the time for TCPLS to detect that a TCP connection was disrupted by a 
middlebox and reinject the lost TLS records. Note that adding these kind 
of control messages in MPTCP would be achieved by adding a TCP Options, 
which is in cleartext and in a much more limited space.

Maxime

---

From: "Scharf, Michael" <[email-masked]>
Date: Fri, 25 Mar 2022 14:47:33 +0000

For what it is worth, my MCTCP prototype in 2010, which was similar to the TCPLS design, used such a user-space scheduler based on “tcp_info” data. And, obviously, MCTCP scheduled its own (TLS-like) records and relied on error recovery of each TCP connection, with all the pros and cons of that approach.

It was a KISS design. My MCTCP implementation could be used on an unmodified Linux kernel TCP/IP stack just in the user space. Implementing coupled congestion control (RFC 6356) required a small kernel patch with some few lines of code to modify the congestion control window of each coupled connection from the user space scheduler.

Michael

From: tcpm <[email-masked]> On Behalf Of Maxime Piraux
Sent: Friday, March 25, 2022 11:51 AM
To: Yoshifumi Nishida <[email-masked]>
Cc: [email-masked]
Subject: Re: [tcpm] a question for TCPLS

Hi Yoshi,
Thank you for this question. It is very interesting as it touches on a fundamental choice of TCPLS.

Indeed, TCPLS does not act at the TCP segment level. Instead, TCPLS makes decisions at the TLS record level. This decision is driven by several factors. First, when TLS is used atop TCP, TCP segments of a TLS record become bound together, e.g. the receiver cannot decrypt their data before the whole record is received. The loss of one segment, no matter if it is in order or not, can block the whole record data from being decrypted and processed, remaining in the receiver buffers until it becomes complete.

When TLS is used atop MPTCP, this effect can be further amplified by the scheduling. Spreading the segments of a single TLS record over two paths can create an increase in transmission time, as segments arriving on the fast path are of no use before segments sent on the slow path are received. In this case, splitting the application data in two independent records could make more sense to the application. In TLS+MPTCP, one could split this data into two TLS records, but as MPTCP maintains a single bytestream, they will be reordered before decryption at the receiver. So to effectively benefit from this finer scheduling, MPTCP has to be aware of the TLS segmentation happening in the TCP bytestream, with the limitation of the receiver reordering sometimes going against scheduling decisions.

As far as I'm aware, there is no easy way to implement this currently. So far, most MPTCP schedulers have been taking decisions based on inputs from the network and not from the application. Conceding this level of granularity enables TCPLS to be implemented purely atop TCP, and we believe this tradeoff is of value.

Observing packet losses, dupacks, RTOs and RTT variations could still be achieved, for instance by using tcp_info metrics. It can then be used to drive the TLS record scheduling.

At any point of a TCPLS session, the sender is aware of the TLS records that were received, decrypted and processed by the receiver over each TCP connection thanks to the ACK frame. These ACKs can be sent on any TCP connection of the session. TCPLS can take the decision to reinject the content of a TLS record on another TCP connection at any point, but it delegates handling TCP segment loss to the TCP connection. The decision of reinjecting the content of a TLS record can be made based on many inputs, for instance a timer on the receipt of the ACK frame, metrics from tcp_info indicating that the connection is performing badly and of course a timeout of the TCP connection.

We also defined a Connection Reset frame for the receiver to notify the sender that it received a TCP RST on a TCP connection, this can shorten the time for TCPLS to detect that a TCP connection was disrupted by a middlebox and reinject the lost TLS records. Note that adding these kind of control messages in MPTCP would be achieved by adding a TCP Options, which is in cleartext and in a much more limited space.

Maxime

Hi,

connections effectively.

Thanks,

--

Yoshi