Subject: Re: [tcpm] (ref. RFC 7323) Increase of shift count from 14 to 15.

From: Yoshifumi Nishida <[email-masked]>
Date: Sun, 4 Sep 2016 03:53:54 -0700

Hi Sanjeev,

I think there might be some useful cases if we can change shift count
in the middle of the session.
However, one tricky thing for this is you'll need to send new shift
count reliably. Because when an end node sends a new shift count, it
will be problematic if the peer cannot receive it.
But, TCP doesn't have a good mechanism to send TCP options reliably
during data transfer. So, in order to build this feature, we'll need
to think about how to solve it. That's the reason I think it'll be a
rather broad topic.

Thanks,
--
Yoshi

On Tue, Aug 30, 2016 at 9:02 PM, Sanjeev Ranot
<[email-masked]> wrote:
> Hi Yoshi,
>
> Yes, using shf.cnt of 15 in connection establishment phase is sufficient
> enough, but  while thinking from migration and interoperability this
> suggestion of changing shf.cnt during Data Transfer Phase came to mind.
>
> Present behavior is that if window limit is reached then we cannot
> transfer no matter how big cwnd is. This limits the rate of growth of
> congestion window if not growth completely. Now with high speed interfaces
> it will not always be the case that we increase size of MTU (as for jumbo
> frames) to get the bandwidth with support of bundling (etherChannels).
>
> Generally when shf.cnt is chosen it defines the size of SND.WND
> within range from 0 to max SND.WND. SND.WND proves to be a
> constraint in situation discussed below.
>
> Note: With PAWS we need to make sure that it is sufficiently large
> enough to provide ample monotonically non-decreasing numbers so that
> they can be differentiated or make its use in such a way so that
> its always sufficient to distinguish at-least between 2 successive
> windows. RFC 7323 defines such a way by putting same TSval for all
> segments belonging to the same window.
>
> Before citing situation for the need of dynamic exchange of shf.cnt we
> need to discuss why sequence number of greater than 32 bit is not needed
> because of PAWS and PAWS timestamp Interface transfer speed and window
> size are related
>
> With PAWS, each wrap of timestamp must be >= 2*MSL. This MSL is based
> on transfer rate. Assuming 1 timestamp tick is 1 ms, hence if we
> transfer 2^30 bytes of data in each window i.e. with shf.cnt = 14 then
> maximum transfer rate which we need to get this big a window is of
> 1 Tera bytes (TbE) with PAWS in place
>
> 1000 ms (1s) = 1000 Gb (1TbE)
> 1ms  = 1 Gb ~ 2^30
>
> Therefore with PAWS timestamp tick of 1ms allows 1 Gb of window size that
> we can transmit safely without data corruption due to duplicate data or
> ACK segments. This minimum timestamp click needed for PAWS safety is
> provided with a base consideration of sequence number of 32 bits in size.
> It takes 49.6 days for a 32 bit sequence number to wrap again when
> timestamp tick is 1ms.  If in future we were to increase sequence numbers
> to 64 bits, then this timestamp tick can be reduced or we MAY never need
> PAWS again thereafter if 64 bit sequence number is used and we can never
> increase bandwidth speed no matter how much we bundle cables.
>
> The situation discussed assume that we are never going to increase the
> sequence number size and we are good with PAWS to provide us safety from
> duplicate data and ACK segment or of old packets from previous incarnations
> of the same TCP connection.
>
> Another example with shf.cnt = 15 :
> With shf.cnt = 15
> 1000ms  = 2000 Gb (2TbE)
> 1ms        = 2000000000 bits = 2 Gb ~ 2^31
> Therefore with a speed of 2Tbps and 1 ms timestamp tick for PAWS we can
> get a window size of 2 Gb.
>
> Migration Notes:
> This means that when we introduce a shf.cnt of 15, then we MAY either let
> PAWS timestamp tick to remain same as used for 1 Gb of window size with
> shf.cnt of 14 if its sufficient enough or we may think to readjust if we
> want to gain more speed by lowering the previous PAWS timestamp.
>
> Similary with a speed of 4Tbps and 1 ms timestamp tick for PAWS we can
> get a window size of 4 Gb.
>
> Situation:
> TCP connection is going on for days using same shf.cnt which was
> exchanged on day 1 when this TCP connection was established. Suppose
> due to incredible high transfer rate of the inter-continent link
> cwnd == SND.WND was already reached at end of day 1 and now the
> transfer rate is limited by SND.WND (decided by shf.cnt) when only
> very small fraction of link speed is utilized. Now suppose a very
> huge surplus of memory is made available by hot-swappable memory
> card inserted on both ends while TCP connection is running its
> Data Transfer Phase. Shf.cnt was already decided in beginning of
> this TCP connection and it was decided on the basis of amount on
> total memory available at that time to be allocated to its RCV.BUF.
> It required restarting this ongoing connection to have a new shf.cnt
> that will now consider this new additional memory surplus (i.e.
> increase RCV.BUF) and start all over again from start or further
> from the last index received is area of improvement.
>
> My proposal is that (this is separate from the on going work on shf.cnt =
> 15) :
> If TCP stack have support of exchanging shf.cnt in data transfer phase
> when large amount of memory is available dynamically then we can achieve
> significant high transfer rates without ever having to restart the
> connection.
>
> This feature comes in heading "Dynamically adjustable TCP window" and
> shf.cnt_interoperability.pdf mentions one of the possible mechanisms
> for implementing this introducing 7 new state variables and 1 TCP flag.
>
> Also, just to reiterate regarding migration purposes please look into
> PAWS and discuss with stake holders if they are ok with existing
> timestamp for new shf.cnt of 15.
>
>
> Thanks & Regards,
> Sanjeev Ranot
>
>
>
> On 29 August 2016 at 13:02, Yoshifumi Nishida <[email-masked]>
> wrote:
>>
>> Hi Sanjeev,
>>
>> On Sun, Aug 28, 2016 at 1:55 AM, Sanjeev Ranot
>> <[email-masked]> wrote:
>> > Hi Yoshi,
>> >
>> > thanks for replying. I have done some work in these couple of days on
>> > how exchange and interoperability of shift count 15 or greater will
>> > work.
>> >
>> > Basically my idea is for the new TCP stack (which will support shf.cnt
>> > >=
>> > 15)
>> > to exchange shf.cnt even in Data Transfer Phase and not only in
>> > connection
>> > establishment. I have presented this exchange work in the document
>> > enclosed.
>> >
>> > Please have a look at the document and let me know for any info or
>> > clarification
>>
>> Thanks for the doc.
>> I personally think you might want to clarify why we want to change
>> shift count in data transfer phase a bit more.
>> This is because even if we use shift count=15, I think we can specify
>> rather small window size by using a small number for the timestamp
>> value. So, if the stack supports shift count=15, I think it can just
>> use it from the beginning of the data transfer in most cases.
>>
>> If your focus is to have a feature to update shift count during data
>> transfer phase in general, I think it can be a separate proposal from
>> increasing max shift count. Because it looks a more broad topic to me.
>>
>> Thanks,
>> --
>> Yoshi
>
>

---

From: Sanjeev Ranot <[email-masked]>
Date: Mon, 5 Sep 2016 09:48:11 +0530

Hi Yoshi,

thanks for the reply. I really want you and team to take forward this idea
of
dynamically changing the shift count whenever in future you start work on
this.
I will be glad if it gets implemented.

At present its out of my capacity to take it forward because I'm looking to
relocate permanently abroad by getting some job and so is trying to meet
the ends meet.

As per your concerns, I too had similar concerns of reliably communicating
this information to the peer therefore in the document which contains
implementation of the idea i.e.
*"shf.cnt_interoperability.pdf"*
has a note:

*"The condition is that these shf.cnt update will be transmitted only
reliablyon data segment or on non-empty ack segment (NOT using empty
acksegment or pure ack)"*

Thanks again for your and team's valuable time and input.

Thanks & Regards,
Sanjeev Ranot

---

From: David Borman <[email-masked]>
Date: Tue, 6 Sep 2016 09:24:49 -0500

The biggest problem with changing the shift count midstream is that TCP has no reliable mechanism for doing option negotiation midstream.  The only packets that are reliably delivered are packets that contain data or have the SYN or FIN bit set.  If the receiver changes its shift count mid-stream, the sender side needs to know where in the packet stream the new value applies.  You have to allow for lost or reordered packets, and the receiver, in a one-way data transfer, is only sending ACK packets, none of which are reliably transferred and have no ordering, the receivers sequence number remains constant, making it impossible to reliably identify a point in time where the new value takes effect.

One way to reliably change the shift count is to put it into every packet, for interoperability older implementations could be identified because they would only have it in the SYN packets.  This was considered years ago but was ruled out: the limitations of only negotiating the shift count during connection establishment was considered acceptable compared to putting it into every packet and using up valuable TCP option space.

To use a shift count of 15, you have to be able to determine whether or not the other side can accept a value of 15.  If you send a 15, but are talking to an older implementation that clamps it at 14 and you don’t know that, then your window values, sent assuming a shift count of 15, will be halved by the receiver who clamped it at 14; that is probably not desirable as the whole point of have a shift count of 15 is to allow larger windows, not smaller windows.

As to why the value is 14, that is explained in RFC 7323, section 2.4.  The logic is based on the assumption that if you allow a scale of 15, that at some point someone will try to use all of it, and limiting it to 14 ensured that we wouldn’t run into any sequence space wrap-around problems.  Whether or not that logic is correct is a separate discussion, but that’s why 14 was chosen.

			-David Borman

> On Sep 4, 2016, at 5:53 AM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Sanjeev,
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Tue, 6 Sep 2016 11:14:27 -0700

Hi, all,

On 9/6/2016 7:24 AM, David Borman wrote:
> The biggest problem with changing the shift count midstream is that TCP has no reliable mechanism for doing option negotiation midstream.
Agreed.

>   The only packets that are reliably delivered are packets that contain data or have the SYN or FIN bit set.

AFAICT, this isn't strictly true.

The data and connection state are reliably "delivered" (or
synchronized), but not the packets per se. Options in those packets
aren't reliably delivered any more than any other packet, esp. because
TCP has no strict requirement that the options of retransmitted segments
don't vary. TCP gets around this by confirming the options exchanged
during the SYN, but no other time during the connection.

You *can* build your own handshake on top of TCP, but it's complicated.
We did this for TCP-AO to support key rollover - the sender offers a new
key, which cannot be used until the receiver confirms it. The same
happens in the opposite direction. As Dave pointed out, this takes up
option space (one extra byte in AO, in addition to the confirmation
being overloaded with the current keyID) and requires that the option
"protocol" consider how to handle handshake failure, packet reordering, etc.

Joe

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 7 Sep 2016 11:55:38 -0700

Hi David,

> To use a shift count of 15, you have to be able to determine whether or not the other side can accept a value of 15.  If you send a 15, but are
> talking to an older implementation that clamps it at 14 and you don’t know that, then your window values, sent assuming a shift count of 15, will
> be halved by the receiver who clamped it at 14; that is probably not desirable as the whole point of have a shift count of 15 is to allow larger
> windows, not smaller windows.

Yes, exactly.
But, I think we might be able to presume people who want to use shift
count 15 will allocate more than 65535 * 2**15 + 65535 * 2**14 bytes
memory for the connection.
This won't be so difficult as I believe they should be able to
allocate at least 65535 * 2 ** 14 bytes memory. Otherwise they don't
need to use shift count 15.
If this is possible, it can always offer 65535 window value when the
peer uses shift count 14.
This allows the peer to use the maximum window size it can handle.

Thanks,
--
Yoshi