Subject: Re: [tcpm] Vicious or Virtuous circle? Adapting reordering window to reordering degree

From: [email-masked]
Date: Wed, 28 Mar 2018 15:50:20 +0200

For the larger TCPM audience,

I want to note here, that nvo3 wg is looking for packet spraying (per-packet ECMP) to increase overall network bandwidth of highly meshed topologies.

The last-hop switch, however, is supposed to have a reorder buffer, to deliver the packets to the client again in-sequence.

So while the exterior view appears to be a network capable of in-order delivery, the impact of this would be increased network delay (reorder buffer until a short timeout expires, or all in-sequence packets arrived at that buffer)...

I believe, currently this design (reorder buffer on the last hop of the network) is a reasonable approach based on current expectations of network delay and protocols deployed, this may not cater for all users (especially at the extremely low latency end of the spectrum, e.g. RDMA use cases).

If any of you have strong opinions on this, please check with the authors of https://tools.ietf.org/html/draft-xiang-nvo3-geneve-packet-spray-00

Best regards,
  Richard

-----Ursprüngliche Nachricht-----
Von: "Praveen Balasubramanian" <[email-masked]>
Gesendet: Montag, 26. März 2018 17:45
An: "Yuchung Cheng" <[email-masked]>; "Olivier Bonaventure" <[email-masked]>; "Bob Briscoe" <[email-masked]>
Cc: "[email-masked]" <[email-masked]>; "[email-masked]" <[email-masked]>
Betreff: Re: [tcpm] Vicious or Virtuous circle? Adapting reordering window to reordering degree

There are a lot of assumptions in not just host TCPIP stacks but in networking stack in general that reordering is not the common case. We put in a lot of work to make the common in order case work blazingly fast all the way from the hardware to the software stack. Yes the stack needs to tolerate some reordering but we are not ready for a world where networks build in reordering as a common case. Any such behavior will cause severe performance issues.

How about if the draft specify a minimum reordering threshold in time (RTT/4) and number of packets (3) and leaves the further extension of increasing the threshold as an optional improvement to handle bad network behavior? Without doing tests I cannot determine the memory and CPU impact of allowing it to grow to SRTT worth in time domain. Also I worry about what Bob is saying that this could become a vicious cycle with increasing reordering -- and even if we fix RACK the rest of hardware and software stack will not be ready for it. Deliberate reordering is not an acceptable network behavior, it should only happen during rare events like route flaps or ECMP flaps.

Thanks

From: tcpm [mailto:[email-masked]] On Behalf Of Olivier Bonaventure
Sent: Thursday, March 22, 2018 12:48 PM
To: Bob Briscoe <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [tcpm] Vicious or Virtuous circle? Adapting reordering window to reordering degree

Bob,

What I was trying to say at the tcpm mic just now.
This could be considered as a first draft of the text you asked me to write....
The Bigger Picture
The reordering degree of a link is not a constant. It evolves (over years) dependent on TCP.

  *   From a link designers viewpoint, parallelizing (eg. link bonding) is the easiest way to get their link to go faster

     *   So their main constraint on speed is reordering and there is huge pressure on them to relax that constraint
     *   The /only/ thing that is holding them back from introducing more reordering is the 3 dupacks in TCP or the RTT/4 rule in RACK-02

  *   It is likely RACK will become widespread
  *   Given RACK-03 now adapts to the reordering window, link designers will introduce more reordering
  *   That is GOOD (virtuous circle), but only up to a point (vicious circle).

     *   We need to write down what that point is
     *   We need to ensure the algorithm stops adapting at that point.

I agree that parallelising is a nice way to expand the capacity of links, typically in the access. However, link designers should not assume that they can parallelise whatever they want and let TCP figure out what is happening and solve any issue.

If TCP packets are sent over two parallel links that have similar characteristic, then it should work fine.. However, as soon as one of these links has different characteristics (in terms of delay, bandwidth, congestion, packet losses), TCP will have difficulties to understand what is happening in the network if it cannot know which packet is sent over which link. A better approach when bonding access links is to expose them to a multipath capable transport layer and let it figure out the characteristics of the different links and use them efficiently.. With IPv6, access link designers have the ability to expose multiple addresses to the endhosts and we should encourage this behaviour when those addresses correspond to links with different characteristics

Olivier

________________________________
DISCLAIMER.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Wed, 28 Mar 2018 06:56:27 -0700

> On Mar 28, 2018, at 6:50 AM, [email-masked] wrote:
> 

We have different ideas of what the “EC” in ECMP mean.

If you have to reorder, then they’re not equal cost.

Joe