Subject: Re: [rmcat] draft-sarker-rmcat-eval-test-00, test case 4.4 RMCAT Flow competing with a long TCP Flow

From: "Eggert, Lars" <[email-masked]>
Date: Thu, 1 May 2014 08:12:00 +0000

Hi,

one additional source of reordering we've seen is caused by the receiver stack itself. At least Linux currently seems to introduce it when the system gets fully loaded (e.g., with 40G NICs) and the scheduler moves queue processing between cores. (This is anecdotal at the moment, we're trying to quantify/qualify it better.)

I'm sure this is a transient problem as far as the kernel source is concerned, but boxes based on the current kernels may live in deployments for quite some time before being updated.

Lars

On 2014-4-30, at 19:02, Fred Baker (fred) <[email-masked]> wrote:

> 
> On Apr 30, 2014, at 4:50 AM, Zaheduzzaman Sarker <[email-masked]> wrote:
> 
>> Are you saying re-ordering is not a phenomena that we can observe in the current Internet? And we should not have test cases to handle re-ordering of packets in RMCAT?
> 
> Packet reordering, loss, and duplication happen. Where and to what degree is a question of the structure of networks that packets pass through and what happens in those networks. The last time I know of someone doing an actual measurement and reporting the results was rather a while ago, but it said that at that time about 5% of arriving traffic was reordered. For systemic reasons, I would imagine this is reduced, and very dependent on what networks and types of networks are crossed.
> 
> Going out on a limb a bit (this is what I suspect, not what I know), I would imagine that today there are three major sources of traffic reordering. One is loss; in radio networks (which include 3g/etc but also include 802.11, 802.15, and 802.16), loss and retransmission are pretty common. It would be interesting to have someone write a paper measuring and analyzing the resultant “best effort” behaviors in those networks. Another is load balancing in the network; while it is common for ECMP routing to send a subset of flows one way and another subset on another, some networks schedule packets on the most available bandwidth or go round robin. That is a recipe for reordering. The other has to do with adaptive bitrate video and services that deliver it by literally duplicating data. In these services, there is a path from the CDN to the consumer that consists of two disjoint paths. The TCP originator is configured with a link layer multicast address for its first hop router, and there are in fact two such routers, one on each of the parallel paths. The packets don’t arrive at the endpoint, but they do arrive at a point just before it, and the first of each pair of packets to arrive is forwarded to the endpoint. You can imagine that the collection point sees frequent duplication and some degree of reordering.
> 
> The single biggest source of reordering, I will postulate, is none of those; it is the result of loss due to TCP’s AIMD behavior. When a packet is dropped in flight due to queue depth or delay, the rest of the packets in that RTT arrive before it, and then its retransmission arrives. Voila! Packet reordering.
> 
> Should TCP handle packet reordering? Yes, of course. Making packet arrival sequence completely random will certainly test that behavior; we could discuss the beneficial effects of running TCP over CSNET in the 1980’s and the resulting hardening of TCP’s algorithms. That doesn’t make that level of reordering a valid simulation of the Internet. A more valid simulation will have one or both of two characteristics - that a packet is dropped, the remaining packets in that RTT pass it, and then it arrives, or that packets frequently arrive disordered but to a degree that doesn’t trigger TCP’s Fast retransmit algorithm.

---

From: Randell Jesup <[email-masked]>
Date: Fri, 02 May 2014 05:49:16 -0400

On 4/30/2014 1:51 PM, Colin Perkins wrote:
> On 30 Apr 2014, at 07:50, Zaheduzzaman Sarker <[email-masked]> wrote:
> One of my students collected some UDP streaming measurements from our university to hosts on residential networks, covering all the main UK ISPs, a couple of years ago. With everything using wired connections and UDP we saw a couple of hundred packets delivered out of order, from ~250 million packets sent. I can dig into the details of the re-ordering patterns if there’s interest. We didn’t analyse them too closely at the time because packet loss and delay spikes were much more common.

Having it not fall over on (some) reordering: good.  My experience has 
been only specific unusual network/hardware situations cause more than 
0.0x% reordering of packet delivery (at the IP level, ignoring things 
like TCP retransmit, which is not an IP-level reordering in the way I'm 
discussing).

I've seen significant reordering only on a network (an old office 
network of mine!) where we had multiple bonded T1's and the packets went 
across one of them randomly (I presume).  I think I got to 
low-single-digit percent or more likely 0.n% (like 0.1-0.9%) reordered.  
There may also be some wireless networks where this is more likely than 
on normal wired; I haven't noticed it on WiFi though.

It's easy to accidentally set up DummyNet/etc to cause massive packet 
reordering, however.  I do not think that should be a primary or even 
secondary testcase, and (to be honest) I don't think we should care much 
about degenerate reordering cases.  I'm not even sure we should bother 
spending much energy in our evaluation tests making it happen at all 
(which isn't to say that the algorithm authors shouldn't consider it and 
if they feel it would affect them, test it or discuss it - plus they 
will see a small amount of it on real networks).

-- 
Randell Jesup -- rjesup a t mozilla d o t com

---

From: Stefan Holmer <[email-masked]>
Date: Fri, 2 May 2014 16:08:25 +0200

On Fri, May 2, 2014 at 11:49 AM, Randell Jesup <[email-masked]>wrote:

>>> Are you saying re-ordering is not a phenomena that we can observe in the
>>> current Internet? And we should not have test cases to handle re-ordering
>>> of packets in RMCAT?
>>>
>> One of my students collected some UDP streaming measurements from our
>> university to hosts on residential networks, covering all the main UK ISPs,
>> a couple of years ago. With everything using wired connections and UDP we
>> saw a couple of hundred packets delivered out of order, from ~250 million
>> packets sent. I can dig into the details of the re-ordering patterns if
>> there’s interest. We didn’t analyse them too closely at the time because
>> packet loss and delay spikes were much more common.
>>
>
> 0.0x% reordering of packet delivery (at the IP level, ignoring things like
> TCP retransmit, which is not an IP-level reordering in the way I'm
> discussing).
>
> I've seen significant reordering only on a network (an old office network
> of mine!) where we had multiple bonded T1's and the packets went across one
> of them randomly (I presume).  I think I got to low-single-digit percent or
> more likely 0.n% (like 0.1-0.9%) reordered.  There may also be some
> wireless networks where this is more likely than on normal wired; I haven't
> noticed it on WiFi though.

> spending much energy in our evaluation tests making it happen at all (which
> isn't to say that the algorithm authors shouldn't consider it and if they
> feel it would affect them, test it or discuss it - plus they will see a
> small amount of it on real networks).

+1