Subject: [DMM] some test results of different network overlay methods

From: "Bogineni, Kalyani" <[email-masked]>
Date: Fri, 16 Mar 2018 18:16:01 +0000

All:

Here are some raw performance test results based on our understanding of the different network
overlay methods. We welcome discussion and comments.

Kalyani and Tom

---

From: Uma Chunduri <[email-masked]>
Date: Fri, 16 Mar 2018 19:40:52 +0000

Great work, Thank you Kalyani & Tom.

2 quick questions:

1.  I presume SR inline is just SRH with 2 SIDs as mentioned - didn't see the topology used. Do intermediate nodes handle these SIDs, with pointer update in SRH?
2.  Also for Geneve  - it's IP4 encap and VNI no TLVs?

--
Uma C.

-----Original Message-----
From: dmm [mailto:[email-masked]] On Behalf Of Bogineni, Kalyani
Sent: Friday, March 16, 2018 11:16 AM
To: dmm <[email-masked]>
Subject: [DMM] some test results of different network overlay methods

All:

Here are some raw performance test results based on our understanding of the different network overlay methods. We welcome discussion and comments.

Kalyani and Tom

---

From: Tom Herbert <[email-masked]>
Date: Fri, 16 Mar 2018 12:57:51 -0700

> 1.  I presume SR inline is just SRH with 2 SIDs as mentioned - didn't see the topology used. Do intermediate nodes handle these SIDs, with pointer update in SRH?

Two hosts connected back to back. SR processing done by end host.

> 2.  Also for Geneve  - it's IP4 encap and VNI no TLVs?
No TLVs. GUE uses RCO extension. Other than that all the variants
should be out of the box with no options set. Encapsulations are
v4/v4, SRv6 and ILA are all IPv6.

I'll post the all the configuration scripts to github once I have some time.

Tom

> _______________________________________________
> dmm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dmm

---

From: "Pablo Camarillo (pcamaril)" <[email-masked]>
Date: Sun, 18 Mar 2018 13:39:58 +0000

Hi Tom, Kalyani,

If the SR processing of the two SIDs is done by the end host, then I believe the results for SR are obviously not valid.

Also in all the hardware-based routing platforms for which we have done interop https://tools.ietf.org/html/draft-filsfils-spring-srv6-interop-00#section-2, we have SRv6 running at linecard rate.

Note that I find this performance testing work interesting, and I’m willing to help you carry them. However, for next releases I would also consider leveraging projects like linux foundation fdio CSIT https://wiki.fd.io/view/CSIT/Description, in order to have a more methodical and accurate results (ILA has higher throughput than native IPv6 with a lower TPS).

Thanks.

Cheers,

Pablo.

________________________________
De:Tom Herbert <[email-masked]>
Enviado:viernes, 16 de marzo de 2018 7:58 p. m.
Para:Uma Chunduri
Cc: dmm
Asunto: Re: [DMM] some test results of different network overlay methods

Tom

---

From: Tom Herbert <[email-masked]>
Date: Sun, 18 Mar 2018 16:28:01 +0000

On Sun, Mar 18, 2018, 1:40 PM Pablo Camarillo (pcamaril) <[email-masked]>
wrote:

> Hi Tom, Kalyani,
>
>
>
> If the SR processing of the two SIDs is done by the end host, then I
> believe the results for SR are obviously not valid.
>

Pablo,

Why isn't this valid?

>
>
> Also in all the hardware-based routing platforms for which we have done
> interop
> https://tools.ietf.org/html/draft-filsfils-spring-srv6-interop-00#section-2,
> we have SRv6 running at linecard rate.
>
>
>
> Note that I find this performance testing work interesting, and I’m
> willing to help you carry them. However, for next releases I would also
> consider leveraging projects like linux foundation fdio CSIT
> https://wiki.fd.io/view/CSIT/Description, in order to have a more
> methodical and accurate results (ILA has higher throughput than native IPv6
> with a lower TPS).
>
>
>
To be clear, this is testing a real TCP stack and host terminating
encapsulation (not just router performance). For this purpose the tests are
methodical and accurate. The big difference in performance have a lot to do
how well NICs deal with different encapsulations and EH in case of SR. The
most interesting result, I think, is IPIP (and SR/IPIP) with drop in TPS.
This indicates that device (in this case ixgbe) isn't getting 5-tuple hash
for RSS. It's not parsing over headers to find ports for hash.
Interestingly, it was able to parse into L4 when just SR header is present.
Capabilities for things like this will vary between NICs.

Tom

> Thanks.
>
>
>
> Cheers,
>
> Pablo.
>
> ------------------------------
> *De:*Tom Herbert <[email-masked]>
> *Enviado:*viernes, 16 de marzo de 2018 7:58 p. m.
> *Para:*Uma Chunduri
> *Cc:* dmm
> *Asunto:* Re: [DMM] some test results of different network overlay methods
>
> > 1.  I presume SR inline is just SRH with 2 SIDs as mentioned - didn't
> see the topology used. Do intermediate nodes handle these SIDs, with
> pointer update in SRH?
>
> > From: dmm [mailto:[email-masked] <[email-masked]>] On
> Behalf Of Bogineni, Kalyani
> > Here are some raw performance test results based on our understanding of
> the different network overlay methods. We welcome discussion and comments.
> >
> > Kalyani and Tom

---

From: "Pablo Camarillo (pcamaril)" <[email-masked]>
Date: Tue, 20 Mar 2018 13:26:00 +0000

Hi Tom,

Inline.

Cheers,
Pablo.
From: Tom Herbert <[email-masked]>
Date: Sunday, 18 March 2018 at 16:28
To: "Pablo Camarillo (pcamaril)" <[email-masked]>
Cc: dmm <[email-masked]>, Uma Chunduri <[email-masked]>
Subject: Re: [DMM] some test results of different network overlay methods

On Sun, Mar 18, 2018, 1:40 PM Pablo Camarillo (pcamaril) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Tom, Kalyani,

Pablo,

Why isn't this valid?

What is the use case of an end linux host performing decapsulation or End SID processing?  By the time a packet reaches a Linux host terminating TCP it will contain only a single IP destination address == the host terminating the TCP session.

I expect that the only valuable use case to measure is encapsulation or insertion, see the  performance results https://irtf.org/anrw/2017/anrw17-final3.pdf for insert and encap on Linux.

If the end host is processing two SIDs, it means that you are doing twice SR processing. You will have an extra ‘End’ function processing, which involves updating the DA to the next segment, doing an IP lookup and forwarding.
Then you will restart and do again the processing. Overall, you do twice SR processing.

For example if I send  10 sids to a destination, all of which terminate on that destination, it will have to do SR END function processing 10 times.  This is true but there is no practical reason to do this.

In practice when host A sends a packet with 2 SIDs, one will be to host B and the next to host C.  Only B will do END function processing, C will simply receive the packet.  Therefore while your test is accurate it has no practical use.

To be clear, this is testing a real TCP stack and host terminating encapsulation (not just router performance). For this purpose the tests are methodical and accurate.
See above

The big difference in performance have a lot to do how well NICs deal with different encapsulations and EH in case of SR. The most interesting result, I think, is IPIP (and SR/IPIP) with drop in TPS. This indicates that device (in this case ixgbe) isn't getting 5-tuple hash for RSS. It's not parsing over headers to find ports for hash. Interestingly, it was able to parse into L4 when just SR header is present. Capabilities for things like this will vary between NICs.

Indeed, this is very interesting. Thanks for pointing it out. I believe NICs providers will eventually fix this.

Tom

Thanks.

Cheers,

Pablo.

________________________________
De:Tom Herbert <[email-masked]<mailto:[email-masked]>>
Enviado:viernes, 16 de marzo de 2018 7:58 p. m.
Para:Uma Chunduri
Cc: dmm
Asunto: Re: [DMM] some test results of different network overlay methods

Tom

> To: dmm <[email-masked]<mailto:[email-masked]>>
> [email-masked]<mailto:[email-masked]>

_______________________________________________
dmm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/dmm