Subject: Packet Number Encryption Performance

From: Nick Banks <[email-masked]>
Date: Thu, 21 Jun 2018 20:48:00 +0000

Hello QUIC WG,

I recently implemented PNE for WinQuic (using bcrypt APIs) and I decided to get some performance numbers to see what the overhead of PNE was. I figured the rest of the WG might be interested.

My test just encrypts the same buffer (size dependent on the test case) 10,000,000 times and measured the time it took. The test then did the same thing, but also encrypted the packet number as well. I ran all that 10 times in total. I then collected the best times for each category to produce the following graphs and tables (full excel doc attached):

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

I used a server grade lab machine I had at my disposal, running the latest Windows 10 Server DataCenter build. Again, these numbers are for crypto only. No QUIC or UDP is included.

Thanks,
- Nick

---

From: "Deval, Manasi" <[email-masked]>
Date: Thu, 21 Jun 2018 23:25:19 +0000

The 10% increase on server may be quite significant on a client machine. This may be an issue for mobile devices.

Manasi

From: QUIC [mailto:[email-masked]] On Behalf Of Nick Banks
Sent: Thursday, June 21, 2018 1:48 PM
To: [email-masked]
Subject: Packet Number Encryption Performance

Hello QUIC WG,

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 21 Jun 2018 17:55:44 -0700

Thanks for sharing these numbers, Nick! A few questions:
1. It's quite possibly I'm missing something basic, but: Assuming the same
number of packets (10M in your experiments), why do smaller packets result
in lower rates? Shouldn't encrypting the same number of smaller packets
result in a higher rate?
2. PNE seems to add about 700-800ms of latency, which understandably
decreases as % overhead as the packet size increases. But the difference in
rates seems to increase as the packet size increases, which is exactly
counter-intuitive. How do you compute rate?
3. What's the PN size used?

On Thu, Jun 21, 2018 at 4:25 PM Deval, Manasi <[email-masked]>
wrote:

> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Nick Banks
> *Sent:* Thursday, June 21, 2018 1:48 PM
> *To:* [email-masked]
> *Subject:* Packet Number Encryption Performance
>
>
>
> Hello QUIC WG,
>
>
>
> I recently implemented PNE for WinQuic (using bcrypt APIs) and I decided
> to get some performance numbers to see what the overhead of PNE was. I
> figured the rest of the WG might be interested.
>
>
>
> My test just encrypts the same buffer (size dependent on the test case)
> 10,000,000 times and measured the time it took. The test then did the same
> thing, but also encrypted the packet number as well. I ran all that 10
> times in total. I then collected the best times for each category to
> produce the following graphs and tables (full excel doc attached):
>
>
>
>
>
> *Time (ms)*
>
> *Rate (Mbps)*
>
> *Bytes*
>
> *NO PNE*
>
> *PNE*
>
> *PNE Overhead*
>
> *No PNE*
>
> *PNE*
>
> *4*
>
> 2284.671
>
> 3027.657
>
> 33%
>
> 140.064
>
> 105.692
>
> *16*
>
> 2102.402
>
> 2828.204
>
> 35%
>
> 608.827
>
> 452.584
>
> *64*
>
> 2198.883
>
> 2907.577
>
> 32%
>
> 2328.45
>
> 1760.92
>
> *256*
>
> 2758.3
>
> 3490.28
>
> 27%
>
> 7424.86
>
> 5867.72
>
> *600*
>
> 4669.283
>
> 5424.539
>
> 16%
>
> 10280
>
> 8848.68
>
> *1000*
>
> 6130.139
>
> 6907.805
>
> 13%
>
> 13050.3
>
> 11581.1
>
> *1200*
>
> 6458.679
>
> 7229.672
>
> 12%
>
> 14863.7
>
> 13278.6
>
> *1450*
>
> 7876.312
>
> 8670.16
>
> 10%
>
> 14727.7
>
> 13379.2
>
>
>
> I used a server grade lab machine I had at my disposal, running the latest
> Windows 10 Server DataCenter build. Again, these numbers are for crypto
> only. No QUIC or UDP is included.
>
>
>
> Thanks,
>
> - Nick
>
>
>

---

From: Nick Banks <[email-masked]>
Date: Fri, 22 Jun 2018 01:21:36 +0000

I used a 4 byte packet number. I computed rate simply by dividing the total number of bits encrypted (10M * 8 * size) by the total time to encrypt. It’s all in the Excel doc.

As to why the rate increases with size of the packet, that is because the per byte overhead is being reduced both in terms of function call overhead and crypto cpu cycles. Finally, why the PNE cost seems to change, I don’t know.

Sent from my Windows 10 phone
[HxS - 15254 - 16.0.10228.20075]

________________________________
From: Jana Iyengar <[email-masked]>
Sent: Thursday, June 21, 2018 5:55:44 PM
To: Deval, Manasi
Cc: Nick Banks; QUIC WG
Subject: Re: Packet Number Encryption Performance

Thanks for sharing these numbers, Nick! A few questions:
1. It's quite possibly I'm missing something basic, but: Assuming the same number of packets (10M in your experiments), why do smaller packets result in lower rates? Shouldn't encrypting the same number of smaller packets result in a higher rate?
2. PNE seems to add about 700-800ms of latency, which understandably decreases as % overhead as the packet size increases. But the difference in rates seems to increase as the packet size increases, which is exactly counter-intuitive. How do you compute rate?
3. What's the PN size used?

On Thu, Jun 21, 2018 at 4:25 PM Deval, Manasi <[email-masked]<mailto:[email-masked]>> wrote:
The 10% increase on server may be quite significant on a client machine. This may be an issue for mobile devices.

Manasi

From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Nick Banks
Sent: Thursday, June 21, 2018 1:48 PM
To: [email-masked]<mailto:[email-masked]>
Subject: Packet Number Encryption Performance

Hello QUIC WG,

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

---

From: Jana Iyengar <[email-masked]>
Date: Thu, 21 Jun 2018 18:46:46 -0700

On Thu, Jun 21, 2018 at 6:21 PM Nick Banks <[email-masked]> wrote:

> I used a 4 byte packet number. I computed rate simply by dividing the
> total number of bits encrypted (10M * 8 * size) by the total time to
> encrypt. It’s all in the Excel doc.
>

That can't be right -- how do you get a 4-byte packet size then?
The rate computation helps. Thank you.

Basically the % overhead is correct; the graph looks like the difference
increases because the range of the Y is quite large (2 orders of
magnitude). A log scale is slightly better. But plotting the PNE overhead
normalizes the range, and is probably visually clearer.

As to why the rate increases with size of the packet, that is because the
> per byte overhead is being reduced both in terms of function call overhead
> and crypto cpu cycles. Finally, why the PNE cost seems to change, I don’t
> know.
>

The PNE cost doesn't seem to change much, the relative cost does as your %
overhead numbers show.

A thought: if you're encrypting the same buffer 10M times, are you
basically escaping read/write costs, since I presume both buffers are
always is in the cache. This might give us a skewed picture... if it's not
too much work, can you change a byte in the data and increment/decrement
the PN between runs, so that the encryption itself happens on new data on
each run?

The cost does seem high, but this is relative to the cost of crypto itself
as part of total transmission cost. I expect PNE to be visible even when
looking at total transmission cost, but I expect it to be much lower than
the 10% that shows up in your microbenchmark.

Thanks again for sharing these results!

> ------------------------------
> *From:* Jana Iyengar <[email-masked]>
> *Sent:* Thursday, June 21, 2018 5:55:44 PM
> *To:* Deval, Manasi
> *Cc:* Nick Banks; QUIC WG
> *Subject:* Re: Packet Number Encryption Performance
>
>> I used a server grade lab machine I had at my disposal, running the
>> latest Windows 10 Server DataCenter build. Again, these numbers are for
>> crypto only. No QUIC or UDP is included.
>>
>>
>>
>> Thanks,
>>
>> - Nick
>>
>>
>>
>

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 22 Jun 2018 11:21:17 +0900

Hi Nick,

Thank you for bringing the numbers to the list.

I have just run a small benchmark using Quicly, and I see comparable
numbers.

To be precise, I see 10.0% increase of CPU cycles when encrypting a Initial
packet of 1,280 octets. I expect that we will see similar numbers on other
QUIC stacks that also use picotls (with OpenSSL as a backend). Note that
the number is only comparing the cost of encryption, the overhead ratio
will be much smaller if we look at the total number of CPU cycles spent by
a QUIC stack as a whole.

Looking at the profile, the overhead consists of three operations that each
consumes comparable CPU cycles: core AES operation (using AES-NI), CTR
operation overhead, CTR initialization. Note that picotls at the moment
provides access to CTR crypto beneath the AEAD interface, which is to be
used by the QUIC stacks.

I would assume that we can cut down the overhead to somewhere between 2% to
4%, but it might be hard to go down to somewhere near 1%, because we cannot
parallelize the AES operation of PNE with that of AEAD (see
https://github.com/openssl/openssl/blob/OpenSSL_1_1_0h/crypto/aes/asm/aesni-x86_64.pl#L24-L39
about the impact of parallelization).

I do not think that 2% to 4% of additional overhead to the crypto is an
issue for QUIC/HTTP, but current overhead of 10% is something that we might
want to decrease. I am glad to be able to learn that now.

2018-06-22 5:48 GMT+09:00 Nick Banks <nibanks=[email-masked]
>:

-- 
Kazuho Oku

---

From: Nick Banks <[email-masked]>
Date: Fri, 22 Jun 2018 02:54:31 +0000

Hi Kazuho,

Thanks for sharing your numbers as well! I'm bit confused where you say you can reduce the 10% overhead to 2% to 4%. How do you plan on doing that?

________________________________
From: Kazuho Oku <[email-masked]>
Sent: Thursday, June 21, 2018 7:21:17 PM
To: Nick Banks
Cc: [email-masked]
Subject: Re: Packet Number Encryption Performance

Hi Nick,

I have just run a small benchmark using Quicly, and I see comparable numbers.

To be precise, I see 10.0% increase of CPU cycles when encrypting a Initial packet of 1,280 octets. I expect that we will see similar numbers on other QUIC stacks that also use picotls (with OpenSSL as a backend). Note that the number is only comparing the cost of encryption, the overhead ratio will be much smaller if we look at the total number of CPU cycles spent by a QUIC stack as a whole.

Looking at the profile, the overhead consists of three operations that each consumes comparable CPU cycles: core AES operation (using AES-NI), CTR operation overhead, CTR initialization. Note that picotls at the moment provides access to CTR crypto beneath the AEAD interface, which is to be used by the QUIC stacks.

I would assume that we can cut down the overhead to somewhere between 2% to 4%, but it might be hard to go down to somewhere near 1%, because we cannot parallelize the AES operation of PNE with that of AEAD (see https://github.com/openssl/openssl/blob/OpenSSL_1_1_0h/crypto/aes/asm/aesni-x86_64.pl#L24-L39<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenssl%2Fopenssl%2Fblob%2FOpenSSL_1_1_0h%2Fcrypto%2Faes%2Fasm%2Faesni-x86_64.pl%23L24-L39&data=02%7C01%7Cnibanks%40microsoft.com%7C11d55f17333e4a795d7008d5d7e6d93c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636652308843994134&sdata=kqMz4SsN%2F2ErGK06Qz8Z0vUzpl4MiipnNE2wAMUb46c%3D&reserved=0> about the impact of parallelization).

I do not think that 2% to 4% of additional overhead to the crypto is an issue for QUIC/HTTP, but current overhead of 10% is something that we might want to decrease. I am glad to be able to learn that now.

2018-06-22 5:48 GMT+09:00 Nick Banks <nibanks=[email-masked]<mailto:nibanks=[email-masked]>>:
Hello QUIC WG,

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

--
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 22 Jun 2018 12:22:39 +0900

2018-06-22 11:54 GMT+09:00 Nick Banks <[email-masked]>:

> Hi Kazuho,
>
>
>
> Thanks for sharing your numbers as well! I'm bit confused where you say
> you can reduce the 10% overhead to 2% to 4%. How do you plan on doing that?
>

As stated in my previous mail, the 10% of overhead consists of three parts,
each consuming comparable number of CPU cycles. The two among the three is
related to the abstraction layer and how CTR is implemented, while the
other one is the core AES-ECB operation cost.

It should be able to remove the costly abstraction layer.

It should also be possible to remove the overhead of CTR, since in PNE, we
need to XOR at most 4 octets (applying XOR is the only difference between
CTR and ECB). That cost should be something that should be possible to be
nullified.

Considering these aspects, and by looking at the numbers on the OpenSSL
source code (as well as considering the overhead of GCM), my expectation
goes to 2% to 4%.

> *From:* Kazuho Oku <[email-masked]>
> *Sent:* Thursday, June 21, 2018 7:21:17 PM
> *To:* Nick Banks
> *Cc:* [email-masked]
> To be precise, I see 10.0% increase of CPU cycles when encrypting a
> Initial packet of 1,280 octets. I expect that we will see similar numbers
> on other QUIC stacks that also use picotls (with OpenSSL as a backend).
> Note that the number is only comparing the cost of encryption, the overhead
> ratio will be much smaller if we look at the total number of CPU cycles
> spent by a QUIC stack as a whole.
>
> Looking at the profile, the overhead consists of three operations that
> each consumes comparable CPU cycles: core AES operation (using AES-NI), CTR
> I would assume that we can cut down the overhead to somewhere between 2%
> to 4%, but it might be hard to go down to somewhere near 1%, because we
> cannot parallelize the AES operation of PNE with that of AEAD (see
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenssl%2Fopenssl%2Fblob%2FOpenSSL_1_1_0h%2Fcrypto%2Faes%2Fasm%2Faesni-x86_64.pl%23L24-L39&data=02%7C01%7Cnibanks%40microsoft.com%7C11d55f17333e4a795d7008d5d7e6d93c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636652308843994134&sdata=kqMz4SsN%2F2ErGK06Qz8Z0vUzpl4MiipnNE2wAMUb46c%3D&reserved=0>
> 2018-06-22 5:48 GMT+09:00 Nick Banks <nibanks=40microsoft.com@
> dmarc.ietf.org>:
>
>> Hello QUIC WG,
>>
>>
>>

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 21 Jun 2018 22:17:08 -0700

There might be an additional hidden overhead at large scale if staged PNE
causes the packet to pulled in and out of L1 cache several times either
send or receive side or by creating additional write buffer pressure by
touching an inbound packet, notably on NUMA architectures. This effect can
have more overhead than encryption itself if it happens.

On 22 June 2018 at 05.22.56, Kazuho Oku ([email-masked]) wrote:

--
Kazuho Oku

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 22 Jun 2018 07:57:45 +0200

Hi Kazuho,

> source code (as well as considering the overhead of GCM), my expectation

Well, probably that we don't need the 10 full AES rounds and could
have enough of 2 like some of the SHA3 proposals did, or even maybe
just one. These ones take a few tens of CPU cycles on modern CPUs,
possibly less, and are probably more than enough to prevent the
linkability people are concerned with.

Just my two cents.
Willy

---

From: Kazuho Oku <[email-masked]>
Date: Fri, 22 Jun 2018 15:44:46 +0900

2018-06-22 12:22 GMT+09:00 Kazuho Oku <[email-masked]>:

> As stated in my previous mail, the 10% of overhead consists of three
> parts, each consuming comparable number of CPU cycles. The two among the
> three is related to the abstraction layer and how CTR is implemented, while
> the other one is the core AES-ECB operation cost.
>

Just did some experiments and it seems that the expectation was correct.

The benchmarks tell me that the overhead goes down from 10.0% to 3.8%, by
doing the following:

* remove the overhead of CTR abstraction (i.e. use the ECB backend and do
XOR by ourselves)
* remove the overhead of the abstraction layer (i.e. call the method
returned by EVP_CIPHER_meth_get_do_cipher instead of calling
EVP_EncryptUpdate)

Of course the changes are specific to OpenSSL, but I would expect that you
can expect similar numbers assuming that you have access to an optimized
AES implementation.

>> 2018-06-22 5:48 GMT+09:00 Nick Banks <nibanks=40microsoft.com@dmarc
>> .ietf.org>:
>>
>>> Hello QUIC WG,
>>>
>>>
>>>

-- 
Kazuho Oku

---

From: Ian Swett <[email-masked]>
Date: Fri, 22 Jun 2018 07:45:20 -0400

Thanks for digging into the details of this, Kazuho.  <4% increase in
crypto cost is a bit more than I originally expected(~2%), but crypto is
less than 10% of my CPU usage, so it's still less than 0.5% total, which is
acceptable to me.

On Fri, Jun 22, 2018 at 2:45 AM Kazuho Oku <[email-masked]> wrote:

>> It should also be possible to remove the overhead of CTR, since in PNE,
>> we need to XOR at most 4 octets (applying XOR is the only difference
>> between CTR and ECB). That cost should be something that should be possible
>> to be nullified.
>>
>>> 2018-06-22 5:48 GMT+09:00 Nick Banks <
>>> nibanks=[email-masked]>:
>>>
>>>> Hello QUIC WG,
>>>>
>>>>
>>>>
>>>> I recently implemented PNE for WinQuic (using bcrypt APIs) and I
>>>> decided to get some performance numbers to see what the overhead of PNE
>>>> was. I figured the rest of the WG might be interested.
>>>>
>>>>
>>>>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 22 Jun 2018 16:26:44 +0000

Ian, do you expect that fraction of overall cost to hold once the UDP stack is optimized? Is your measurement on top of the recent kernel improvements? I expect crypto fraction of overall cost to keep increasing over time as the network stack bottlenecks are eliminated.

Kazuho, should the draft describe the optimizations you are making? Or are these are too OpenSSL specific?

From: QUIC [mailto:[email-masked]] On Behalf Of Ian Swett
Sent: Friday, June 22, 2018 4:45 AM
To: Kazuho Oku <[email-masked]>
Cc: Nick Banks <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Packet Number Encryption Performance

Thanks for digging into the details of this, Kazuho.  <4% increase in crypto cost is a bit more than I originally expected(~2%), but crypto is less than 10% of my CPU usage, so it's still less than 0.5% total, which is acceptable to me.

On Fri, Jun 22, 2018 at 2:45 AM Kazuho Oku <[email-masked]<mailto:[email-masked]>> wrote:

2018-06-22 12:22 GMT+09:00 Kazuho Oku <[email-masked]<mailto:[email-masked]>>:

2018-06-22 11:54 GMT+09:00 Nick Banks <[email-masked]<mailto:[email-masked]>>:
Hi Kazuho,

As stated in my previous mail, the 10% of overhead consists of three parts, each consuming comparable number of CPU cycles. The two among the three is related to the abstraction layer and how CTR is implemented, while the other one is the core AES-ECB operation cost.

It should also be possible to remove the overhead of CTR, since in PNE, we need to XOR at most 4 octets (applying XOR is the only difference between CTR and ECB). That cost should be something that should be possible to be nullified.

Considering these aspects, and by looking at the numbers on the OpenSSL source code (as well as considering the overhead of GCM), my expectation goes to 2% to 4%.

The benchmarks tell me that the overhead goes down from 10.0% to 3.8%, by doing the following:

* remove the overhead of CTR abstraction (i.e. use the ECB backend and do XOR by ourselves)
* remove the overhead of the abstraction layer (i.e. call the method returned by EVP_CIPHER_meth_get_do_cipher instead of calling EVP_EncryptUpdate)

Of course the changes are specific to OpenSSL, but I would expect that you can expect similar numbers assuming that you have access to an optimized AES implementation.

________________________________
From: Kazuho Oku <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, June 21, 2018 7:21:17 PM
To: Nick Banks
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: Packet Number Encryption Performance

Hi Nick,

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

---

From: Nick Banks <[email-masked]>
Date: Fri, 22 Jun 2018 16:34:38 +0000

I just want to add, my implementation already uses ECB from bcrypt (and I do the XOR) already. Bcrypt doesn’t expose CTR mode directly.

Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

________________________________
From: Praveen Balasubramanian
Sent: Friday, June 22, 2018 9:26:44 AM
To: Ian Swett; Kazuho Oku
Cc: Nick Banks; IETF QUIC WG
Subject: RE: Packet Number Encryption Performance

Hi Nick,

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

---

From: Ian Swett <[email-masked]>
Date: Fri, 22 Jun 2018 14:51:38 -0400

I expect crypto to increase as a fraction of CPU, but I don't expect it to
go much higher than 10%.

But who knows, maybe 2 years from now everything else will be very
optimized and crypto will be 15%?

On Fri, Jun 22, 2018 at 12:34 PM Nick Banks <nibanks=
[email-masked]> wrote:

> I just want to add, my implementation already uses ECB from bcrypt (and I
> do the XOR) already. Bcrypt doesn’t expose CTR mode directly.
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows 10
>
>
> *From:* Praveen Balasubramanian
> *Sent:* Friday, June 22, 2018 9:26:44 AM
> *To:* Ian Swett; Kazuho Oku
> *Cc:* Nick Banks; IETF QUIC WG
> *Subject:* RE: Packet Number Encryption Performance
>
>
> Ian, do you expect that fraction of overall cost to hold once the UDP
> stack is optimized? Is your measurement on top of the recent kernel
> improvements? I expect crypto fraction of overall cost to keep increasing
> over time as the network stack bottlenecks are eliminated.
>
>
>
> *From:* QUIC [mailto:[email-masked]] *On Behalf Of *Ian Swett
> *Sent:* Friday, June 22, 2018 4:45 AM
> *To:* Kazuho Oku <[email-masked]>
> *Cc:* Nick Banks <[email-masked]>; IETF QUIC WG <[email-masked]>

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 23 Jun 2018 09:01:35 +0900

> optimized and crypto will be 15%?

Ian, thank you for sharing the proportion of CPU cycles we are likely to
spend for crypto.

Your numbers relieves me, because even if the cost of crypto goes to 15%,
the overhead of PNE will be less than 1% (0.15*0.04=0.006).

I would also like to note that it is likely that HyperThreading, when used,
will eliminate the overhead of PNE.

This is because IIUC PNE is a marginal additional use of the AES-NI engine,
which have been mostly idle. The overhead of crypto is small (i.e. 15%)
that we will rarely see contention on the engine. While one hyperthread
does AES, the other hyperthread will run at full speed doing other
operations.

Also considering the fact that the number of CPU cycles spent per QUIC
packet does not change a lot with PNE, I would not be surprised to see *no*
decrease of throughput when PNE is used on a HyperThreading architecture.
In such case, what we will only observe is the raise of the utilization
ratio of the AES-NI engine.

>
> On Fri, Jun 22, 2018 at 12:34 PM Nick Banks <nibanks=40microsoft.com@
> dmarc.ietf.org> wrote:
>
>> Of course the changes are specific to OpenSSL, but I would expect that
>> you can expect similar numbers assuming that you have access to an
>> optimized AES implementation.
>>
>>
>>
>>
>>
>>
>>

-- 
Kazuho Oku

---

From: Nick Banks <[email-masked]>
Date: Sat, 23 Jun 2018 00:34:07 +0000

Hey Guys,

I spent the better part of the day getting some performance traces of WinQuic without PNE. Our implementation supports both user mode and kernel mode, so I got numbers for both. The following table shows the relative CPU cost of different parts of the QUIC send path:

User Mode

Kernel Mode

UDP Send

64.7%

41.4%

Encryption

22.5%

38.5%

Packet Building/Framing

7%

15%

Miscellaneous

5.8%

5.1%

These numbers definitely show that encryption is a much larger portion of CPU.

- Nick

From: Kazuho Oku <[email-masked]>
Sent: Friday, June 22, 2018 5:02 PM
To: Ian Swett <[email-masked]>
Cc: Nick Banks <[email-masked]>; Praveen Balasubramanian <[email-masked]>; IETF QUIC WG <[email-masked]>
Subject: Re: Packet Number Encryption Performance

2018-06-23 3:51 GMT+09:00 Ian Swett <[email-masked]<mailto:[email-masked]>>:
I expect crypto to increase as a fraction of CPU, but I don't expect it to go much higher than 10%.

But who knows, maybe 2 years from now everything else will be very optimized and crypto will be 15%?

Ian, thank you for sharing the proportion of CPU cycles we are likely to spend for crypto.

Your numbers relieves me, because even if the cost of crypto goes to 15%, the overhead of PNE will be less than 1% (0.15*0.04=0.006).

I would also like to note that it is likely that HyperThreading, when used, will eliminate the overhead of PNE.

This is because IIUC PNE is a marginal additional use of the AES-NI engine, which have been mostly idle. The overhead of crypto is small (i.e. 15%) that we will rarely see contention on the engine. While one hyperthread does AES, the other hyperthread will run at full speed doing other operations.

Also considering the fact that the number of CPU cycles spent per QUIC packet does not change a lot with PNE, I would not be surprised to see *no* decrease of throughput when PNE is used on a HyperThreading architecture. In such case, what we will only observe is the raise of the utilization ratio of the AES-NI engine.

On Fri, Jun 22, 2018 at 12:34 PM Nick Banks <nibanks=[email-masked]<mailto:[email-masked]>> wrote:
I just want to add, my implementation already uses ECB from bcrypt (and I do the XOR) already. Bcrypt doesn’t expose CTR mode directly.

Sent from Mail<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986&data=02%7C01%7Cnibanks%40microsoft.com%7Ce6efa363bc2547167a8c08d5d89c80e7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636653089059268836&sdata=aeU7j5lhCIM18KG5c6opG4p7amwhCWbRnfnm8sDW7hc%3D&reserved=0> for Windows 10

From: QUIC [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Ian Swett
Sent: Friday, June 22, 2018 4:45 AM
To: Kazuho Oku <[email-masked]<mailto:[email-masked]>>
Cc: Nick Banks <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Packet Number Encryption Performance

Hi Nick,

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Sat, 23 Jun 2018 16:00:09 +0900

IIUC, Ian should be talking about performance numbers that you would see on
a server that is handling real workload. Not the breakdown of numbers
within the QUIC stack.

Let's talk about nanoseconds per packet rather than the ratio, because
ratio depends on what you use as a "total" is.

My benchmarks tell me that, on OpenSSL 1.1.0 running on Core i7 @ 2.5GHz
(MacBook Pro 15" Mid 2015), the numbers are:

without-PNE: 585ns / packet
with-PNE: 607ns / packet (* this is the optimized version with 3.8%
overhead)

Assuming that your peak rate is 1Gbit/CPU core (would be enough to saturate
25Gb Ethernet), the ratio of CPU cycles for running the crypto will be:

all crypto: 0.125GB/sec / 1280bytes/packet * 607ns/packet = 5.93 %
PNE: 0.125GB/sec / 1280bytes/packet * 22.4ns/packet = 0.22 %

Of course, these numbers are what is expected when a server is sending or
receiving full-sized packets in one direction. In reality, there would be
at least some flow in the opposite direction, so the number will be higher,
but not as much as 2x. Or if you are sending small-sized packet in *both*
directions *and* saturating the link, the ratio will be higher.

But looking at the numbers, I'd assume that 10% is a logical number for
workload Ian has, and also for people who are involved in content
distribution.

As shown, I think that sharing the PPS and average packet size that our
workload will have, along with the raw numbers of the crypto (i.e. nsec /
packet) will give us a better understanding on how much the actual overhead
is. Do you mind sharing your expectations?

2018-06-23 9:34 GMT+09:00 Nick Banks <[email-masked]>:

> Hey Guys,
>
>
>
> I spent the better part of the day getting some performance traces of
> WinQuic *without PNE*. Our implementation supports both user mode and
> kernel mode, so I got numbers for both. The following table shows the
> relative CPU cost of different parts of the QUIC send path:
>
>
>
>
>
> *User Mode*
>
> *Kernel Mode*
>
> *UDP Send*
>
> 64.7%
>
> 41.4%
>
> *Encryption*
>
> 22.5%
>
> 38.5%
>
> *Packet Building/Framing*
>
> 7%
>
> 15%
>
> *Miscellaneous*
>
> 5.8%
>
> 5.1%
>
>
>
> *Sent:* Friday, June 22, 2018 5:02 PM
> *To:* Ian Swett <[email-masked]>
> *Cc:* Nick Banks <[email-masked]>; Praveen Balasubramanian <
> [email-masked]>; IETF QUIC WG <[email-masked]>
>
> I would also like to note that it is likely that HyperThreading, when
> used, will eliminate the overhead of PNE.
>
>
>
> This is because IIUC PNE is a marginal additional use of the AES-NI
> engine, which have been mostly idle. The overhead of crypto is small (i.e.
> 15%) that we will rarely see contention on the engine. While one
> hyperthread does AES, the other hyperthread will run at full speed doing
> other operations.
>
>
>
> Sent from Mail
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgo.microsoft.com%2Ffwlink%2F%3FLinkId%3D550986&data=02%7C01%7Cnibanks%40microsoft.com%7Ce6efa363bc2547167a8c08d5d89c80e7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636653089059268836&sdata=aeU7j5lhCIM18KG5c6opG4p7amwhCWbRnfnm8sDW7hc%3D&reserved=0>
> for Windows 10
>
>

-- 
Kazuho Oku

---

From: Nick Banks <[email-masked]>
Date: Sat, 23 Jun 2018 14:54:15 +0000

I agree that the numbers for real work loads will be interesting, but I do believe that raw QUIC stack numbers are also important. They will represent your best case scenario.

For my testing, I have a fairly simple test of opening a single unidirectional stream and just continually sending data until the core gets saturated. From that, I grab the performance trace, and specifically look at the composition of the QUIC stack’s send path. The send path constitutes generally all of the CPU for the whole connection anyways (with just a little overhead seen from receiving and processing ACKs). The machine I used was another server grade machine running the latest Windows 10 Server Datacenter.

So the percentages I shared are for just the QUIC send path (again, no PNE with these numbers). The numbers are for the percentage of total CPU for all packets sent, but since all packets were pretty much the same size, the numbers should still hold for a single packet. And bottom line, encryption is a lot more of an impact than 10%. And as we bring in more performance improvements for UDP send path in Windows, we expect encryption to be higher and higher percentage.

- Nick

From: Kazuho Oku <[email-masked]>
Sent: Saturday, June 23, 2018 12:00 AM
To: Nick Banks <[email-masked]>
Cc: Ian Swett <[email-masked]>; IETF QUIC WG <[email-masked]>; Praveen Balasubramanian <[email-masked]>
Subject: Re: Packet Number Encryption Performance

IIUC, Ian should be talking about performance numbers that you would see on a server that is handling real workload. Not the breakdown of numbers within the QUIC stack.

Let's talk about nanoseconds per packet rather than the ratio, because ratio depends on what you use as a "total" is.

My benchmarks tell me that, on OpenSSL 1.1.0 running on Core i7 @ 2.5GHz (MacBook Pro 15" Mid 2015), the numbers are:

without-PNE: 585ns / packet
with-PNE: 607ns / packet (* this is the optimized version with 3.8% overhead)

Assuming that your peak rate is 1Gbit/CPU core (would be enough to saturate 25Gb Ethernet), the ratio of CPU cycles for running the crypto will be:

Of course, these numbers are what is expected when a server is sending or receiving full-sized packets in one direction. In reality, there would be at least some flow in the opposite direction, so the number will be higher, but not as much as 2x. Or if you are sending small-sized packet in *both* directions *and* saturating the link, the ratio will be higher.

But looking at the numbers, I'd assume that 10% is a logical number for workload Ian has, and also for people who are involved in content distribution.

As shown, I think that sharing the PPS and average packet size that our workload will have, along with the raw numbers of the crypto (i.e. nsec / packet) will give us a better understanding on how much the actual overhead is. Do you mind sharing your expectations?

2018-06-23 9:34 GMT+09:00 Nick Banks <[email-masked]<mailto:[email-masked]>>:
Hey Guys,

User Mode

Kernel Mode

UDP Send

64.7%

41.4%

Encryption

22.5%

38.5%

Packet Building/Framing

7%

15%

Miscellaneous

5.8%

5.1%

- Nick

From: Kazuho Oku <[email-masked]<mailto:[email-masked]>>
Sent: Friday, June 22, 2018 5:02 PM
To: Ian Swett <[email-masked]<mailto:[email-masked]>>
Cc: Nick Banks <[email-masked]<mailto:[email-masked]>>; Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>

Hi Nick,

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sat, 23 Jun 2018 12:39:24 -0400

Hi Nick, how about numbers for small packets? So I presume the overhead of
PNE will be much larger here because PNE does not pipeline while a 64 byte
payload just about manages til an AES pipeline depending on arch.

Mikkel

On 23 June 2018 at 16.54.42, Nick Banks (
nibanks=[email-masked]) wrote:

I agree that the numbers for real work loads will be interesting, but I do
believe that raw QUIC stack numbers are also important. They will represent
your best case scenario.

For my testing, I have a fairly simple test of opening a single
unidirectional stream and just continually sending data until the core gets
saturated. From that, I grab the performance trace, and specifically look
at the composition of the QUIC stack’s send path. The send path constitutes
generally all of the CPU for the whole connection anyways (with just a
little overhead seen from receiving and processing ACKs). The machine I
used was another server grade machine running the latest Windows 10 Server
Datacenter.

So the percentages I shared are for just the QUIC send path (again, no PNE
with these numbers). The numbers are for the percentage of total CPU for
all packets sent, but since all packets were pretty much the same size, the
numbers should still hold for a single packet. And bottom line, encryption
is a lot more of an impact than 10%. And as we bring in more performance
improvements for UDP send path in Windows, we expect encryption to be
higher and higher percentage.

- Nick

*From:* Kazuho Oku <[email-masked]>
*Sent:* Saturday, June 23, 2018 12:00 AM
*To:* Nick Banks <[email-masked]>
*Cc:* Ian Swett <[email-masked]>; IETF QUIC WG <[email-masked]>;
Praveen Balasubramanian <[email-masked]>
*Subject:* Re: Packet Number Encryption Performance

Hey Guys,

*User Mode*

*Kernel Mode*

*UDP Send*

64.7%

41.4%

*Encryption*

22.5%

38.5%

7%

15%

*Miscellaneous*

5.8%

5.1%

These numbers definitely show that encryption is a much larger portion of
CPU.

- Nick

Sent from Mail

Ian, do you expect that fraction of overall cost to hold once the UDP stack
is optimized? Is your measurement on top of the recent kernel improvements?
I expect crypto fraction of overall cost to keep increasing over time as
the network stack bottlenecks are eliminated.

Kazuho, should the draft describe the optimizations you are making? Or are
these are too OpenSSL specific?

Hi Kazuho,

Thanks for sharing your numbers as well! I'm bit confused where you say you
can reduce the 10% overhead to 2% to 4%. How do you plan on doing that?

Hi Nick,

Hello QUIC WG,

I recently implemented PNE for WinQuic (using bcrypt APIs) and I decided to
get some performance numbers to see what the overhead of PNE was. I figured
the rest of the WG might be interested.

*Time (ms)*

*Rate (Mbps)*

*Bytes*

*NO PNE*

*PNE*

*PNE Overhead*

*No PNE*

*PNE*

*4*

2284.671

3027.657

33%

140.064

105.692

*16*

2102.402

2828.204

35%

608.827

452.584

*64*

2198.883

2907.577

32%

2328.45

1760.92

*256*

2758.3

3490.28

27%

7424.86

5867.72

*600*

4669.283

5424.539

16%

10280

8848.68

*1000*

6130.139

6907.805

13%

13050.3

11581.1

*1200*

6458.679

7229.672

12%

14863.7

13278.6

*1450*

7876.312

8670.16

10%

14727.7

13379.2

Thanks,

- Nick

-- 

Kazuho Oku

-- 

Kazuho Oku

-- 

Kazuho Oku

-- 

Kazuho Oku

-- 

Kazuho Oku

---

From: Nick Banks <[email-masked]>
Date: Sat, 23 Jun 2018 16:49:20 +0000

I didn’t run any QUIC end-to-end measurements for sending small packets, so I cannot definitely say what the overhead is, percentage-wise, of encryption, but I do think it would be higher.

- Nick

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Sent: Saturday, June 23, 2018 9:39 AM
To: IETF QUIC WG <[email-masked]>; Nick Banks <[email-masked]>; Kazuho Oku <[email-masked]>
Cc: Ian Swett <[email-masked]>; Praveen Balasubramanian <[email-masked]>
Subject: RE: Packet Number Encryption Performance

Hi Nick, how about numbers for small packets? So I presume the overhead of PNE will be much larger here because PNE does not pipeline while a 64 byte payload just about manages til an AES pipeline depending on arch.

Mikkel

On 23 June 2018 at 16.54.42, Nick Banks (nibanks=[email-masked]<mailto:nibanks=[email-masked]>) wrote:
I agree that the numbers for real work loads will be interesting, but I do believe that raw QUIC stack numbers are also important. They will represent your best case scenario.

- Nick

From: Kazuho Oku <[email-masked]<mailto:[email-masked]>>
Sent: Saturday, June 23, 2018 12:00 AM
To: Nick Banks <[email-masked]<mailto:[email-masked]>>
Cc: Ian Swett <[email-masked]<mailto:[email-masked]>>; IETF QUIC WG <[email-masked]<mailto:[email-masked]>>; Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>>
Subject: Re: Packet Number Encryption Performance

User Mode

Kernel Mode

UDP Send

64.7%

41.4%

Encryption

22.5%

38.5%

Packet Building/Framing

7%

15%

Miscellaneous

5.8%

5.1%

- Nick

This is because IIUC PNE is a marginal additional use of the AES-NI engine, which have been mostly idle. The overhead of crypto is small (i.e.. 15%) that we will rarely see contention on the engine. While one hyperthread does AES, the other hyperthread will run at full speed doing other operations.

The benchmarks tell me that the overhead goes down from 10.0% to 3..8%, by doing the following:

Hi Nick,

I would assume that we can cut down the overhead to somewhere between 2% to 4%, but it might be hard to go down to somewhere near 1%, because we cannot parallelize the AES operation of PNE with that of AEAD (see https://github.com/openssl/openssl/blob/OpenSSL_1_1_0h/crypto/aes/asm/aesni-x86_64.pl#L24-L39<https://na01.safelinks.protection..outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenssl%2Fopenssl%2Fblob%2FOpenSSL_1_1_0h%2Fcrypto%2Faes%2Fasm%2Faesni-x86_64.pl%23L24-L39&data=02%7C01%7Cnibanks%40microsoft.com%7C11d55f17333e4a795d7008d5d7e6d93c%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636652308843994134&sdata=kqMz4SsN%2F2ErGK06Qz8Z0vUzpl4MiipnNE2wAMUb46c%3D&reserved=0> about the impact of parallelization).

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Sun, 24 Jun 2018 07:10:33 +0900

> believe that raw QUIC stack numbers are also important. They will represent

To clarify, the reason I asked about the raw numbers and the use-case is
because without that I do not know how to interpret your numbers.

My understanding is that your benchmark is about encrypting and sending
large QUIC packets.

Assuming that BCRYPT provides comparable performance to OpenSSL, the
numbers you show (i.e. 22.5% in user-mode, 38.5% in kernel-mode) means that
you want to reduce crypto cost when a single CPU core is emitting somewhere
between 5Gbps to 8 Gbps.

I wonder how much we would be interested in such case. Let me explain why.

Assuming that we have many connections, each CPU core will only handle
somewhere around 1Gbps to saturate a 25Gbps link. As I have stated in my
previous mail, the crypto cost will be around or less than 10% in such case.

Assuming that we are interested in utilizing a 25Gbps link for a single
connection (or a small number of connections), I think that we would be
generally considering of distributing both UDP send and encryption to
multiple CPU cores, because even without crypto, it is hard to utilize the
entire bandwidth using just one core (the same goes for the receive side).
And if you distribute the load to your CPU cores, the crypto cost becomes
somewhere around or below 10%.

This is my understanding. But I also understand that it could well be the
case that I do not understand what your workload looks like. That's why I
wonder if you could share the raw numbers and the use-case.

> IIUC, Ian should be talking about performance numbers that you would see
> on a server that is handling real workload. Not the breakdown of numbers
> within the QUIC stack.
>
>
>
> Assuming that your peak rate is 1Gbit/CPU core (would be enough to
> saturate 25Gb Ethernet), the ratio of CPU cycles for running the crypto
> will be:
>
>
>

-- 
Kazuho Oku

---

From: Nick Banks <[email-masked]>
Date: Sun, 24 Jun 2018 15:55:24 +0000

Hi Kazuho,

WinQuic will be a generic QUIC transport, with HTTP being just one of the scenarios we have planned for it. The goal for the transport is always to provide the most performant solution possible, using as few CPU cycles as possible, so the higher layers can have them. The test I used would most closely mimic a large file transfer over QUIC, were most of the CPU would end up being used by the QUIC transport.

So again, the goal here is to measure the cost of encryption (and eventually PNE) compared to other QUIC transport layer work. That will give us the baseline we can provide to other layers on top of us. Obviously, the total CPU percentage of encryption will be smaller when you include other layers on top of QUIC, but that will be dependent on the different protocols built on QUIC, and the particular work load / scenario being executed. We shouldn’t assume any particular protocol/scenario and say we are fine with using too much CPU at the lower layer, because the higher layer will be using even more CPU anyways.

As far as raw numbers, we are not prepared to share them at this time.

Thanks,
- Nick

________________________________
From: Kazuho Oku <[email-masked]>
Sent: Saturday, June 23, 2018 3:10:33 PM
To: Nick Banks
Cc: IETF QUIC WG; Ian Swett; Praveen Balasubramanian
Subject: Re: Packet Number Encryption Performance

2018-06-23 23:54 GMT+09:00 Nick Banks <[email-masked]<mailto:[email-masked]>>:
I agree that the numbers for real work loads will be interesting, but I do believe that raw QUIC stack numbers are also important. They will represent your best case scenario.

To clarify, the reason I asked about the raw numbers and the use-case is because without that I do not know how to interpret your numbers.

My understanding is that your benchmark is about encrypting and sending large QUIC packets.

Assuming that BCRYPT provides comparable performance to OpenSSL, the numbers you show (i.e. 22.5% in user-mode, 38.5% in kernel-mode) means that you want to reduce crypto cost when a single CPU core is emitting somewhere between 5Gbps to 8 Gbps.

Assuming that we have many connections, each CPU core will only handle somewhere around 1Gbps to saturate a 25Gbps link. As I have stated in my previous mail, the crypto cost will be around or less than 10% in such case.

Assuming that we are interested in utilizing a 25Gbps link for a single connection (or a small number of connections), I think that we would be generally considering of distributing both UDP send and encryption to multiple CPU cores, because even without crypto, it is hard to utilize the entire bandwidth using just one core (the same goes for the receive side). And if you distribute the load to your CPU cores, the crypto cost becomes somewhere around or below 10%.

This is my understanding. But I also understand that it could well be the case that I do not understand what your workload looks like. That's why I wonder if you could share the raw numbers and the use-case.

- Nick

User Mode

Kernel Mode

UDP Send

64.7%

41.4%

Encryption

22.5%

38.5%

Packet Building/Framing

7%

15%

Miscellaneous

5.8%

5.1%

- Nick

Hi Nick,

Time (ms)

Rate (Mbps)

Bytes

NO PNE

PNE

PNE Overhead

No PNE

PNE

4

2284.671

3027.657

33%

140.064

105.692

16

2102.402

2828.204

35%

608.827

452.584

64

2198.883

2907.577

32%

2328.45

1760.92

256

2758.3

3490.28

27%

7424.86

5867.72

600

4669.283

5424.539

16%

10280

8848.68

1000

6130.139

6907.805

13%

13050.3

11581.1

1200

6458.679

7229.672

12%

14863.7

13278.6

1450

7876.312

8670.16

10%

14727.7

13379.2

Thanks,
- Nick

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

--
Kazuho Oku

---

From: Rui Paulo <[email-masked]>
Date: Mon, 25 Jun 2018 10:52:39 -0700

> Also considering the fact that the number of CPU cycles spent per QUIC packet does not change a lot with PNE, I would not be surprised to see *no* decrease of throughput when PNE is used on a Hyper…

I don’t disagree but this is a very brand/model specific claim and we can’t assume the cost of PNE will be reasonable everywhere.

—
Rui Paulo

---

From: Jana Iyengar <[email-masked]>
Date: Tue, 26 Jun 2018 11:25:16 -0700

A higher order thought.

Crypto and PNE cost CPU. The only interesting question is if the relative
cost of PNE is "too high" for deployment in particular use cases.

The relative cost of crypto and PNE will change as the workload changes, as
the average connection throughput changes, as the application protocol
changes, as the underlying platform changes, and as the cost of the rest of
the send path changes. We do not have enough deployment experience to
determine that the cost of PNE is going to be too high for the HTTP use
case. And until we have deployment experience, contextualizing these raw
numbers will be difficult and will provide limited insight.

For non-HTTP and DC-like use cases, it is perfectly reasonable to roll a
local version to learn more about the dominant costs in those environments.
I would caution against diving too deep into arguments about cost without
actual deployment experience, especially for DC-like workloads.

I'm keen to see how these numbers move as the implementation matures and
how it changes in test/real deployments serving HTTP workloads.

On Mon, Jun 25, 2018 at 10:53 AM Rui Paulo <[email-masked]> wrote:

> I don’t disagree but this is a very brand/model specific claim and we
> can’t assume the cost of PNE will be reasonable everywhere.
>
> —
> Rui Paulo
>
>
>
>