Subject: Integer encodings in QUIC frames

From: Martin Thomson <[email-masked]>
Date: Thu, 19 Oct 2017 11:25:44 +1100

Hi All,

Some of you might have heard discussion about the difficulty of
parsing QUIC frames, or heard me talk about a proposal for reducing
the number of different variations.

I have a proposal in PR #877 [1] that replaces virtually all of the
integer encodings in frames to use the same variable-length encoding.
This includes the ACK delay timestamp.

That encoding takes two bits to describe the length of the integer (1,
2, 4, or 8 octets) and the remainder is big-endian goodness.  This
produces ranges that are a bit unusual: 6-, 14-, 30-, or 62-bit.
However, this should cover most use cases and is pretty fast to decode
and to comprehend.

My testing [2] shows that this is about ~50% slower than using a
simple htonl or equivalent [3], but it can save a lot of bytes,
depending on usage model.

This would only affect the insides of frames.  No changes are made to
the packet header, connection ID, version or error codes, for which
fixed-sized fields are convenient for several reasons.

The change would mean different limits.  We would have 4 times fewer
packets and flow control octets, but 2^30 times more streams.

This PR doesn't change the HTTP mapping yet.  I have two PRs open in
addition to this that make a matching change to the HTTP mapping.
#887 [4] makes the minimum change, effectively ignoring the changes at
the lower layer and sticking to 32-bit identifiers.  #888 [5] imports
the integer format more widely.  I prefer the latter, but want to hear
more opinions, because it would have consequences for HTTP/2
compatibility (see the PR for details).  I expect that the HTTP
working group will also have some things to say about that, so I plan
to ask them too.

I haven't proposed more significant changes to the ACK frame; the
encoding change addresses some of the identified shortcomings, but
probably not all of them.  More significant changes might be proposed
separately.

--Martin

[1] https://github.com/quicwg/base-drafts/pull/877/files
[2] https://github.com/martinthomson/quic-int
[3] All the usual caveats apply.  Such as, my machine might not be
representative, and that I didn't spend any time optimizing.
Interesting note though: Feeding my code a counter as input makes it
~35% faster than an simple endian swap, probably because it uses less
than one quarter of the number of octets.
[4] https://github.com/quicwg/base-drafts/pull/887/files
[5] https://github.com/quicwg/base-drafts/pull/888/files

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 19 Oct 2017 04:29:29 +0200

Hi Martin,

On Thu, Oct 19, 2017 at 11:25:44AM +1100, Martin Thomson wrote:
> compatibility (see the PR for details).

I too support such a model, which can even be further optimized. I'd
even go a bit further : modern architectures are little endian and
support unaligned reads (x86_64, armv7, armv8). By adopting your model
and storing the word length in the lower bits of the first byte and by
using a little endian representation, you can be much faster on all
platforms which matter nowadays (ie: most clients are armv8 and most
servers are x86) :

    word = *(uint64_t *)ptr;
    len = word & 3;
    ptr += 1 << len;
    word >>= 2;
    word &= (1 << (8 * (1 << len) - 2)) - 1;  // &= mask[len]

It does not even perform any jump nor comparison. I could include it
into your test program but don't have much time at the moment :-/

> [1] https://github.com/quicwg/base-drafts/pull/877/files

Just FYI there's an error in the range of values table, the ranges for
lengths 2, 4, 8 should be decreased by 1.

Cheers,
Willy

---

From: Martin Thomson <[email-masked]>
Date: Thu, 19 Oct 2017 14:13:23 +1100

>     word = *(uint64_t *)ptr;

Yeah, we more or less already had that debate and I didn't want a
re-run.  I do agree that it seems faster in theory (see below), but
the law of diminishing returns applies here.  Also, that code isn't
portable, so it would need a forest of #ifdef statements to safeguard
it.

>     word &= (1 << (8 * (1 << len) - 2)) - 1;  // &= mask[len]

The one optimization I did do, which made my code about 4 times
faster, was to remove this sort of calculation.  I don't completely
understand why, but a simple if statement was much faster than this.

I tried your code and it was a lot slower than my big-endian code on
decoding, though encode was maybe a tiny fraction faster).  A table
lookup (mask[len]) was a small bit faster, but it was still in the
order of 2.5 times slower.  I think that the branch doesn't cost as
much as you think (it could just be that I have a good branch
predictor on my 5-year-old CPU).  Every time I play with code
optimization, I'm surprised by this sort of result.

See https://github.com/martinthomson/quic-int/pull/1

> lengths 2, 4, 8 should be decreased by 1.

Thanks, fixed.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 19 Oct 2017 05:28:36 +0200

> portable, so it would need a forest of #ifdef statements to safeguard

Sure but that's an implementation detail, as the portable code should
only include byte-per-byte reading. We all have #ifdef for different
architectures in our code where that's relevant.

> understand why, but a simple if statement was much faster than this.

It's very possible due to the word length distribution. Branch predictors
are pretty good and when you realize that most consecutive words are of
the same size, you quickly understand why jumps are cheap. My first
attempt which I removed before posting was to use a switch/case, which
the compiler easily optimizes as "jmp table[len]".

> optimization, I'm surprised by this sort of result.

Oh I'm never surprized, in general I experiment, then tweak and only
once it works as expected I figure why I initially took the wrong
direction ;-)

> See https://github.com/martinthomson/quic-int/pull/1

Ah thanks a lot, when I have time I'll play with it. I'm currently finishing
the reimplementation of H2 in haproxy and prefer not to switch to some
very addicting stuff before that's done you see ;-)

Cheers,
Willy

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 19 Oct 2017 04:41:58 -0700

On fast portable unaligned integer access and endian conversion:

If you have a C implementation feel free to use the portable library to
abstract endian and unaligned read access in a portable manner. It is just
a few header files dealing with endian detection, endian conversion,
unaligned access, and compiler warnings, and fallbacks to unsupported
platforms:

https://github.com/dvidelabs/flatcc/tree/master/include/flatcc/portable

The interesting file is:
https://github.com/dvidelabs/flatcc/blob/master/include/flatcc/portable/punaligned.h

It is used by the C flatbuffers implementation, but flatcc is not needed to
use the portable sub library, nor are all files in the the portable library
- just follow the includes.

The library is valuable because it is being used by end users on various
non-standard platforms, although I cannot tell how many are using it.

With the functions
unaligned_write_htobe16
unaligned_write_htobe32
unaligned_write_htobe64
unaligned_read_be16toh
unaligned_read_be32toh
unaligned_read_be64toh

I believe I also have other lengths between 8 and 64 bits, but they are not
currently in the public library. They are basically wrappers that combine
calls to the above, so only the above have direct native implementations. I
have some incomplete QUIC implementation that has a switch to read integers
based on a length argument. When QUIC changes from little to big endian it
was a very trivial change because they are similar functions for other
endian conversions.

I also use a subset of this in other projects for hash functions where you
need fast unaliged access and sometimes platform neutral wireformat.

If you have platform where a slow fallback is in effect in the portable
library, but you also know of a faster implementation that you are able to
test, feel free to add a PR. Notably unaligned reads are a bit conservative
while native endian conversion supports many platforms.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 19 October 2017 at 05.28.45, Willy Tarreau ([email-masked]) wrote:

> See https://github.com/martinthomson/quic-int/pull/1

Ah thanks a lot, when I have time I'll play with it. I'm currently
finishing
the reimplementation of H2 in haproxy and prefer not to switch to some
very addicting stuff before that's done you see ;-)

Cheers,
Willy

---

From: Ian Swett <[email-masked]>
Date: Thu, 19 Oct 2017 08:35:41 -0400

I'm supportive of this change as is, though I think we should consider not
changing the ACK delay field for now.  I heard enough different opinions on
what should be done with it that I don't want consensus on that to hold up
this PR.  That being said, if everyone is now happy with this ack delay
format(or not less happy than before), feel free to land it as is.

Your more 'maximal' HTTP change(#888) looked fine as well.

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 19 Oct 2017 22:04:24 +0200

Hi Martin,

> the compiler easily optimizes as "jmp table[len]".

So I found the reason, I managed to rewrite it a bit differently and make
it 20-75% faster than the BE code, but there's a fundamental issue in
doing it this way. First, a copy-paste of the results before I lose them :

willy@pcw:/data/git/mirror/quic-int$ ./bench64 r 2>/dev/null 
Encoding and decoding 1000 integers over 1000 iterations
--- Type ---     Encode          Decode          Length
memcpy:             1380            1798            8000
endian:             2197            1650            8000
highbitbe:         19316           15845            8987
highbitle:         20546           19824            8987
quic:               3557            3558            8000
quicle:             3551            2030            8000
willy@pcw:/data/git/mirror/quic-int$ ./bench64 t 2>/dev/null 
Encoding and decoding 1000 integers over 1000 iterations
--- Type ---     Encode          Decode          Length
memcpy:             1395            1809            8000
endian:            12651            1640            8000
highbitbe:         14433            8459            4254
highbitle:         13336           10397            4254
quic:               3789            3782            3754
quicle:             3741            3157            3754
willy@pcw:/data/git/mirror/quic-int$ ./bench64 c 2>/dev/null 
Encoding and decoding 1000 integers over 1000 iterations
--- Type ---     Encode          Decode          Length
memcpy:             1392            1809            8000
endian:             2313            1642            8000
highbitbe:         10551            4261            1872
highbitle:          5175            5113            1872
quic:               3526            3481            1936
quicle:             2669            2678            1936

So the problem here which doesn't happen with the BE code is that
the word has to be shifted whatever its length since we're using
the first, hence lower, bits to store the length, causing a bit of
serialization in the operations. In the BE case, short values don't
even have to have a mask applied since they're lower than the
bound. That's one less operation. I also noticed that the compiler
didn't decide to use the jump array and produces ugly code (though
not necessarily suboptimal). By cutting the evaluation size in to
(0..1, and 2..3) it's significantly faster.

I tried again by computing the shifts by hand to avoid any jump
and we stall the pipeline a lot there due to the many shift that
happen and serialize operation again.

So on this one I'm convinced, better have sort of a big endian
mode for variable length encoding.

If you want I can send you the patch for the extra experiments,
but I doubt you'll make any use of it.

Cheers,
Willy

---

From: Roberto Peon <[email-masked]>
Date: Thu, 19 Oct 2017 20:14:44 +0000

I’m a bit worried that this spec simplification will actually increase the complexity of a performant implementation.

-=R

On 10/19/17, 1:05 PM, "QUIC on behalf of Willy Tarreau" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Martin,

    

    

    Cheers,
    Willy

---

From: Willy Tarreau <[email-masked]>
Date: Thu, 19 Oct 2017 22:24:14 +0200

Hi Roberto,

On Thu, Oct 19, 2017 at 08:14:44PM +0000, Roberto Peon wrote:
> I'm a bit worried that this spec simplification will actually increase the complexity of a performant implementation.

No, Martin's implementation is quite good in fact. I thought that by
turning the words to little endian we could go even faster but that's not
the case. At least with the big endian version, we have the ability to
read a word at once, check it, reverse it and apply the mask if needed.
The code remains quite trivial.

Willy

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thu, 19 Oct 2017 16:39:42 -0400

There is nothing inherent in the new encoding format that is more difficult
than it was before. The length of the stored values was just encoded all
over the place. Either way you must deal with unaligned access, endian
conversion, and different integer lengths.

It is not a simple task, but the the default platform neutral
implementation is not very slow in the overall picture, but it can be very
fast with the proper native support, and that takes a lot of work to get
right. But again, it applies regardless of how integers are encoded.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 19 October 2017 at 22.24.26, Willy Tarreau ([email-masked]) wrote:

Hi Roberto,

On Thu, Oct 19, 2017 at 08:14:44PM +0000, Roberto Peon wrote:
> I'm a bit worried that this spec simplification will actually increase
the complexity of a performant implementation.

Willy

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 19 Oct 2017 17:21:06 -0400

> conversion, and different integer lengths.

I think it will be at least a _little_ more difficult:  One advantage
of the sizes being encoded in the STREAM type byte is that one can
calculate required length and check it quickly, e.g.

    static int
    gquic_le_parse_stream_frame (const unsigned char *buf,
                    size_t rem_packet_sz, stream_frame_t *stream_frame)
    {
        /* 1fdoooss */
        const unsigned char *p = buf;
        const unsigned char *const pend = p + rem_packet_sz;
        CHECK_SPACE(1, p, pend);
        const char type = *p++;
        const unsigned data_len   = (type >> 4) & 2;
        const unsigned offset_len = ((type >> 2) & 7) + 1 - !((type >> 2) & 7);
        const unsigned stream_id_len = 1 + (type & 3);
        const unsigned need = data_len + offset_len + stream_id_len;
        CHECK_SPACE(need, p, pend);

Reading lengths and values of variable-sized integers will be more
involved (one must also make sure he does not read past the buffer).
I like the consistencly of the proposed approach, though.

  - Dmitri.

---

From: Martin Duke <[email-masked]>
Date: Thu, 19 Oct 2017 15:21:49 -0700

It's important to be clear that length of stored values is *still* "encoded
all over the place". Some fields have length encoded into the first two
bits, while others still use flags in the frame header.

I would argue that this change actually makes things more complicated.
However, it is strictly superior in addressing particular issues in the ack
frame, like gaps of more than 255 packet numbers.

On Thu, Oct 19, 2017 at 1:39 PM, Mikkel Fahnøe Jørgensen <[email-masked]

> There is nothing inherent in the new encoding format that is more
> difficult than it was before. The length of the stored values was just
> encoded all over the place. Either way you must deal with unaligned access,
> endian conversion, and different integer lengths.
>

---

From: Leif Hedstrom <[email-masked]>
Date: Thu, 19 Oct 2017 16:50:14 -0600

> On Oct 18, 2017, at 6:25 PM, Martin Thomson <[email-masked]> wrote:
> 
> Hi All,
> 

I always wondered what sort of real-world impact this micro optimization of variable integer length actually has. Do we have numbers on that? It seems like a complicated implementation in general, with perhaps questionable benefits?

Cheers,

— leif

---

From: Roberto Peon <[email-masked]>
Date: Thu, 19 Oct 2017 23:42:50 +0000

Fair enough. Worrying about nothing is my favorite kind.
-=R

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Thursday, October 19, 2017 at 1:40 PM
To: Willy Tarreau <[email-masked]>, Roberto Peon <[email-masked]>
Cc: Martin Thomson <[email-masked]>, QUIC WG <[email-masked]>
Subject: Re: Integer encodings in QUIC frames

There is nothing inherent in the new encoding format that is more difficult than it was before. The length of the stored values was just encoded all over the place. Either way you must deal with unaligned access, endian conversion, and different integer lengths.

It is not a simple task, but the the default platform neutral implementation is not very slow in the overall picture, but it can be very fast with the proper native support, and that takes a lot of work to get right. But again, it applies regardless of how integers are encoded.

Kind Regards,
Mikkel Fahnøe Jørgensen

On 19 October 2017 at 22.24.26, Willy Tarreau ([email-masked]<mailto:[email-masked]>) wrote:
Hi Roberto,

Willy

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 20 Oct 2017 02:15:05 +0000

Happy with the new format for the ACK delay field. Anything other than the (current) custom floating point format is forward progress even if temporary.

From: QUIC [mailto:[email-masked]] On Behalf Of Ian Swett
Sent: Thursday, October 19, 2017 5:36 AM
To: Martin Thomson <[email-masked]>
Cc: QUIC WG <[email-masked]>
Subject: Re: Integer encodings in QUIC frames

I'm supportive of this change as is, though I think we should consider not changing the ACK delay field for now.  I heard enough different opinions on what should be done with it that I don't want consensus on that to hold up this PR.  That being said, if everyone is now happy with this ack delay format(or not less happy than before), feel free to land it as is.

On Wed, Oct 18, 2017 at 8:25 PM, Martin Thomson <[email-masked]<mailto:[email-masked]>> wrote:
Hi All,

--Martin

[1] https://github.com/quicwg/base-drafts/pull/877/files<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fpull%2F877%2Ffiles&data=02%7C01%7Cpravb%40microsoft.com%7Cb074779200f34f924e3708d516edfa59%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636440133726231720&sdata=%2Bmu6qwieygpXpev6%2FXh9j1ecCmTeTn54qSk4ezY2SGU%3D&reserved=0>
[2] https://github.com/martinthomson/quic-int<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmartinthomson%2Fquic-int&data=02%7C01%7Cpravb%40microsoft.com%7Cb074779200f34f924e3708d516edfa59%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636440133726241728&sdata=UyGQieIwLsIkGSVfSrxR%2FatnCBLFRMpBoQFNWID%2F%2Fj8%3D&reserved=0>
[3] All the usual caveats apply.  Such as, my machine might not be
representative, and that I didn't spend any time optimizing.
Interesting note though: Feeding my code a counter as input makes it
~35% faster than an simple endian swap, probably because it uses less
than one quarter of the number of octets.
[4] https://github.com/quicwg/base-drafts/pull/887/files<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fpull%2F887%2Ffiles&data=02%7C01%7Cpravb%40microsoft.com%7Cb074779200f34f924e3708d516edfa59%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636440133726241728&sdata=7wqkeT9zNPr5qNuw3TBNm2RPNFp8cTgpSf9JU0cVd5I%3D&reserved=0>
[5] https://github.com/quicwg/base-drafts/pull/888/files<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fquicwg%2Fbase-drafts%2Fpull%2F888%2Ffiles&data=02%7C01%7Cpravb%40microsoft.com%7Cb074779200f34f924e3708d516edfa59%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636440133726241728&sdata=U71exs6x0dXllI2%2BZclwlBuY6YhRfKpjQpVo3kkZ7WU%3D&reserved=0>

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 20 Oct 2017 04:58:40 +0200

> involved (one must also make sure he does not read past the buffer).

In fact what matters to me is that a naive approach is reasonably easy to
implement, and that given some efforts, it's possible to implement a very
efficient version where needed. For example, dealing with reading past the
buffer is never easy when you deal with multi-byte reads at once, but in
practice when you do this, your code will check if there's a *risk* of
getting there and will fall back to the one byte at a time code in such
conditions. When you parse a 16kB buffer and only the 7 last bytes are
processed this way, it means it's fast 99.95% of the time.

> I like the consistencly of the proposed approach, though.

Same for me. If we could ensure that there's a single method to emit and
parse any integer, it would be nice. That's definitely part of a nicely
designed protocol.

Willy

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 20 Oct 2017 05:17:13 +0200

On Thu, Oct 19, 2017 at 04:50:14PM -0600, Leif Hedstrom wrote:
> I always wondered what sort of real-world impact this micro optimization of
> variable integer length actually has. Do we have numbers on that? It seems
> like a complicated implementation in general, with perhaps questionable
> benefits?

you can easily spend 10-20% of your program's time processing frame fields,
so it can be very important especially on the server side, to maintain a
good scalability. Reducing the *minimum* time needed to process a frame is
also important to better resist DDoS.

I just ran a quick test on my H2 implementation in haproxy, and by disabling
similar optimizations on int32 processing (ie reading the stream ID one byte
at a time instead of one word at a time), the performance drops from 860k
req/s to 840k/s. At such loads, every cycle counts when it's repeated a lot.

Willy

---

From: Leif Hedstrom <[email-masked]>
Date: Thu, 19 Oct 2017 21:49:48 -0600

> On Oct 19, 2017, at 9:17 PM, Willy Tarreau <[email-masked]> wrote:
> 

Right. I probably didn’t word my question well. What am asking is, is it worth the complexity of this at all, to save a few bytes on the wires?

Cheers,

— Leif

---

From: Willy Tarreau <[email-masked]>
Date: Fri, 20 Oct 2017 06:12:26 +0200

On Thu, Oct 19, 2017 at 09:49:48PM -0600, Leif Hedstrom wrote:
> Right. I probably didn't word my question well. What am asking is, is it
> worth the complexity of this at all, to save a few bytes on the wires?

Very often, yes. Saving bytes on the wire is mainly what makes H2 much
faster than H1.

Willy

---

From: Ian Swett <[email-masked]>
Date: Fri, 20 Oct 2017 08:37:59 -0400

I would agree it's definitely valuable.  Also, even if variable length
coding costs a bit of framing CPU, it's that many fewer bytes to
send/receive, decrypt/encrypt and move into CPU cache.  In the typical
case, I would guess it's a net CPU savings, at least for this type of
varint.  In the case of a single stream frame, using varints for stream ids
and offsets saves ~12 bytes, which is ~1% overhead for full sized packets
and more for smaller payloads.

---

From: Martin Thomson <[email-masked]>
Date: Mon, 23 Oct 2017 14:23:20 +1100

One thing we learned from SPDY and HTTP/2 was that the first few round
trips are critical to performance.  If requests in particular can be
made smaller, you avoid running into the limit that the congestion
controller places on your total outbound bandwidth.  Making integers
one octet rather than 8 makes a big difference.

In web tests, this limit often hits hardest on the second flight of
messages, after the single request turns into hundreds.  With 0-RTT
that happens after just a single RTT, so the congestion controller
probably still has the tap set to a pretty narrow setting.

(I tried looking for a good citation, but I couldn't find one.  Maybe
others have pointers at the ready.)

On Fri, Oct 20, 2017 at 11:37 PM, Ian Swett <[email-masked]> wrote:
> send/receive, decrypt/encrypt and move into CPU cache.  In the typical case,
> I would guess it's a net CPU savings, at least for this type of varint.  In
> the case of a single stream frame, using varints for stream ids and offsets
> saves ~12 bytes, which is ~1% overhead for full sized packets and more for
> smaller payloads.
>

---

From: Christian Huitema <[email-masked]>
Date: Sun, 22 Oct 2017 20:38:40 -0700

> probably still has the tap set to a pretty narrow setting.

There is another reason -- encryption. One byte compressed away is one
fewer byte to encrypt. So as long as your compression system costs less
CPU than the encryption, you win.

-- 
Christian Huitema

---

From: Roland Zink <[email-masked]>
Date: Mon, 23 Oct 2017 12:14:12 +0200

Compression together with encryption also sounds somewhat dangerous. An 
attacker may learn something by monitoring when length changes are 
happening.

Roland

---

From: Willy Tarreau <[email-masked]>
Date: Mon, 23 Oct 2017 13:07:58 +0200

> attacker may learn something by monitoring when length changes are

Here it's quite different from dictionary-based compression which
is brute-forceable to disclose exact contents, in the worst case
you don't disclose contents, just basic approximate of magnitude
which tells you a quantity is multiplied by 256 when you see an
extra byte. Given that often this may be used for stuff like content
sizes, stream IDs and such elements, it hardly discloses anything of
interest. However it's true that if some sensitive parts must really
be kept secret, it might be better to avoid using variable length
encoding for them.

Willy

---

From: Roland Zink <[email-masked]>
Date: Mon, 23 Oct 2017 14:07:56 +0200

Not sure if those things are really of no interest. Assuming for a 
moment that it is possible to detect the size of the stream ID and there 
is a single short stream ID for a long living stream then it is possible 
to follow the stream after all other streams are using longer stream 
IDs. In most cases this shouldn't be possible but there might be some 
corner cases. When necessary there should be the mitigation to use some 
padding or use longer than necessary encodings.

Regards,

Roland

---

From: Roberto Peon <[email-masked]>
Date: Mon, 23 Oct 2017 17:58:27 +0000

True! 

-=R

On 10/22/17, 8:23 PM, "QUIC on behalf of Martin Thomson" <[email-masked] on behalf of [email-masked]> wrote: