Subject: Fwd: New Version Notification for draft-kazuho-quic-perf-metrics-00.txt (Re: Measurement bit(s) or not)

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Feb 2018 01:51:28 +0900

2018-02-11 2:34 GMT+09:00  <[email-masked]>:
> Then Kazuho's square signal and Mikkel's Pi (or any other consensual
> self-synchronizing sequence) ramification came up. They are both appealing
> for their elegance and low complexity on QUIC endpoints. Beyond their quirks
> acknowledged here, here are a few considerations for troubleshooting:
> (snip)
> Given these, a benevolent, troubleshooting-minded passive midpoint will
> clearly vote for the square. Now the obvious question is: is this
> acceptable, or deemed too easy for a Murphy, Inc. active middlebox to see
> upstream losses and benevolently wreak havoc by delaying packets ?

Thank you for raising the question.

While I think that using square wave is a neat idea, I am not so sure
if the wavelength is correct, or if it would continue to be correct
for the lifetime of the transport protocol. I assume that other
patterns that have been proposed (e.g., PI, random) share the same
weakness.

Therefore, I have written a proposal of a very different approach,
which is the following I-D.

    Performance Metrics Subprotocol for QUIC
    https://datatracker.ietf.org/doc/draft-kazuho-quic-perf-metrics/

The I-D proposes to add a subprotocol that is used between an on-path
observer and the QUIC server to obtain the performance metrics. At the
latency of 1 RTT, the subprotocol can observe accurate upstream
loss/reorder rate, as well as SRTT / RTTVAR.

I believe that it is not difficult to implement on the server-side
(please refer to section 4). Client needs no change.

Since the proposed approach does not hard-code any information for
observation on the wire, the hard part of the design issue goes away.
We can adjust what (and how) to observe independently from the QUIC
transport. I think that this is not only a win in terms of design but
also might have positive effect on moving the standardization process
forward, since we would no longer be constrained by the discussion of
what should be observable "on-the-wire."

Privacy concern regarding observation becomes minor, since observation
becomes "observable" and deniable by an endpoint.

Although SRTT / RTTVAR is observable, I am not sure if it can be used
to replace the Spin Bit proposal. It depends on how often and how fast
an observer needs to see the metrics. Primary target of my proposal is
to help diagnosing issues.

Please let me know what you think.

---------- Forwarded message ----------
From:  <[email-masked]>
Date: 2018-02-12 1:43 GMT+09:00
Subject: New Version Notification for draft-kazuho-quic-perf-metrics-00.txt
To: Kazuho Oku <[email-masked]>

A new version of I-D, draft-kazuho-quic-perf-metrics-00.txt
has been successfully submitted by Kazuho Oku and posted to the
IETF repository.

Name:           draft-kazuho-quic-perf-metrics
Revision:       00
Title:          Performance Metrics Subprotocol for QUIC
Document date:  2018-02-12
Group:          Individual Submission
Pages:          8
URL:
https://www.ietf.org/internet-drafts/draft-kazuho-quic-perf-metrics-00.txt
Status:         https://datatracker.ietf.org/doc/draft-kazuho-quic-perf-metrics/
Htmlized:       https://tools.ietf.org/html/draft-kazuho-quic-perf-metrics-00
Htmlized:
https://datatracker.ietf.org/doc/html/draft-kazuho-quic-perf-metrics-00

Abstract:
   This memo proposes a subprotocol that can be used to query the
   performance metrics of a QUIC path.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 11 Feb 2018 10:05:29 -0800

I need to read it more closely to understand the finer details, but my
immediate reaction is that I do not like this at all.

It appears to add unnecessary complication and there could be a host of
hidden security issues. The basic fact that someone other than the
endpoints can affect the traffic can lead to all sorts of issues and the
information transmitted might be abused in a number of ways, for example to
learn about the kind of traffic being sent, the internal layout of a
subnetwork by leaking presence of other middleware, the OS and applications
in use etc.

I much prefer a simple no-nonsense solution over trying to handle protocol
lifetime variations. The QUIC protocol could fix issues in new versions and
until then a simple predictable signal should suffice, even if it isn’t
perfect. This also makes middleware simpler, since a signal is always
present in a predictable manner.

We could have some fractal square waves to deal by having a short repeating
signal that periodically changes at different wave lengths, e.g. a a
repeating 8 bit pattern where each bit is flipped at the period of the wave
length it represents. That ought to be enough. The pattern repeats every 8
packets. The wave lengths could be 16, 32, 64, 128,  256, 512, 1024, 2048
so that same 8 bits are always transmitted twice before a bit changes which
should help synchronise it. It might not be perfect, but it ought to be
sufficient.

I’m also starting to think that just exposing a sequential counter per path
is much simpler even if it leads to ossification - see segment packet
number proposal https://github.com/quicwg/base-drafts/issues/1105

Kind Regards,
Mikkel Fahnøe Jørgensen

On 11 February 2018 at 17.51.37, Kazuho Oku ([email-masked]) wrote:

---------- Forwarded message ----------
From: <[email-masked]>
Date: 2018-02-12 1:43 GMT+09:00
Subject: New Version Notification for draft-kazuho-quic-perf-metrics-00.txt
To: Kazuho Oku <[email-masked]>

Abstract:

Please note that it may take a couple of minutes from the time of
submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

-- 
Kazuho Oku

---

From: <[email-masked]>
Date: Sun, 11 Feb 2018 19:36:04 +0100

On 11/02/2018 17:51, Kazuho Oku wrote:
> 2018-02-11 2:34 GMT+09:00<[email-masked]>:

Again, these shortcomings have been acknowledged here, and could be addressed 
later, once the more general question I'm asking is settled:

  => Do "we", or do "we" not, want to disclose a signal that allows a midpoint 
observer to deduce:

	- both upstream and downstream half-RTTs (spin bit)
	- upstream losses (consensual sequence: square, Pi, fractal, whatever)

  Pros: manageability, troubleshooting
  Cons: some Murphy middlebox ossifying the Internet instead of dying in flames

Once the answer is known and agreed upon, dedicated research on the best design 
for "measurement bits" will be timely. Not before.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Sun, 11 Feb 2018 10:47:16 -0800

Just to give a name to the fractal proposal: it is actually a grey code and
these have a long history of use in coding theory

https://en.wikipedia.org/wiki/Gray_code

On 11 February 2018 at 19.36.06, [email-masked] (
[email-masked]) wrote:

> patterns that have been proposed (e.g., PI, random) share the same

Again, these shortcomings have been acknowledged here, and could be
addressed
later, once the more general question I'm asking is settled:

=> Do "we", or do "we" not, want to disclose a signal that allows a
midpoint
observer to deduce:

Pros: manageability, troubleshooting
Cons: some Murphy middlebox ossifying the Internet instead of dying in
flames

Once the answer is known and agreed upon, dedicated research on the best
design
for "measurement bits" will be timely. Not before.

Ce message et ses pieces jointes peuvent contenir des informations
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu
ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.

This message and its attachments may contain confidential or privileged
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and
delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Feb 2018 06:12:57 +0900

Hi Mikkel,

Thank you for the response. Please see my comments in-line.

2018-02-12 3:05 GMT+09:00 Mikkel Fahnøe Jørgensen <[email-masked]>:
> hidden security issues. The basic fact that someone other than the endpoints
> can affect the traffic can lead to all sorts of issues and the information
> transmitted might be abused in a number of ways, for example to learn about
> the kind of traffic being sent, the internal layout of a subnetwork by
> leaking presence of other middleware, the OS and applications in use etc.

That is generally true.

But I am not sure why giving the option to the server to expose the
values being listed (# of packets sent, lost, rtt, number of lost
packets between the packets that the on-path device has received) is a
concern.

They are the stats that can be collected by observing a TCP connection.

Note that in the proposed approach, the exposure only happens if the
server wants to expose them, and the server can choose what to expose.
I think that this is a good thing in terms of privacy concern,
compared to unconditionally using some bits to expose information that
can be collected by a passible observer.

> perfect.

I think that I have a different goal here.

I want to provide network operators a good set of tools that they can
use for diagnosing issues of QUIC v1. I do not want to see people not
using QUIC due to the network operator not being diagnose and fix the
issues.

We might come up with a good signal encoded on each packet, as we
define multiple versions of QUIC. But that means spending multiple
years (if not multiple decades). I am not sure if I want to wait that
long.

That is the motive behind my proposal of using a subprotocol that can
be optionally implemented by a server and an on-path observer, and
that also can evolve independently from QUIC.

>  This also makes middleware simpler, since a signal is always
> packets. The wave lengths could be 16, 32, 64, 128,  256, 512, 1024, 2048 so
> that same 8 bits are always transmitted twice before a bit changes which
> is much simpler even if it leads to ossification - see segment packet number
> proposal https://github.com/quicwg/base-drafts/issues/1105
>
>
>
> Kind Regards,
> Mikkel Fahnøe Jørgensen
>
>

-- 
Kazuho Oku

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Feb 2018 06:32:24 +0900

2018-02-12 3:36 GMT+09:00  <[email-masked]>:
> addressed later, once the more general question I'm asking is settled:
>
> midpoint observer to deduce:
>
> design for "measurement bits" will be timely. Not before.

Hi Lili, thank you for your response.

While I like the orderly approach that you propose, I might also argue
that "if" we want to disclose a signal to observers might depend on
"how" the signal looks like.

And to me it seemed that, until now, we have shared a hidden premise
that the observation is a passive action.

But as I stated in my I-D, it is possible to define the observation
scheme as a request-response protocol.

The two (i.e. passive observation and use of request-response protocol
to obtain the metrics) have very different characteristics; something
that we might want to consider before making the decision.

That it one of the reasons behind my submitting the I-D at this moment.

-- 
Kazuho Oku

---

From: <[email-masked]>
Date: Sun, 11 Feb 2018 22:50:28 +0100

On 11/02/2018 22:32, Kazuho Oku wrote:
> 2018-02-12 3:36 GMT+09:00<[email-masked]>:
>>

Well, from doing network troubleshooting every day, I can tell you that passive 
analysis is enormously superior to active one:
  - you (midpoint) may be too far and too untrusted from the server to request 
any debug info from it (which is wht your protocol is about)
  - you may not know in advance which connection will display the problem of 
interest, and hence need to request debug for many of them
  - the problem may occur early in the connection's life, and the debug request 
may come too late
  - this simply doesn't scale up: you cannot do stats unless you request really 
too much debug info (and may be flagged by the server)
  - the potential for threats coming through the debug request channel is unknown.

For this reason, please let us first see what is acceptable for a passive 
measurement signal (including the spin bit).

---

From: "Fossati, Thomas (Nokia - GB/Cambridge)" <[email-masked]>
Date: Sun, 11 Feb 2018 21:51:13 +0000

On 11/02/2018, 16:51, "QUIC on behalf of Kazuho Oku" <[email-masked] on behalf of [email-masked]> wrote:
> [...]

Hi Kazuho

Thanks very much for taking the time to articulate your proposal.

A few notes:

- It looks like a quite significant DoS surface (both server and
  client-side) is enabled by accepting REQUESTs from anyone.

- Couldn't we expose the same info via an already existing interface,
  e.g. IPFIX / SNMP?

- If this is not a requirement for QUIC endpoints, what do you think
  is the incentive for servers to cooperate?

Cheers

---

From: "Roni Even (A)" <[email-masked]>
Date: Mon, 12 Feb 2018 06:11:47 +0000

+1
This is not monitoring  it can serve to get the statistics of a call/session and this total session information is better served using for example SNMP or similar management protocol and not QUIC
Roni

-----Original Message-----
From: QUIC [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Sunday, February 11, 2018 11:50 PM
To: Kazuho Oku
Cc: IETF QUIC WG
Subject: Re: Fwd: New Version Notification for draft-kazuho-quic-perf-metrics-00.txt (Re: Measurement bit(s) or not)

On 11/02/2018 22:32, Kazuho Oku wrote:
>>   =>  Do "we", or do "we" not, want to disclose a signal that allows 
>> a  midpoint observer to deduce:
>>
>>  Once the answer is known and agreed upon, dedicated research on the 
>> best  design for "measurement bits" will be timely. Not before.
>

Well, from doing network troubleshooting every day, I can tell you that passive analysis is enormously superior to active one:
  - you (midpoint) may be too far and too untrusted from the server to request any debug info from it (which is wht your protocol is about)
  - you may not know in advance which connection will display the problem of interest, and hence need to request debug for many of them
  - the problem may occur early in the connection's life, and the debug request may come too late
  - this simply doesn't scale up: you cannot do stats unless you request really too much debug info (and may be flagged by the server)
  - the potential for threats coming through the debug request channel is unknown.

For this reason, please let us first see what is acceptable for a passive measurement signal (including the spin bit).

---

From: Kazuho Oku <[email-masked]>
Date: Mon, 12 Feb 2018 17:13:36 +0900

Hi Thomas,

Thank you for your response. Please see my comments below.

>   client-side) is enabled by accepting REQUESTs from anyone.

First, let me assert that only on-path devices that observe a QUIC
packet used for that path can build a legitimate REQUEST.

Regarding DoS surface, a server typically accepts packet from any
address, so it is not an issue for the server in terms of bandwidth.
Regarding computational cost, it would be much cheaper to process the
REQUEST packet and send the response than to handle a handshake packet
(which involves a public-key crypto). Therefore, I do not think that
it would be an attractive attack surface in terms of computational
cost.

So the only aspect of this potentially being a DoS vector is the
possibility of it being used to saturate the bandwidth from the server
to the client. I agree that it is a concern and that the server should
rate-limit how often it sends the RESPONSE packets.

> - Couldn't we expose the same info via an already existing interface,

We can expect higher chance of the feature being deployed, if it can
be implemented solely by the QUIC server, and if it can be deployed
without any additional hassles.

Since the server with the knowledge of the packet number encryption
key (which is required to calculate distances) is often located behind
a load balancer, it makes sense to send the REQUEST packet with its
destination address and Connection ID set to the values that would be
found in a QUIC packet sent by a client.

We can expect that a QUIC server is always capable to send a packet to
the client. Therefore, it makes sense for a server to respond in the
form of a QUIC packet sent to the client.

>   is the incentive for servers to cooperate?

Let my first point out that any feature that is specific to
observation becomes an optional feature, because endpoints do not need
to implement them to be able to communicate.

At the same time, I think that designing observation as something that
can be implemented solely as part of the QUIC server and deployable
without any other restriction gives us a higher chance of being it
deployed, as I explained above.

-- 
Kazuho Oku

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 12 Feb 2018 00:23:52 -0800

The proposal could also be implemented as an application protocol on top of
QUIC so network monitoring software could connect, authenticate and
subscribe to data. This would with the security of QUIC instead of against
it, and it could break some ground compared to ageing monitoring protocols
that might have their own security issues. I’d be much more in favour of
such a solution.

I don’t 2018 is the year to put more clear text on the wire, especially not
when to goal is to remove something as mundane as a sequential counter from
clear text.

In either case I would assume that it would not be an acceptable
replacement for passive monitoring.

---

From: Mikkel Fahnøe Jørgensen <[email-masked]>
Date: Mon, 12 Feb 2018 00:29:43 -0800

QUIC

A way to do this could be to post monitoring data on an MQTT message queue,
especially if that queue is secured with QUIC. I’ve build application
monitoring solutions in a similar manner and it works quite well.

---

From: "Brian Trammell (IETF)" <[email-masked]>
Date: Mon, 12 Feb 2018 10:20:15 +0100

hi Kazuho,

Many thanks for the draft. I've had a quick look at it. I like the architecture it proposes -- hey, since the endpoints have full access to all of these parameters, why not just ask them? -- but I'm afraid I agree with the chorus of negative voices here that this raises too many new issues to really be workable.

We looked at something like this for PLUS -- a generic path-request mechanism allowing a middlebox to ask an endpoint for a value -- and decided to drop it from the proposal because the trust and DoS issues were just too hairy.

First, the mechanism adds up to 1RTT of delay, depending on the relative position of the requesting device and the endpoint it wants to request information from. Second, the reported metrics only have the resolution available from the queried endpoint -- as we've seen in our work with the spin bit, completely exposed signals don't suffer from this issue. But most importantly, adding a hook for on path devices to query endpoints adds significant complexity and attack surface, and seems extremely difficult to secure against collaborating malicious devices on-and-off-path.

One nice thing about purely passive signals is that we know what the information content is, we have a pretty good handle on how to restrict their design to reduce the risk of additional endpoint fingerprinting, and if we find the right tradeoff in endpoint complexity and measurement fidelity, can get a lot of win for minimal implementation effort and very little if any attack surface.

I certainly think measurement and debugging approaches where both servers and clients use existing mechanisms or extensions thereof (IPFIX or something like Prometheus on the server side; extensions to JS APIs for browsers or some other instrumentation on the client side) are a useful part of diagnosing issues with QUIC traffic; indeed, standardizing one or two of these would be a useful side project to reduce the shock widespread QUIC deployment represents to the hosting and web development communities. But neither scales very well to getting information to other-than-client and other-than-server entities.

Thanks, chsser,

Brian

> On 11 Feb 2018, at 17:51, Kazuho Oku <[email-masked]> wrote:
>