Subject: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

From: Cociglio Mauro <[email-masked]>
Date: Wed, 8 Apr 2020 08:25:07 +0000

Dear QUIC WG Members.

We submitted to IPPM WG a draft where we described a new 2 bit packet loss methodology (https://tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-01).
The measurement of packet loss is under discussion in QUIC WG and our draft introduces two alternatives about it.
The first one is a spin-bit dependent signal and uses a single bit. The second one, described in the following linked slides, is a standalone solution based on a two bits loss signal and on alternate marking (RFC8321).
This last methodology improves, in our opinion, the algorithm proposed by Orange and Akamai described in https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-03.

The 2 bits are the sQuare bit (Q-bit) and the Reflection square bit (R-bit).
The Q-bit doesn't change from the Ferrieux-Hamchaoui draft but the R-bit substitutes the L-bit.
This avoids the L-bit dependence from an internal protocol variable, a problem raised in the last QUIC interim meeting.

You can find the slides, we prepared for the IPPM interim meeting, at the following link: https://github.com/ietf-ippm/meeting-materials/blob/aa486f7ead28b9f55c5ef499e5c9ed33ab06daf9/ietf107-virtual/Slides/07-spinbit-measurements.pdf

Comments and suggestions are always welcome.

Of course we are available to present our proposals in the next QUIC WG meeting or to arrange a Webex side meeting if needed.

Best regards.

Mauro, Fabio, Giuseppe, Massimo and Riccardo

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)
Tel.: +390112285028
Mobile: +393357669751
_____________________

Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie. 

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks. 

Rispetta l'ambiente. Non stampare questa mail se non è necessario.

---

From: Ian Swett <[email-masked]>
Date: Wed, 8 Apr 2020 15:43:59 -0400

Thanks for sharing this.  From my perspective, this is an improvement upon
the previous proposal in terms of robustness.  This design feels very
similar to the spin bit, and seems trivial to implement, which are also
nice properties.

Also, QUIC doesn't really have retransmissions, so the 'R bit' name always
made me a bit uncomfortable.

I'm not saying QUIC should adopt this yet, but I'd be interested in seeing
a privacy analysis completed for it.

Ian

On Wed, Apr 8, 2020 at 4:25 AM Cociglio Mauro <mauro.cociglio=
[email-masked]> wrote:

> Dear QUIC WG Members.
>
>
>
> We submitted to IPPM WG a draft where we described a new 2 bit packet loss
> methodology (
> https://tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-01).
>
> *The measurement of packet loss is under discussion in QUIC WG and our
> draft introduces two alternatives about it.*
>
> *T**he first one is a spin-bit dependent signal and uses a single bit.
> The second one, described in the following linked slides, is a
> standalone solution based on a two bits loss signal and on alternate
> marking (RFC8321)..*
>
> *This last methodology **improves, in our opinion, the algorithm proposed
> by Orange and Akamai described in* *https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-03
> <https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-03>**.*
>
>
>
> *The 2 bits are the sQuare bit (Q-bit) and the Reflection square bit
> (R-bit).*
>
> *The Q-bit doesn’t change from the Ferrieux-Hamchaoui draft but the R-bit
> substitutes the L-bit.*
>
> *This avoids the L-bit dependence from an internal protocol variable, a
> problem raised in the last QUIC interim meeting.*
>
>
>
> You can find the slides, we prepared for the IPPM interim meeting, at the
> following link:
> https://github.com/ietf-ippm/meeting-materials/blob/aa486f7ead28b9f55c5ef499e5c9ed33ab06daf9/ietf107-virtual/Slides/07-spinbit-measurements.pdf
>
>
>
> Of course we are available to present our proposals in the next QUIC WG
> meeting or to arrange a Webex side meeting if needed.
>
>
>
> Best regards.
>
>
>
> Tel.: +390112285028 <+39%20011%20228%205028>
>
> Mobile: +393357669751 <+39%20335%20766%209751>
>
> _____________________
>
>
>
>
> <https://on.tim.it/banner-mail-dip>
> Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle
> persone indicate. La diffusione, copia o qualsiasi altra azione derivante
> dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora
> abbiate ricevuto questo documento per errore siete cortesemente pregati di
> darne immediata comunicazione al mittente e di provvedere alla sua
> distruzione, Grazie.
>
> * This e-mail and any attachments is confidential and may contain
> privileged information intended for the addressee(s) only. Dissemination,
> copying, printing or use by anybody else is unauthorised. If you are not
> the intended recipient, please delete this message and any attachments and
> advise the sender by return e-mail, Thanks. *
>
> *Rispetta l'ambiente. Non stampare questa mail se non è necessario.*
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
>

---

From: Cociglio Mauro <[email-masked]>
Date: Thu, 9 Apr 2020 16:21:16 +0000

Thanks Ian for your email.
We intend to proceed with the privacy analysis, which is the same as that of the Ferrieux-Hamchaoui draft for the part of the Q-bit (and the R-bit should be very similar to Q-bit  in terms of privacy).
R-bit stands for “Reflected square bit”, not for “Retransmission bit”, but of course the letter can be changed.
Best regards and happy Easter.

Mauro

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)
Tel.: +390112285028
Mobile: +393357669751
_____________________

Da: Ian Swett <ianswett=[email-masked]>
Inviato: mercoledì 8 aprile 2020 21:44
A: Cociglio Mauro <[email-masked]>
Cc: [email-masked]; Lubashev, Igor <[email-masked]>; [email-masked]; [email-masked]; Riccardo Sisto <[email-masked]>; IETF IPPM WG ([email-masked]) <[email-masked]>
Oggetto: [EXT] Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Thanks for sharing this.  From my perspective, this is an improvement upon the previous proposal in terms of robustness.  This design feels very similar to the spin bit, and seems trivial to implement, which are also nice properties.

Also, QUIC doesn't really have retransmissions, so the 'R bit' name always made me a bit uncomfortable.

I'm not saying QUIC should adopt this yet, but I'd be interested in seeing a privacy analysis completed for it.

Ian

On Wed, Apr 8, 2020 at 4:25 AM Cociglio Mauro <mauro.cociglio=[email-masked]<mailto:[email-masked]>> wrote:
Dear QUIC WG Members.

We submitted to IPPM WG a draft where we described a new 2 bit packet loss methodology (https://tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-01).
The measurement of packet loss is under discussion in QUIC WG and our draft introduces two alternatives about it.
The first one is a spin-bit dependent signal and uses a single bit. The second one, described in the following linked slides, is a standalone solution based on a two bits loss signal and on alternate marking (RFC8321)..
This last methodology improves, in our opinion, the algorithm proposed by Orange and Akamai described in https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-03.

The 2 bits are the sQuare bit (Q-bit) and the Reflection square bit (R-bit).
The Q-bit doesn’t change from the Ferrieux-Hamchaoui draft but the R-bit substitutes the L-bit.
This avoids the L-bit dependence from an internal protocol variable, a problem raised in the last QUIC interim meeting.

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)
Tel.: +390112285028<tel:+39%20011%20228%205028>
Mobile: +393357669751<tel:+39%20335%20766%209751>
_____________________

ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Thu, 9 Apr 2020 12:28:03 -0500

18

On Thu, Apr 9, 2020 at 11:22 AM Cociglio Mauro <mauro.cociglio=
[email-masked]> wrote:

> We intend to proceed with the privacy analysis, which is the same as that
> of the *Ferrieux-Hamchaoui draft for the part of the Q-bit (and the R-bit
> should be very similar to Q-bit  in terms of privacy).*
>

If you need to change the letter, "S-bit" for "Square bit" would be one of
the best field names I've seen in a while (sooner or later, every QUIC
developer will ask, "what's a square bit?").

I'm mostly serious, and only partially bike-shedding.

The same to you!

Spencer

>
>
> Mauro
>
>
>
> _____________________
>
> Mauro Cociglio
>
> TIM - CT.TA.EI
>
> Via G. Reiss Romoli, 274
>
> 10148 - Torino (Italy)
>
> Tel.: +390112285028
>
> Mobile: +393357669751
>
> _____________________
>
>
>
>
>
> *Da:* Ian Swett <ianswett=[email-masked]>
> *Inviato:* mercoledì 8 aprile 2020 21:44
> *A:* Cociglio Mauro <[email-masked]>
> *Cc:* [email-masked]; Lubashev, Igor <[email-masked]>;
> [email-masked]; [email-masked]; Riccardo
> Sisto <[email-masked]>; IETF IPPM WG ([email-masked]) <
> [email-masked]>
> *Oggetto:* [EXT] Re: [ippm] New QUIC Packet Loss Measurement
> (draft-cfb-ippm-spinbit-measurements)
>
>
>

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 9 Apr 2020 14:15:52 -0400

> I'm mostly serious, and only partially bike-shedding.

"S" is already taken in the QUIC packet header: it's the Spin Bit [1].

A possible alternative to "Q" is "W," as in "wave" (the square wave).

  - Dmitri.

1. https://tools.ietf.org/html/draft-ietf-quic-transport-27#section-17.3

---

From: Dmitri Tikhonov <[email-masked]>
Date: Thu, 9 Apr 2020 16:08:12 -0400

Hello,

The proposed "Reflected Square" loss detection mechanism [1] is
not an improvement over the existing "L-bit" mechanism [2] by Akamai
and Orange.  This is because the Reflected Square is missing a feature:
it does not provide downstream loss information in all scenarios.

To calculate downstream loss, the Reflected Square probe must be able
to observe traffic in both directions.  The problem with this is that the
opposite flow may not go through the tapped node!  This means that for
these cases, such probe is useless as it cannot figure out whether the
loss is upstream or downstream.

Contrast this with an L-bit probe:  Given a single flow, it measures
upstream, downstream, and end-to-end loss.  (Obviously, Reflected Square
also can't measure end-to-end loss using a single flow.)

The Reflected Square proposal is a severe degradation in functionality
compared with the L-bit proposal.

  - Dmitri.

1. https://tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-01
2. https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-03

---

From: "Bulgarella Fabio (Guest)" <[email-masked]>
Date: Fri, 10 Apr 2020 11:13:11 +0000

Hello Ian,

I confirm that the implementation of our solution is pretty simple.

Follows the link to our implementation in quic-go; the last two functions of linked source file handle the logic needed to process Q and R bits.

https://github.com/fabiobulgarella/quic-go/blob/tim-pl2-qr/packet_packer.go

Thanks and happy easter,

Fabio B.

Telecom Italia (TIM)

________________________________
Da: ippm <[email-masked]> per conto di Ian Swett <ianswett=[email-masked]>
Inviato: mercoledì 8 aprile 2020 21:43
A: Cociglio Mauro
Cc: [email-masked]; IETF IPPM WG ([email-masked]); Lubashev, Igor; [email-masked]; [email-masked]; Riccardo Sisto
Oggetto: [EXT] Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Ian

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)

[https://img.tim.it/sdr/maild/mail_dip_680x190_Banner_DISNEY_TIMVISION_LANCIO_v2.jpg]<https://on.tim.it/banner-mail-dip>
Questo messaggio e i suoi allegati sono indirizzati esclusivamente alle persone indicate. La diffusione, copia o qualsiasi altra azione derivante dalla conoscenza di queste informazioni sono rigorosamente vietate. Qualora abbiate ricevuto questo documento per errore siete cortesemente pregati di darne immediata comunicazione al mittente e di provvedere alla sua distruzione, Grazie.

---

From: Cociglio Mauro <[email-masked]>
Date: Fri, 10 Apr 2020 13:41:51 +0000

I agree with you Spencer but unfortunately S-bit already stands for Spin bit in the QUIC header.
Sorry ;‑)

Mauro

Da: Spencer Dawkins at IETF <[email-masked]>
Inviato: giovedì 9 aprile 2020 19:28
A: Cociglio Mauro <[email-masked]>
Cc: Ian Swett <[email-masked]>; [email-masked]; IETF IPPM WG ([email-masked]) <[email-masked]>; Lubashev, Igor <[email-masked]>; [email-masked]; [email-masked]; Riccardo Sisto <[email-masked]>
Oggetto: Re: [EXT] Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

18

On Thu, Apr 9, 2020 at 11:22 AM Cociglio Mauro <mauro.cociglio=[email-masked]<mailto:[email-masked]>> wrote:
Thanks Ian for your email.
We intend to proceed with the privacy analysis, which is the same as that of the Ferrieux-Hamchaoui draft for the part of the Q-bit (and the R-bit should be very similar to Q-bit  in terms of privacy).
R-bit stands for “Reflected square bit”, not for “Retransmission bit”, but of course the letter can be changed.

If you need to change the letter, "S-bit" for "Square bit" would be one of the best field names I've seen in a while (sooner or later, every QUIC developer will ask, "what's a square bit?").

The same to you!

Spencer

Mauro

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)
Tel.: +390112285028
Mobile: +393357669751
_____________________

Da: Ian Swett <ianswett=[email-masked]<mailto:[email-masked]>>
Inviato: mercoledì 8 aprile 2020 21:44
A: Cociglio Mauro <[email-masked]<mailto:mauro.cociglio@telecomitalia..it>>
Cc: [email-masked]<mailto:[email-masked]>; Lubashev, Igor <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; Riccardo Sisto <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG ([email-masked]<mailto:[email-masked]>) <[email-masked]<mailto:[email-masked]>>
Oggetto: [EXT] Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Ian

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)

ippm mailing list

---

From: "Bulgarella Fabio (Guest)" <[email-masked]>
Date: Fri, 10 Apr 2020 09:50:56 +0000

Hello Dmitri,
thanks for your analysis.

Without doing any considerations about the internal mechanisms of our algorithms, I would say:

- our solution [1], observing just one direction, gives information about the loss experienced within the operator's domain of the observed channel (exploiting the Qbit) and the E2E loss of the opposite channel (using Rbit and Qbit together);
- your solution [2], observing just one direction, gives information about the loss experienced within the operator's domain plus the E2E loss of the observed channel (so no information about the opposite channel at all);
- both solutions needs to observe both directions to evaluate the whole client-server connection.

Being these mechanisms intended for connection-oriented loss evaluation, and therefore to evaluate the entire user experience, we prefer a solution that, observing just one direction, produces measurements related to both directions, approaching the intent for which this kind of explicit E2E measurements were born.

Fabio B.
Telecom Italia (TIM)

1. https://tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-01#section-7
2. https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-03

________________________________________
Da: Dmitri Tikhonov <[email-masked]>
Inviato: giovedì 9 aprile 2020 22:08
A: Cociglio Mauro
Cc: [email-masked]; Giuseppe Fioccola; Bulgarella Fabio (Guest); Lubashev, Igor; [email-masked]; [email-masked]; Nilo Massimo; 'Riccardo Sisto'; IETF IPPM WG ([email-masked])
Oggetto: [EXT] Reflected Square can't measure downstream loss [Was: New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)]

Hello,

  - Dmitri.

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 10 Apr 2020 15:45:18 -0500

> I agree with you Spencer but unfortunately S-bit already stands for Spin

I was the kind of AD who followed the discussions of the Spin bit with
great interest (I'm trying to remember which IETF meeting I introduced
myself as "the Spin bit AD"), but didn't notice that detail. Thanks to you
and Dmitri for pointing it out :-)

Best,

Spencer

> Mauro
>
>
>
> *Da:* Spencer Dawkins at IETF <[email-masked]>
> *Inviato:* giovedì 9 aprile 2020 19:28
> *Cc:* Ian Swett <[email-masked]>; [email-masked]; IETF IPPM WG (
> [email-masked]) <[email-masked]>; Lubashev, Igor <[email-masked]>;
> Sisto <[email-masked]>
> *Oggetto:* Re: [EXT] Re: [ippm] New QUIC Packet Loss Measurement
> *A:* Cociglio Mauro <[email-masked]
> <mauro.cociglio@telecomitalia..it>>

---

From: Christian Huitema <[email-masked]>
Date: Fri, 10 Apr 2020 13:55:58 -0700

On 4/10/2020 1:45 PM, Spencer Dawkins at IETF wrote:
> On Fri, Apr 10, 2020 at 8:41 AM Cociglio Mauro
> <[email-masked]
> <mailto:[email-masked]>> wrote:
>
> myself as "the Spin bit AD"), but didn't notice that detail. Thanks to
> you and Dmitri for pointing it out :-)

It seems logical to spin-bit this.

-- Christian Huitema

---

From: "Lubashev, Igor" <[email-masked]>
Date: Tue, 21 Apr 2020 16:31:44 +0000

I like the QR draft from the perspective that it mostly works (and lots of credit is due for things that work).

I have a few concerns, though.

1. The most trivial one is that QR is actually harder to implement correctly for endpoints, given the tricky processing required to identify when a marking interval has ended.  Experimental data that Akamai & Orange compiled shows that there are networks that do ECMP among links with significantly different latencies.  They will cause a lot more reordering to a burst of packets than the recommended “observe 5 packets (in a row?) of the opposite markings” rule can address.  You might need to potentially defer the decision on where exactly the marking period boundary is until you’ve seen almost half of the next marking period.  Requiring every endpoint to implement this right is tough.

2. Since the most important direction (at least for downloads and media streaming) is server-to-client, to get this signal with QR, you need both endpoints implementing the logic, with the heaviest lift on the clients (see above).  For QL, a client that does not want to implement QL logic but is ok with the loss data reporting can just update the TP and header protection mask – all the lift is on the server side.

3. I am concerned with the quality of the signal you get from QR.  While the upstream loss signal is strong (same Q-bit), the downstream/end-to-end loss signal is weak and is likely to result in very delayed and inconclusive signal.  Google’s experience (as I understand it) and protocol research (https://erg.abdn.ac.uk/~downloads/ackscaling.pdf) point to the benefit of ACK-ing only 1:10 packets after the initial slow start (draft-fairhurst-quic-ack-scaling).  Since such behavior has a beneficial impact on networks, mobile devices, and endpoint cpu, and no negative impact on the protocol performance, I expect it to be in common use.  With 1:10 ACK ratio, R-bit signal is significantly delayed as it is lagging the loss by at least 1400*(Q=64)*2*10=1.8MB of data transferred, which is at least 38 RTTs for a 15Mbps stream over a 25ms link (L-bit signal is delayed by an RTO).

4. QR scheme only works for symmetric path segments, when the observer can capture both directions of the flow. You also cannot compute end-to-end loss in any direction, unless both endpoints implement the entire QR algorithm.  In contrast, QL works for single-direction observers, and whichever endpoint implements QL algorithm ensures end-to-end loss signal for the packets it sends, and the quality of that loss signal solely depends on the quality of that endpoint’s implementation (not the quality of the other endpoint’s implementation).  QL seems more deployable.

  *   Igor

From: Ian Swett <[email-masked]>
Sent: Wednesday, April 8, 2020 3:44 PM
To: Cociglio Mauro <mauro.cociglio=[email-masked]>
Cc: [email-masked]; Lubashev, Igor <[email-masked]>; [email-masked]; [email-masked]; Riccardo Sisto <[email-masked]>; IETF IPPM WG ([email-masked]) <[email-masked]>
Subject: Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Ian

We submitted to IPPM WG a draft where we described a new 2 bit packet loss methodology (https://tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-01<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dcfb-2Dippm-2Dspinbit-2Dmeasurements-2D01&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=UeTW_fLvxHy_ewVvm0y6kIqZoXUR8PYWgxAOxmQScYw&e=>).
The measurement of packet loss is under discussion in QUIC WG and our draft introduces two alternatives about it.
The first one is a spin-bit dependent signal and uses a single bit. The second one, described in the following linked slides, is a standalone solution based on a two bits loss signal and on alternate marking (RFC8321)..
This last methodology improves, in our opinion, the algorithm proposed by Orange and Akamai described in https://tools.ietf.org/html/draft-ferrieuxhamchaoui-quic-lossbits-03<https://urldefense.proofpoint.com/v2/url?u=https-3A__tools.ietf.org_html_draft-2Dferrieuxhamchaoui-2Dquic-2Dlossbits-2D03&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=V649Uqu8mfuBWvMRBTUkpEWfpEqlLfufGs67XXzvNFo&e=>.

You can find the slides, we prepared for the IPPM interim meeting, at the following link: https://github.com/ietf-ippm/meeting-materials/blob/aa486f7ead28b9f55c5ef499e5c9ed33ab06daf9/ietf107-virtual/Slides/07-spinbit-measurements.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ietf-2Dippm_meeting-2Dmaterials_blob_aa486f7ead28b9f55c5ef499e5c9ed33ab06daf9_ietf107-2Dvirtual_Slides_07-2Dspinbit-2Dmeasurements.pdf&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=3l6t3FYQn0iz_xX0sxNFKUnh7bwYFMOpQ0z_zNhXHpg&e=>

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)

[https://img.tim.it/sdr/maild/mail_dip_680x190_Banner_DISNEY_TIMVISION_LANCIO_v2.jpg]<https://urldefense.proofpoint.com/v2/url?u=https-3A__on.tim.it_banner-2Dmail-2Ddip&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=z55QpRg48UYrhUvomUNAlkMCUxdQT20Mz1VhZkNrLQI&e=>

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_mailman_listinfo_ippm&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=YF4GnFEgwmSakC8Bv_763sIZZR3g8t7wdsuiOl6C8fU&e=>

---

From: "Bulgarella Fabio (Guest)" <[email-masked]>
Date: Wed, 22 Apr 2020 19:11:38 +0000

Hello Igor,
here some thoughts about your concerns.

1. If big differences in delay are present between two parallel paths on which the same connection is divided, this is certainly an unwanted behavior, because it could create problems to the protocol itself even before damaging the performance measurements. As a network operator, we try to avoid such scenarios. However, our mechanism works correctly in most cases and is still resistant to normal reordering. Moreover, with a strong amount of reordering the Qbit (and the SpinBit) would not work either.

2. It's true that your mechanism allows the measurement of E2E and upstream loss of the observed direction. However, these measurement methodologies are born with the idea of providing information on the performance of the entire connection, as was done with the SpinBit. Actually with our mechanism, when observing a single direction, we still are able to measure the E2E loss of the opposite channel (using the R-bit).

3. The ACK ratio of 1:10 should only occur in the absence of loss. By introducing loss, the ratio between ACK and data is certainly better. So the quality of our signal improves just when it is needed; we don't care if we get delayed measurements when everything is working great. This behavior is confirmed by our laboratory tests where we have seen that loss can be easily detected in an extremely accurate manner even in the case of highly unbalanced flows.

L-Bit has an RTO delay but to evaluate an accurate loss ratio you have still to observe a significant number of packets.

4. First of all, our solution is not dependent on any specific protocol implementation unlike the QL and therefore there shouldn’t be problems related to the quality of the implementation. In other words, our portion of code is always the same, in every protocol implementation.
Secondly, as anticipated in point 2, by observing only one direction we can measure the losses in the operator's domain (Qbit) and the end-to-end in the opposite direction (Rbit). So, our solution does work also for asymmetric path segments. We can also place two indipendent observers wherever on the two directions and they will still produce coherent E2E measures.

The QL solution does not give any information about the opposite direction, it only measures what it observes, that is the current direction. Having two independent measures, one per direction, can be an advantage in some aspects, but a disadvantage for others.

Best regards,
Fabio B.

________________________________
Da: ippm <[email-masked]> per conto di Lubashev, Igor <ilubashe=[email-masked]>
Inviato: martedì 21 aprile 2020 18:31
A: Ian Swett; Cociglio Mauro
Cc: Riccardo Sisto; [email-masked]; [email-masked]; [email-masked]; IETF IPPM WG ([email-masked])
Oggetto: [EXT] Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

  *   Igor

Ian

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)

---

From: Dmitri Tikhonov <[email-masked]>
Date: Wed, 22 Apr 2020 16:05:31 -0400

Hello,

On Wed, Apr 22, 2020 at 07:11:38PM +0000, Bulgarella Fabio (Guest) wrote:
> 4. First of all, our solution is not dependent on any specific protocol
> implementation unlike the QL and therefore there shouldn’t be problems
> related to the quality of the implementation. In other words, our
> portion of code is always the same, in every protocol implementation.

I do not understand this part.  Problems in which protocol
implementation do you mean -- the loss measurement protocol
or some other protocol?  If the former, I do not understand
what you mean by "our portion of code:"  Each implementer will
write his own code, with his own bugs.

> Secondly, as anticipated in point 2, by observing only one direction
> we can measure the losses in the operator's domain (Qbit) and the
> end-to-end in the opposite direction (Rbit). So, our solution does work
> also for asymmetric path segments.

I already covered this, but I believe it can bear restating:
Besides loss measurement, a probe should aid to locate the source
of the loss: upstream or downstream.  A single QR probe cannot do it.
Do you not agree that locating loss source is one of the probe's goals?

  - Dmitri.

---

From: "Bulgarella Fabio (Guest)" <[email-masked]>
Date: Thu, 23 Apr 2020 13:27:23 +0000

Hello Dmitri,

> write his own code, with his own bugs.

Sorry, I didn't explain this very well. What I meant was that the implementation of our solution is "untied" from the network protocol one.
In QR logic there are no dependencies on internal protocol events (packet declared lost) that can differ in different implementations.
Changes in the network protocol implementation will not require any changes in our logic. The same cannot be said for your solution.

> Do you not agree that locating loss source is one of the probe's goals?

It is more interesting to know if a problem occurs in our domain or outside. And this is done by the Qbit (With 2 probes).
However, it is not uncommon to be able to intercept the whole traffic of a connection by observing a single point.
In this case, all the necessary measurements can be made.

Fabio B.

________________________________________
Da: Dmitri Tikhonov <[email-masked]>
Inviato: mercoledì 22 aprile 2020 22:05
A: Bulgarella Fabio (Guest)
Cc: Lubashev, Igor; Ian Swett; Cociglio Mauro; [email-masked]; Riccardo Sisto; [email-masked]; [email-masked]; IETF IPPM WG ([email-masked])
Oggetto: [EXT] Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Hello,

  - Dmitri.

---

From: "Lubashev, Igor" <[email-masked]>
Date: Thu, 23 Apr 2020 16:41:27 +0000

Fabio, thanks for the reply.

  *   "Moreover, with a strong amount of reordering the Qbit (and the SpinBit) would not work either."

Strong reordering exists.  That's why we are recommending the minimum Q to be 64.  This comes from real data, not lab data.  If 5 packets were enough to tell the end of marking periods, we would had recommended minimum Q to be 8.

  *   R-bit signal delay.

R signal is still going to be significantly delayed, especially with slower connections (not every connection is going to be streaming media at a sustained rate of 15Mbps).  So if for a 15Mbps average-rate connection you can expect to see R-signal with a delay between 8-RTTs (1:2 ACK) and 38-RTTs (1:10 ACK), the delay increases to 24-RTTs and 114-RTTs for a 5Mbps average-rate connection.

  *   R-bit signal quality.

L-bit signal provides a crisp shape of the loss - whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch).  By contrast, R-signal only indicates an average loss over 2 to 10 Q-bit marking periods.  The share of the loss is erased.

I think for that alone, L-signal is more useful for troubleshooting.

  *   L-bit signal depends on each implementation's determination of when to "declare a packet lost" (which may differ for each implementation), while R-signal depends only on more generic behaviors.

This is an important difference, and it seems accurate on its surface.  In reality, every QUIC implementation will keep track of packets and declare them lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.  Likewise, every QUIC implementation will send some ACKs, but their timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend on the "marking period detection" implementation quality and "Q-signal averaging" implementation details.

  *   End-to-end loss: "We can also place two independent observers wherever on the two directions and they will still produce coherent E2E measures."
  *   End-to-end loss: "by observing only one direction we can measure the [...] the end-to-end in the opposite direction (Rbit).

Correct, as long as both endpoints implement the entire QR scheme (even if this is an unusual setup - measuring path A requires observing path B).

Also, accurate accounting with QR scheme requires correlation of both directions, with the R-bit direction significantly lagging the Q-bit direction.  It should be understood how this scheme behaves in cases of changing Connection IDs.

Best wishes,

  *   Igor

From: Bulgarella Fabio (Guest) <fabio.bulgarella=[email-masked]>
Sent: Wednesday, April 22, 2020 3:12 PM
To: Lubashev, Igor <[email-masked]>; Ian Swett <[email-masked]>; Cociglio Mauro <[email-masked]>
Cc: Riccardo Sisto <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; IETF IPPM WG ([email-masked]) <[email-masked]>
Subject: Re: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

Hello Igor,

4. First of all, our solution is not dependent on any specific protocol implementation unlike the QL and therefore there shouldn't be problems related to the quality of the implementation. In other words, our portion of code is always the same, in every protocol implementation.
Secondly, as anticipated in point 2, by observing only one direction we can measure the losses in the operator's domain (Qbit) and the end-to-end in the opposite direction (Rbit). So, our solution does work also for asymmetric path segments. We can also place two indipendent observers wherever on the two directions and they will still produce coherent E2E measures.

Best regards,
Fabio B.

1. The most trivial one is that QR is actually harder to implement correctly for endpoints, given the tricky processing required to identify when a marking interval has ended.  Experimental data that Akamai & Orange compiled shows that there are networks that do ECMP among links with significantly different latencies.  They will cause a lot more reordering to a burst of packets than the recommended "observe 5 packets (in a row?) of the opposite markings" rule can address.  You might need to potentially defer the decision on where exactly the marking period boundary is until you've seen almost half of the next marking period.  Requiring every endpoint to implement this right is tough.

2. Since the most important direction (at least for downloads and media streaming) is server-to-client, to get this signal with QR, you need both endpoints implementing the logic, with the heaviest lift on the clients (see above).  For QL, a client that does not want to implement QL logic but is ok with the loss data reporting can just update the TP and header protection mask - all the lift is on the server side.

3. I am concerned with the quality of the signal you get from QR.  While the upstream loss signal is strong (same Q-bit), the downstream/end-to-end loss signal is weak and is likely to result in very delayed and inconclusive signal.  Google's experience (as I understand it) and protocol research (https://erg.abdn.ac.uk/~downloads/ackscaling.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__erg.abdn.ac.uk_-7Edownloads_ackscaling.pdf&d=DwMF-g&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=u3hS0QnHLn0-P-Lmy8-Yhm7L0G310RjwTQfw2XLvfJk&s=8qb_Zcm3YASCVrMZiZ5Le_3_nPxd8AC7bUbVwXl4L9I&e=>) point to the benefit of ACK-ing only 1:10 packets after the initial slow start (draft-fairhurst-quic-ack-scaling).  Since such behavior has a beneficial impact on networks, mobile devices, and endpoint cpu, and no negative impact on the protocol performance, I expect it to be in common use.  With 1:10 ACK ratio, R-bit signal is significantly delayed as it is lagging the loss by at least 1400*(Q=64)*2*10=1.8MB of data transferred, which is at least 38 RTTs for a 15Mbps stream over a 25ms link (L-bit signal is delayed by an RTO).

4. QR scheme only works for symmetric path segments, when the observer can capture both directions of the flow. You also cannot compute end-to-end loss in any direction, unless both endpoints implement the entire QR algorithm.  In contrast, QL works for single-direction observers, and whichever endpoint implements QL algorithm ensures end-to-end loss signal for the packets it sends, and the quality of that loss signal solely depends on the quality of that endpoint's implementation (not the quality of the other endpoint's implementation).  QL seems more deployable.

  *   Igor

Ian

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)

[Image removed by sender.]<https://urldefense.proofpoint.com/v2/url?u=https-3A__on.tim.it_banner-2Dmail-2Ddip&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=e_BQ3UDNSWkXu_ID1rkGv3CCZs89sezu-6Ibkvltf7Y&s=z55QpRg48UYrhUvomUNAlkMCUxdQT20Mz1VhZkNrLQI&e=>

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks..

---

From: Dmitri Tikhonov <[email-masked]>
Date: Mon, 27 Apr 2020 11:29:58 -0400

Hello Fabio,

On Thu, Apr 23, 2020 at 01:27:23PM +0000, Bulgarella Fabio (Guest) wrote:
> It is more interesting to know if a problem occurs in our domain or
> outside. And this is done by the Qbit (With 2 probes).

Consider the following setup:

    Client ... (OpA) ... (OpB) ... (OpC) ... Server

We are OpB, and Client's and Server's lawyers are banging at our door
with a legally binding SLA signed with our blood.  What do you think is
a realistic way of getting rid of the situation:

    a) Here are captures that prove that loss is in OpC, or in your
       premises, Mr Server.

    b) Here are captures that have no losses in OpA or OpB.  Have a
       good day.

Since a unidirectional QR capture does not include any evidence of
loss downstream, it would be dismissed by the Server lawyers as just
not capturing the right traffic.  An equivalent packet dump with TCP
would include evidence of loss in the form of packet retransmission,
and so would QL scheme.

Simultaneous captures by different organizations are often hard to
coordinate, especially for problems that do not reproduce all the time.
At any given point of the investigation, you depend only on your own
local resources and need to be able to provide evidence that makes sense
to everybody.

> However, it is not uncommon to be able to intercept the whole traffic
> of a connection by observing a single point.  In this case, all the
> necessary measurements can be made.

How common is "not uncommon:" 90%? 50%?  Here is a study that shows how
common symmetric flows are: out of six observations of four different
Internet links, the percentage of symmetric flows was 57%, 7%, 2%, 3%,
2%, and 3%:

  https://www.caida.org/research/traffic-analysis/asymmetry/

The way routing works is asymmetric in essence, as the destination decides
what it announces, which leads to the way to reach it.  When you revert
the arrow, the source cannot but just follow the routes.  The result is
a high prevalence of asymmetric routing.

  - Dmitri.

---

From: "Bulgarella Fabio (Guest)" <[email-masked]>
Date: Tue, 28 Apr 2020 13:50:57 +0000

​

Hello Igor,
thanks for your reply.

This exchange of e-mails is very useful to understand which of the two solutions has the best balance between positive and negative aspects, at least in the measurement scenarios of interest for telecommunication operators. The important thing is that one of the two solution is chosen and a packet loss feature is finally included in the Quic protocol (after the necessary checks in terms of privacy and security). However, we can already say, from the point of view of Telecom Italia and Orange, that the Qbit is a fixed point of this type of measures. It remains to decide which is better between L-bit and R-bit.

First of all, let's remember the reasons that led us to design the Rbit.
We found some weaknesses in the Lbit:

- the LBit logic is strictly tied to the protocol implementation.
For example in LSQUIC there are 6 different hook points where this variable is set. QL performance mechanism logic is intertwined with the protocol logic. So, the concern here is that future changes in the protocol or in new implementations need a careful analysis of the code to avoid different behaviors and different measurement results;

- loss of packets carrying only ACK frames (are detected?).
We saw in the Montreal demo (using Ericsson SpinDump tool), the last July, that if Quic packets only contain ACK frames the Lbit is not set. Actually, during a file download we inserted losses in the upload direction that were not detected by the system. Alexandre confirmed this behavior. Have you changed something since then?

- the loss measurement signal is inaccurate in case of losses:
The Lbit is a simple signaling protocol. If you miss a Lbit marked packet you miss a signal of loss. If losses are not equally distributed between L-marked and not L-marked packets, it's very likely that the produced measurements are wrong or at least inaccurate. The two approaches are completely different. Rbit works exactly as the Qbit. It counts the lost packets, it's not a signaling. So, as the Qbit, the correct percentage of lost packets is detected.

Now some inline answers to your considerations, even though at this point it would be convenient to organize a comparison involving also other people interested in the topic, maybe in a next IETF meeting. We are also available to carry out comparative tests on the same flows (marking simultaneously with both solutions) using for example the Ericsson SpinDump tool which has already been used in the previous hackathons as QUIC performance measurement tool.

Best Regards,
Fabio B.

> Strong reordering exists.  That’s why we are recommending the minimum Q to be 64. This comes from real data, not lab data.  If 5 packets were enough to tell the end of marking periods, we would had recommended minimum Q to be 8.

The length of the period is not chosen only in function of reordering but also to avoid that burst losses completely cancel one or more periods. However, stimulated by Igor's concerns, we thought of an improvement regarding the tolerance of our system against out of sequence. In a nutshell, it would be possible to intercept reordered packets received after the detection of an incoming "Qbit marking period", using them to update the computed average until the current "Rbit marking period" is completed in generation. After the necessary laboratory tests, we could add this new feature to the next revision of our draft.

> R signal is still going to be significantly delayed, especially with slower connections (not every connection is going to be streaming media at a sustained rate of 15Mbps). So if for a 15Mbps avera…

As already said in the previous mail, based on our tests, this is not a big problem because it's still possible to obtain enough measurements to evaluate the packet loss of a session. Moreover, the measurement delay is shortened when packet loss occurs.

> L-bit signal provides a crisp shape of the loss – whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch). By contrast, R-signal only indicates an average loss over 2 to 10 Q-bit marking periods.  The share of the loss is erased. I think for that alone, L-signal is more useful for troubleshooting.

Although we calculate the E2E loss through an average (only if the reflection is on a slower channel than the signal reception channel), we are still able to accurately calculate the loss using a correction factor (see https: //tools.ietf.org/html/draft-cfb-ippm-spinbit-measurements-00#section-7.4).

> This is an important difference, and it seems accurate on its surface.  In reality, every QUIC implementation will keep track of packets and declare them lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.  Likewise, every QUIC implementation will send some ACKs, but their timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend on the “marking period detection” implementation quality and “Q-signal averaging” implementation details.

See first point in the L-bit weeknesses paragraph. Moreover, the two functions "marking period detection" and "Q-signal averaging", really simple in implementation, are completely detached from the rest of the protocol logic and their behaviors are expressly indicated in the draft.

> Correct, as long as both endpoints implement the entire QR scheme (even if this is an unusual setup – measuring path A requires observing path B). Also, accurate accounting with QR scheme requires correlation of both directions, with the R-bit direction significantly lagging the Q-bit direction.  It should be understood how this scheme behaves in cases of changing Connection IDs.

We do not see the problem of implementing the solution on both endpoints since generally the client, unlike the server, does not have scalability problems. So if you implement it on the server it costs nothing to implement it on the client as well.

________________________________
Da: Lubashev, Igor <ilubashe=[email-masked]>
Inviato: giovedì 23 aprile 2020 18:41
A: Bulgarella Fabio (Guest); Ian Swett; Cociglio Mauro
Cc: Riccardo Sisto; [email-masked]; [email-masked]; [email-masked]; IETF IPPM WG ([email-masked])
Oggetto: [EXT] RE: [ippm] New QUIC Packet Loss Measurement (draft-cfb-ippm-spinbit-measurements)

  *   “Moreover, with a strong amount of reordering the Qbit (and the SpinBit) would not work either.”

Strong reordering exists.  That’s why we are recommending the minimum Q to be 64.  This comes from real data, not lab data.  If 5 packets were enough to tell the end of marking periods, we would had recommended minimum Q to be 8.

  *   R-bit signal delay.

  *   R-bit signal quality.

L-bit signal provides a crisp shape of the loss – whether it is random loss, or packets are lost of batches (and how many packets are in a loss batch).  By contrast, R-signal only indicates an average loss over 2 to 10 Q-bit marking periods.  The share of the loss is erased.

  *   L-bit signal depends on each implementation’s determination of when to “declare a packet lost” (which may differ for each implementation), while R-signal depends only on more generic behaviors.

This is an important difference, and it seems accurate on its surface.  In reality, every QUIC implementation will keep track of packets and declare them lost for the purposes of data retransmission and/or congestion control.  The timing and details of when packets are declared lost (L-bit signal) will vary.  Likewise, every QUIC implementation will send some ACKs, but their timing and frequency (R-signal) will vary.  R-signal quality will also heavily depend on the “marking period detection” implementation quality and “Q-signal averaging” implementation details.

  *   End-to-end loss: “We can also place two independent observers wherever on the two directions and they will still produce coherent E2E measures.”
  *   End-to-end loss: “by observing only one direction we can measure the […] the end-to-end in the opposite direction (Rbit).

Correct, as long as both endpoints implement the entire QR scheme (even if this is an unusual setup – measuring path A requires observing path B).

Best wishes,

  *   Igor

Hello Igor,

Best regards,
Fabio B..

3. I am concerned with the quality of the signal you get from QR.  While the upstream loss signal is strong (same Q-bit), the downstream/end-to-end loss signal is weak and is likely to result in very delayed and inconclusive signal.  Google’s experience (as I understand it) and protocol research (https://erg.abdn.ac.uk/~downloads/ackscaling.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__erg.abdn.ac.uk_-7Edownloads_ackscaling.pdf&d=DwMF-g&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=u3hS0QnHLn0-P-Lmy8-Yhm7L0G310RjwTQfw2XLvfJk&s=8qb_Zcm3YASCVrMZiZ5Le_3_nPxd8AC7bUbVwXl4L9I&e=>) point to the benefit of ACK-ing only 1:10 packets after the initial slow start (draft-fairhurst-quic-ack-scaling).  Since such behavior has a beneficial impact on networks, mobile devices, and endpoint cpu, and no negative impact on the protocol performance, I expect it to be in common use.  With 1:10 ACK ratio, R-bit signal is significantly delayed as it is lagging the loss by at least 1400*(Q=64)*2*10=1.8MB of data transferred, which is at least 38 RTTs for a 15Mbps stream over a 25ms link (L-bit signal is delayed by an RTO).

  *   Igor

Ian

Best regards.

_____________________
Mauro Cociglio
TIM - CT.TA.EI
Via G. Reiss Romoli, 274
10148 - Torino (Italy)

This e-mail and any attachments is confidential and may contain privileged information intended for the addressee(s) only. Dissemination, copying, printing or use by anybody else is unauthorised. If you are not the intended recipient, please delete this message and any attachments and advise the sender by return e-mail, Thanks...