Subject: [ippm] 1st AD review of rfc8889bis

From: Martin Duke <[email-masked]>
Date: Wed, 1 Dec 2021 15:12:08 -0800

Thanks for this doc. Here are some high-level comments in no particular
order:

- My perception is that 8889 is an extension of the concepts in 8321, with
some common elements only defined in 8321, so 8321 should be a normative
reference.

- I am confused as to whether ECMP is covered by 8321 or 8889. The 8321
section on reordering talks about ECMP as a cause, which leads me to
believe it's to be covered; but in the intro 8889 claims this use case. I
suppose it depends on how much you're localizing the measurement?

- The intro claims that 8321 covers BUM traffic. Isn't multicast a classic
point-to-multipoint situation? Indeed, doesn't multicast involve packet
replication, which inherently breaks the packet loss mechanism?

- 8889 brings up IP fragmentation, but this seems like a general concern
for 8321 too. Although it probably ought to be written out in 8321 and not
here, this seems under-addressed here. Some questions:
 (a) if a router has to fragment a packet, should it replicate the marking
on all the fragments?
 (b) at a measurement point, is a packet "lost" if any fragment is lost?
what if the fragments entered separately in the network under test?
 (c) What if the marking location is not replicated across the fragments
(e.g. it's in the transport header)?

- Like 8321, there is a general problem with not addressing the fixed-N
marking scheme at all ("The equation is applied on a per-time-interval
basis...")

- There is a hidden requirement here that there be a measurement point at
all possible egress points. Best to mention that.

- I don't understand the L determination when there are multiple marking
nodes (Sec 7). "The source measurement intervals can be of different
lengths and with different offsets" Some questions:
(a) What is a "source measurement interval"? Does it have anything to do
with L?
(b) So let's say that source nodes start marking at arbitrary times. Isn't
the "offset" then directly a function of L? And if so, how can we scale L
based on the offset m?
(c) if the idea is that there is a clock time when marking should begin,
then why isn't the value of A [clock skew] embedded in d sufficient?

- I can't understand the purpose of "delay measurements on a single-packet
basis" in Section 8. I cannot grok the context of Section 8.2 at all. What
are these techniques trying to achieve? This reads like an informational
exploration of options rather than a normative standard. 8321 has detailed
examples of sample measurements and how you derive a measurement; something
like that would be very useful here.

Martin

---

From: Giuseppe Fioccola <[email-masked]>
Date: Thu, 2 Dec 2021 11:46:50 +0000

Hi Martin,
Thank you very much for your detailed review.
Pease find my answers inline tagged as [GF].

Best Regards,

Giuseppe

From: Martin Duke <[email-masked]>
Sent: Thursday, December 2, 2021 12:12 AM
To: [email-masked]
Cc: IETF IPPM WG <[email-masked]>
Subject: 1st AD review of rfc8889bis

Thanks for this doc. Here are some high-level comments in no particular order:

- My perception is that 8889 is an extension of the concepts in 8321, with some common elements only defined in 8321, so 8321 should be a normative reference.

[GF]: Sure, I will fix it in the next version.

- I am confused as to whether ECMP is covered by 8321 or 8889. The 8321 section on reordering talks about ECMP as a cause, which leads me to believe it's to be covered; but in the intro 8889 claims this use case. I suppose it depends on how much you're localizing the measurement?

[GF]: In RFC8321 it was mentioned ECMP just to explain that reordering can be caused by ECMP paths, but the monitoring was assumed point-to-point in the whole document. I can omit that sentence in RFC8321 if it is confusing. While, RFC8889 covers the general case of multipoint-to-multipoint unicast flows (including ECMP). The BUM traffic was not covered in the original RFC8889, but Alternate Marking can be applied in case of multicast. Maybe, in a later version, we can add a short section in one of the two bis documents to explain how to monitor multicast traffic by simply applying RFC8321 link by link.

- The intro claims that 8321 covers BUM traffic. Isn't multicast a classic point-to-multipoint situation? Indeed, doesn't multicast involve packet replication, which inherently breaks the packet loss mechanism?

[GF]: With the “clustered” approach of RFC8889 you cannot monitor BUM traffic but only multipoint-to-multipoint unicast flows. For BUM traffic, you can apply the basic method of RFC8321 but link by link and therefore split the multicast flow into separate unicast point-to-point links. As said, a new section on BUM traffic could be useful to clarify this point.

- 8889 brings up IP fragmentation, but this seems like a general concern for 8321 too. Although it probably ought to be written out in 8321 and not here, this seems under-addressed here. Some questions:
 (a) if a router has to fragment a packet, should it replicate the marking on all the fragments?
 (b) at a measurement point, is a packet "lost" if any fragment is lost? what if the fragments entered separately in the network under test?
 (c) What if the marking location is not replicated across the fragments (e.g. it's in the transport header)?

[GF]: Agree, few considerations can be included in RFC8321. The assumption in RFC8889 was that the fragmentation can be managed if it happens outside the portion of the network that is monitored.
Regarding your questions:
a) both possibilities could be considered. If you do not replicate marking you may have less problems to apply the methodology but you are missing fragments and possible losses.
b) yes, if we mark all the fragments. If the fragments enter separately it is not a problem since they behave as normal packets (we are assuming fragmentation and reassembly outside the domain).
c) if the marking is not applied to all the fragments, the fragmentation could also happen within the domain but the monitoring is not complete since you are missing fragments.

- Like 8321, there is a general problem with not addressing the fixed-N marking scheme at all ("The equation is applied on a per-time-interval basis...")

[GF]: We can make the same assumption here as per RFC8321.

- There is a hidden requirement here that there be a measurement point at all possible egress points. Best to mention that.

[GF]: Agree, I will do.

- I don't understand the L determination when there are multiple marking nodes (Sec 7). "The source measurement intervals can be of different lengths and with different offsets" Some questions:
(a) What is a "source measurement interval"? Does it have anything to do with L?
(b) So let's say that source nodes start marking at arbitrary times. Isn't the "offset" then directly a function of L? And if so, how can we scale L based on the offset m?
(c) if the idea is that there is a clock time when marking should begin, then why isn't the value of A [clock skew] embedded in d sufficient?

[GF]: For a multipoint-to-multipoint flow, you may have multiple marking nodes, so there is an additional contribution to take into account. Indeed, the marking nodes may apply the markings at different moments and this time offset does not depend from the clock error or from the network delay but it is an intrinsic delay of the router.
Regarding your questions:
a) it is the marking period. I will replace it to be aligned with the terminology of the rest of the document.
b) yes, there is the formula: L - 2m - 2d > 0
c) m is not given by clock skew, indeed A is already included in d.

- I can't understand the purpose of "delay measurements on a single-packet basis" in Section 8. I cannot grok the context of Section 8.2 at all. What are these techniques trying to achieve? This reads like an informational exploration of options rather than a normative standard. 8321 has detailed examples of sample measurements and how you derive a measurement; something like that would be very useful here.

[GF]: I agree with you. Indeed, in my first local revision of the document I reduced a lot these sections. As you noticed, it is only an exploration that it is ok for an experimental document but less relevant for a PS.

Martin

---

From: Martin Duke <[email-masked]>
Date: Thu, 2 Dec 2021 11:22:57 -0800

On Thu, Dec 2, 2021 at 3:46 AM Giuseppe Fioccola <
[email-masked]> wrote:

>
> *From:* Martin Duke <[email-masked]>
>
>
> [GF]: In RFC8321 it was mentioned ECMP just to explain that reordering
> can be caused by ECMP paths, but the monitoring was assumed point-to-point
> in the whole document. I can omit that sentence in RFC8321 if it is
> confusing. While, RFC8889 covers the general case of
> multipoint-to-multipoint unicast flows (including ECMP). The BUM traffic
> was not covered in the original RFC8889, but Alternate Marking can be
> applied in case of multicast. Maybe, in a later version, we can add a short
> section in one of the two bis documents to explain how to monitor multicast
> traffic by simply applying RFC8321 link by link.
>
>
>
> [GF]: With the “clustered” approach of RFC8889 you cannot monitor BUM
> traffic but only multipoint-to-multipoint unicast flows. For BUM traffic,
> you can apply the basic method of RFC8321 but link by link and therefore
> split the multicast flow into separate unicast point-to-point links. As
> said, a new section on BUM traffic could be useful to clarify this point.
>

This distinction seems artificial to me; the split between the drafts is
not the type of traffic but the type of traffic combined with the scope of
the measurement. For example, if you're measuring ECMP at the source and
destination, 8321 is fine. Similarly, as you point out BUM traffic for 8321
only works if you combine it with the links. I don't think a substantive
change is needed here but maybe the text could be a bit clearer?

> [GF]: Agree, few considerations can be included in RFC8321. The assumption
> in RFC8889 was that the fragmentation can be managed if it happens outside
> the portion of the network that is monitored.
>
> a) both possibilities could be considered. If you do not replicate marking
> you may have less problems to apply the methodology but you are missing
> fragments and possible losses.
>
> b) yes, if we mark all the fragments. If the fragments enter separately it
> is not a problem since they behave as normal packets (we are assuming
> fragmentation and reassembly outside the domain).
>
> c) if the marking is not applied to all the fragments, the fragmentation
> could also happen within the domain but the monitoring is not complete
> since you are missing fragments.
>

I agree that there are many possibilities, but a standard should either
provide guidance or (at least) work through the implications of each. It
should be more prescriptive; marking nodes and measurement nodes need to
have common expectations about the marking rules, although they can
deterministically depend on whether or not one can mark fragments (which is
obviously deployment-dependent).

I'm happy to be corrected by an expert, but after thinking about it for a
few minutes I would propose the following:
(i) marking nodes MUST mark all fragments if there is a codepoint to use
(i.e. it is in the IP header or encapsulation), as if they were separate
packets.
(ii) routers that fragment packets in a network under test MUST NOT
replicate marks (thus signalling that it was fragmented internally), but
SHOULD mark the first fragment if they have the capability to do so.
(iii) Measurement points MAY simply ignore unmarked fragments and count
marked fragments as full packets. However, if resources allow, Measurement
Points SHOULD make note of marked initial fragments and only increment its
counter if (a) other fragments are also marked, or (b) it observes all
other fragments and they are unmarked.

> [GF]: For a multipoint-to-multipoint flow, you may have multiple marking
> nodes, so there is an additional contribution to take into account. Indeed,
> the marking nodes may apply the markings at different moments and this time
> offset does not depend from the clock error or from the network delay but
> it is an intrinsic delay of the router.
>

I wasn't clear enough about (b). Let's say node A has period L1, and node B
is trying to set its L but is going to start at an arbitrary time. What is
m for node B? It is a uniform distribution over [0, L1]. so L2 has to be >>
L1. But the same is true of A! The only way to disambiguate the blocks is
for each L to be much bigger than all the other Ls, which is of course
impossible.

The only way I can see for this to work is for the marking nodes to agree a
clock time to switch blocks, with fixed L, and then you're only constrained
by the clock skew and delay as in 8321.

To me this is the central problem 8889 is trying to solve, and it's
under-baked. Maybe there's a constraint I'm not seeing?

> [GF]: I agree with you. Indeed, in my first local revision of the document
> I reduced a lot these sections. As you noticed, it is only an exploration
> that it is ok for an experimental document but less relevant for a PS.
>

great, as we discussed earlier let's delete irrelevant text.

---

From: Giuseppe Fioccola <[email-masked]>
Date: Fri, 3 Dec 2021 14:59:46 +0000

Hi Martin,
My answers inline as [GF].

Regards,

Giuseppe

From: Martin Duke <[email-masked]>
Sent: Thursday, December 2, 2021 8:23 PM
To: Giuseppe Fioccola <[email-masked]>
Cc: [email-masked]; IETF IPPM WG <[email-masked]>
Subject: Re: 1st AD review of rfc8889bis

On Thu, Dec 2, 2021 at 3:46 AM Giuseppe Fioccola <[email-masked]<mailto:[email-masked]>> wrote:

From: Martin Duke <[email-masked]<mailto:[email-masked]>>

This distinction seems artificial to me; the split between the drafts is not the type of traffic but the type of traffic combined with the scope of the measurement. For example, if you're measuring ECMP at the source and destination, 8321 is fine. Similarly, as you point out BUM traffic for 8321 only works if you combine it with the links. I don't think a substantive change is needed here but maybe the text could be a bit clearer?

[GF]: I see. It could be enough to add some clarification and explain for example that to monitor BUM traffic it is possible to apply RFC8321 link by link. In this way a reader will have a much clearer picture.

I agree that there are many possibilities, but a standard should either provide guidance or (at least) work through the implications of each. It should be more prescriptive; marking nodes and measurement nodes need to have common expectations about the marking rules, although they can deterministically depend on whether or not one can mark fragments (which is obviously deployment-dependent).

I'm happy to be corrected by an expert, but after thinking about it for a few minutes I would propose the following:
(i) marking nodes MUST mark all fragments if there is a codepoint to use (i.e. it is in the IP header or encapsulation), as if they were separate packets.
(ii) routers that fragment packets in a network under test MUST NOT replicate marks (thus signalling that it was fragmented internally), but SHOULD mark the first fragment if they have the capability to do so.
(iii) Measurement points MAY simply ignore unmarked fragments and count marked fragments as full packets. However, if resources allow, Measurement Points SHOULD make note of marked initial fragments and only increment its counter if (a) other fragments are also marked, or (b) it observes all other fragments and they are unmarked.

[GF]: I fully agree with your proposals. The marking node MUST mark all the fragments except in the case of fragmentation within the network domain; in this case it is suggested to mark only the first packet. The condition about the counters also makes sense so it could be possible to keep track of both marked and unmarked fragments. I can include these considerations in the next version.

I wasn't clear enough about (b). Let's say node A has period L1, and node B is trying to set its L but is going to start at an arbitrary time. What is m for node B? It is a uniform distribution over [0, L1]. so L2 has to be >> L1. But the same is true of A! The only way to disambiguate the blocks is for each L to be much bigger than all the other Ls, which is of course impossible.

The only way I can see for this to work is for the marking nodes to agree a clock time to switch blocks, with fixed L, and then you're only constrained by the clock skew and delay as in 8321.

To me this is the central problem 8889 is trying to solve, and it's under-baked. Maybe there's a constraint I'm not seeing?

[GF]: Yes, the only way is that the marking nodes switch blocks with fixed L based on a clock time. In theory the formula in RFC8321 applies also in case of multipoint network. But we wanted to generalize that formula and let me explain the rationale. Since we have more marking nodes in a multipoint environment, there can be an additional mismatch to consider since different nodes are marking the traffic and the batches get mixed in a multipoint flow. For example, a marking node may apply the marking with a delay because it is overloaded while the other marking nodes are not. To take into account this possible additional gap between the sources we introduced “m” even if m can be negligible in most of the cases. Also note that the formula (L - 2m - 2d > 0) is exactly the same of RFC8321 if m=0, indeed in case of a point-to-point flow we only have one marking node and m=0. Maybe more explanation in the draft could help here.

[GF]: Ok I will revise.

---

From: Martin Duke <[email-masked]>
Date: Fri, 3 Dec 2021 09:38:23 -0800

On Fri, Dec 3, 2021 at 6:59 AM Giuseppe Fioccola <
[email-masked]> wrote:

> [GF]: I fully agree with your proposals. The marking node MUST mark all
> the fragments except in the case of fragmentation within the network
> domain; in this case it is suggested to mark only the first packet. The
> condition about the counters also makes sense so it could be possible to
> keep track of both marked and unmarked fragments. I can include these
> considerations in the next version.
>

Alright, please think it through and write up some text if you think it
checks out.

> I wasn't clear enough about (b). Let's say node A has period L1, and node
> B is trying to set its L but is going to start at an arbitrary time. What
> is m for node B? It is a uniform distribution over [0, L1]. so L2 has to be
> >> L1. But the same is true of A! The only way to disambiguate the blocks
> is for each L to be much bigger than all the other Ls, which is of course
> impossible.
>
>
>
> The only way I can see for this to work is for the marking nodes to agree
> a clock time to switch blocks, with fixed L, and then you're only
> constrained by the clock skew and delay as in 8321.
>
>
>
> [GF]: Yes, the only way is that the marking nodes switch blocks with fixed
> L based on a clock time. In theory the formula in RFC8321 applies also in
> case of multipoint network. But we wanted to generalize that formula and
> let me explain the rationale. Since we have more marking nodes in a
> multipoint environment, there can be an additional mismatch to consider
> since different nodes are marking the traffic and the batches get mixed in
> a multipoint flow. For example, a marking node may apply the marking with a
> delay because it is overloaded while the other marking nodes are not. To
> take into account this possible additional gap between the sources we
> introduced “m” even if m can be negligible in most of the cases. Also note
> that the formula (L - 2m - 2d > 0) is exactly the same of RFC8321 if m=0,
> indeed in case of a point-to-point flow we only have one marking node and
> m=0. Maybe more explanation in the draft could help here.
>
>

Agreed, let's more precisely define what m is, and clarify that marking
nodes are switching colors based on synchronized clock time.

---

From: Giuseppe Fioccola <[email-masked]>
Date: Mon, 6 Dec 2021 10:06:28 +0000

Hi Martin,
Sure, I will update this draft as well in the coming weeks.

Regards,

Giuseppe

Sent: Friday, December 3, 2021 6:38 PM

On Fri, Dec 3, 2021 at 6:59 AM Giuseppe Fioccola <[email-masked]<mailto:[email-masked]>> wrote:
Hi Martin,
My answers inline as [GF].

Alright, please think it through and write up some text if you think it checks out.

Agreed, let's more precisely define what m is, and clarify that marking nodes are switching colors based on synchronized clock time.