Subject: Re: [nvo3] Comments on NVO3 data plane requirements for OAM

From: "Larry Kreeger (kreeger)" <[email-masked]>
Date: Tue, 11 Nov 2014 20:33:26 +0000

Hi Weiguo,

What do you envision this marking looking like?  e.g. is it just a single flag bit, or large field with a counter or sequence number, or some kind of flow ID?  If not a single flag, how large do you see the field being?

If it is more than a flag (and I assume it would be), and is not mandatory for all implementations, then it seems to fall into the category of optional extensions.

Thanks, Larry

From: Haoweiguo <[email-masked]<mailto:[email-masked]>>
Date: Tuesday, November 11, 2014 10:18 AM
To: Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hi Greg,

I fully agree with you.

The real time OAM is passive performance measurement methods. I would like NVO3 data encapsulation has a field for marking and not affect forwarding of packets, the marking field is only used for performance measurement. The NVO3 packet with this marking flag don't need to be sent to control plane, it is different from OAM(ping/Trace) packet processing.

Thanks

weiguo

________________________________
发件人: Greg Mirsky [[email-masked]<mailto:[email-masked]>]
发送时间: 2014年11月12日 4:07
收件人: Haoweiguo
抄送: [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] Comments on NVO3 data plane requirements for OAM

Hi Weiguo,
marking groups of packets that belong to the particular flow to facilitate measurement of some performance metric, whether loss or delay/delay variation, may be viewed as one of passive performance measurement methods. But such marking should not alter, at least not significantly alter, treatment of data flow in the network. Because of that, I believe, OAM flag should not be used for marking as that will force punting marked packets from fast forwarding path to the control plane. But it might be good to have a field in NVO3 header that may be used for marking and not affect forwarding of packets if altered.

Regards,
Greg

On Tue, Nov 11, 2014 at 12:34 AM, Haoweiguo <[email-masked]<mailto:[email-masked]>> wrote:
Hi All,
I maybe not clearly said in today’s NVO3 meeting, pls allow me to reiterate the OAM data plane requirements on the mail list.

Currently NVO3 data plane encapsulation only includes one OAM flag, it is used for Ping/Trace similar applications. This kind of OAM application is initiated by operators for network connectivity verification, normally when network failure occurs. There is another OAM requirements of real time OAM or synthesizing OAM. It can be used for packet loss detection in real time. When ingress NVE receives traffic from local TS, it gets packet statistics, and mark(coloring) the OAM flag relying on local policy when it performs NVO3 encapsulation. When egress NVEs receives the traffic, it decapsulates NVO3 encapsulation, and gets packet statistics with the real time OAM flag marking. By comparing the packet number of ingress NVE and the sum of all egress NVEs, packet loss can be deduced. This method can be applicable for both unicast and multicast traffic. Local policy on ingress NVE is configured by operators or automatically acquired from centralized orchestration.

Thanks

weiguo

_______________________________________________
nvo3 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nvo3

---

From: Tom Herbert <[email-masked]>
Date: Tue, 11 Nov 2014 13:42:31 -0800

On Tue, Nov 11, 2014 at 12:33 PM, Larry Kreeger (kreeger)
<[email-masked]> wrote:
> Hi Weiguo,
>
> What do you envision this marking looking like?  e.g. is it just a single
> flag bit, or large field with a counter or sequence number, or some kind of
> flow ID?  If not a single flag, how large do you see the field being?
>
> If it is more than a flag (and I assume it would be), and is not mandatory
> for all implementations, then it seems to fall into the category of optional
> extensions.
>
I assume this is a request for in-band measurement as opposed to some
out of band summary mechanism which seems to be more typical of OAM.
If we are adding loss counters/delay metrics to every data packet,
this is starting to look like the sort of data we meed for congestion
control and in fact might be a subset of that.

Tom

> Thanks, Larry
>
> From: Haoweiguo <[email-masked]>
> To: Greg Mirsky <[email-masked]>
> Cc: "[email-masked]" <[email-masked]>
> The real time OAM is passive performance measurement methods. I would like
> NVO3 data encapsulation has a field for marking and not affect forwarding of
> packets, the marking field is only used for performance measurement. The
> NVO3 packet with this marking flag don't need to be sent to control plane,
> it is different from OAM(ping/Trace) packet processing.
>
> Thanks
>
> weiguo
>
> 发件人: Greg Mirsky [[email-masked]]
> 发送时间: 2014年11月12日 4:07
> 收件人: Haoweiguo
> 抄送: [email-masked]
> marking groups of packets that belong to the particular flow to facilitate
> measurement of some performance metric, whether loss or delay/delay
> variation, may be viewed as one of passive performance measurement methods.
> But such marking should not alter, at least not significantly alter,
> treatment of data flow in the network. Because of that, I believe, OAM flag
> should not be used for marking as that will force punting marked packets
> from fast forwarding path to the control plane. But it might be good to have
> a field in NVO3 header that may be used for marking and not affect
> forwarding of packets if altered.
>
> Regards,
> Greg
>
> On Tue, Nov 11, 2014 at 12:34 AM, Haoweiguo <[email-masked]> wrote:
>>
>> Hi All,
>>
>> I maybe not clearly said in today’s NVO3 meeting, pls allow me to
>> reiterate the OAM data plane requirements on the mail list.
>>
>> Currently NVO3 data plane encapsulation only includes one OAM flag, it is
>> used for Ping/Trace similar applications. This kind of OAM application is
>> initiated by operators for network connectivity verification, normally when
>> network failure occurs. There is another OAM requirements of real time OAM
>> or synthesizing OAM. It can be used for packet loss detection in real time.
>> When ingress NVE receives traffic from local TS, it gets packet statistics,
>> and mark(coloring) the OAM flag relying on local policy when it performs
>> NVO3 encapsulation. When egress NVEs receives the traffic, it decapsulates
>> NVO3 encapsulation, and gets packet statistics with the real time OAM flag
>> marking. By comparing the packet number of ingress NVE and the sum of all
>> egress NVEs, packet loss can be deduced. This method can be applicable for
>> both unicast and multicast traffic. Local policy on ingress NVE is
>> configured by operators or automatically acquired from centralized
>> orchestration.
>>
>> Thanks
>>
>> weiguo
>>
>>
>> [email-masked]
> [email-masked]

---

From: Marc Binderberger <[email-masked]>
Date: Tue, 11 Nov 2014 14:26:45 -0800

Hello Larry, Weiguo and Greg,

there seem multiple ideas or requirements for OAM. I mentioned in another 
thread (started by Erik) that we may want "punt, don't forward" as well as 
"punt & forward". The forwarding would stay in the fast data plane.

And instead of flags I propose to use a protocol/next-header field (which 
seems to exist in several of the proposed data plane encapsulations) and let 
the hardware react on this field, so introducing different OAM behaviour is 
"simple" on the encoding side.

The OAM would then carry information like counters in a TLV sitting between 
the initial header and the payload.

> for all implementations, then it seems to fall into the category of 
> optional extensions.

Agree, unless the control plane ensures that features like OAM are only used 
when all nodes support it. This would shift some complexity from the (fast) 
data plane into the control plane.

Regards, Marc

On Tue, 11 Nov 2014 20:33:26 +0000, Larry Kreeger (kreeger) wrote:
> for all implementations, then it seems to fall into the category of 
> optional extensions.
> 
> Thanks, Larry
> 
> NVO3 data encapsulation has a field for marking and not affect forwarding 
> of packets, the marking field is only used for performance measurement. The 
> from fast forwarding path to the control plane. But it might be good to 
> have a field in NVO3 header that may be used for marking and not affect 
>> initiated by operators for network connectivity verification, normally 
>> when network failure occurs. There is another OAM requirements of real 
>> time OAM or synthesizing OAM. It can be used for packet loss detection in 
>> real time. When ingress NVE receives traffic from local TS, it gets packet 
>> statistics, and mark(coloring) the OAM flag relying on local policy when 
>> it performs NVO3 encapsulation. When egress NVEs receives the traffic, it 
>> decapsulates NVO3 encapsulation, and gets packet statistics with the real 
>> time OAM flag marking. By comparing the packet number of ingress NVE and 
>> the sum of all egress NVEs, packet loss can be deduced. This method can be 
>> applicable for both unicast and multicast traffic. Local policy on ingress 
>> NVE is configured by operators or automatically acquired from centralized 
>> orchestration.
>> Thanks
>> weiguo
>>

---

From: Haoweiguo <[email-masked]>
Date: Tue, 11 Nov 2014 23:51:32 +0000

Hi Larry,

For marking purpose, i think one bit maybe OK, fixed fields in NVO3 header is precious. I would like it is set in fixed field, rather than in option field. Because chipset normally can't process optional field, it is hard to realize in-band performance measurement if using optional field for marking. For other real time congestion control function, maybe more bits are needed.

Thanks

weiguo

________________________________
发件人: Larry Kreeger (kreeger) [[email-masked]]
发送时间: 2014年11月12日 4:33
收件人: Haoweiguo; Greg Mirsky
抄送: [email-masked]
主题: Re: [nvo3] Comments on NVO3 data plane requirements for OAM

Hi Weiguo,

Thanks, Larry

Hi Greg,

I fully agree with you.

Thanks

weiguo

Hi Weiguo,

Regards,
Greg

Thanks

weiguo

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 11 Nov 2014 16:03:13 -0800

Hi Tom,
I see very little use for out-of-band performance measurement as it result
hardly characteristic of monitored service. Perhaps we compare
out-or-service and in-service measurement. Marking is to facilitate Passive
performance measurement which is obviously in-service and in-band OAM. As
example of passive measurement it has limitations as well as advantages.
Marking method does not require tagging data packets with anything but mark
in the way that should not alter network treatment of unmarked packet. All
timestamps and counters are to be collected at observation points. Marking
helps to correlate collected information and perform measurements.

Regards,
Greg

On Tue, Nov 11, 2014 at 1:42 PM, Tom Herbert <[email-masked]> wrote:

> optional
> have
> when

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 11 Nov 2014 16:04:30 -0800

Dear All,
agree with Weiguo, single bit flag in fixed position would be sufficient
and HW-friendly.

Regards,
Greg

On Tue, Nov 11, 2014 at 3:51 PM, Haoweiguo <[email-masked]> wrote:

>  Hi Larry,
>
> For marking purpose, i think one bit maybe OK, fixed fields in NVO3 header
> is precious. I would like it is set in fixed field, rather than in option
> field. Because chipset normally can't process optional field, it is hard to
> realize in-band performance measurement if using optional field for
> marking. For other real time congestion control function, maybe more bits
> are needed.
>
> Thanks
>
> weiguo
>  ------------------------------
> *发件人:* Larry Kreeger (kreeger) [[email-masked]]
> *发送时间:* 2014年11月12日 4:33
> *收件人:* Haoweiguo; Greg Mirsky
>
> *抄送:* [email-masked]
> *主题:* Re: [nvo3] Comments on NVO3 data plane requirements for OAM
>
>   Hi Weiguo,
>
>  What do you envision this marking looking like?  e.g. is it just a
> single flag bit, or large field with a counter or sequence number, or some
> kind of flow ID?  If not a single flag, how large do you see the field
> being?
>
>  If it is more than a flag (and I assume it would be), and is not
> mandatory for all implementations, then it seems to fall into the category
> of optional extensions.
>
>  Thanks, Larry
>
>  ------------------------------
> *发件人:* Greg Mirsky [[email-masked]]
> *发送时间:* 2014年11月12日 4:07
> *收件人:* Haoweiguo
> *抄送:* [email-masked]
> *主题:* Re: [nvo3] Comments on NVO3 data plane requirements for OAM
>
>    Hi Weiguo,
>  marking groups of packets that belong to the particular flow to
> facilitate measurement of some performance metric, whether loss or
> delay/delay variation, may be viewed as one of passive performance
> measurement methods. But such marking should not alter, at least not
> significantly alter, treatment of data flow in the network. Because of
> that, I believe, OAM flag should not be used for marking as that will force
> punting marked packets from fast forwarding path to the control plane. But
> it might be good to have a field in NVO3 header that may be used for
> marking and not affect forwarding of packets if altered.
>
>  Regards,
>  Greg
>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 11 Nov 2014 16:22:32 -0800

On Tue, Nov 11, 2014 at 4:03 PM, Greg Mirsky <[email-masked]> wrote:
> performance measurement which is obviously in-service and in-band OAM.

By out of band I mean not piggy backed on a data packet, but still can
follow same path (for example ping to test path).

 As
> helps to correlate collected information and perform measurements.
How would get a time stamp from just a single mark on a packet?

>> > single
>> > mandatory
>> > facilitate

---

From: Marc Binderberger <[email-masked]>
Date: Tue, 11 Nov 2014 18:24:05 -0800

Hello Greg and Weiguo,

> agree with Weiguo, single bit flag in fixed position would be sufficient 

a single bit just turns on and off - but it seems we have two different ideas 
of OAM under discussion meanwhile. And both ideas claim they need an "OAM" 
flag.

Makes already 2 bits :-)

> The NVO3 packet with this marking flag don't need to be sent to control 
plane,
> it is different from OAM(ping/Trace) packet processing.

Really?  How is this working?  To do any processing of this real-time OAM you 
still need to punt a copy of the NVO3 packet or at least the OAM-related 
information to the generic CPU, i.e. get it out of the fast/hw forwarding 
plane.

And then you need some information in the NVO3 packet, I assume?  Timestamps, 
Counters etc.?  I don't think this will fit into any of the headers discussed 
so far unless you use a TLV approach.

>> field. Because chipset normally can't process optional field, it is hard 
>> to realize in-band performance measurement if using optional field for 

If your optional field is defined to be the "first option TLV" then this is 
no difference from a larger fixed header. Still not sure what the chipset is 
supposed to process.

If the NVO3 group thinks this kind of OAM is sort of a must then of course it 
makes sense to define the (fixed) base header with this OAM data. My problem 
here is ...

>> marking. For other real time congestion control function, maybe more bits 

... that you already indicate there may be more/different OAM data in the 
future. Using a fixed header likely means a new, larger fixed header to 
incorporate the additional OAM, which makes older implementations 
incompatible.

What the (fixed?) base header should support is the principle mechanism - we 
seem to discuss a "punt, don't forward" and a "punt & forward" OAM, if I 
understand it right (?).

At least the more "fancy" OAM seems a fit for optional TLV (with some 
position restriction).

This initial OAM we are talking about here, is this just packet loss? So you 
would need to carry some sequence number?

Regards, Marc

On Tue, 11 Nov 2014 16:04:30 -0800, Greg Mirsky wrote:
>> field. Because chipset normally can't process optional field, it is hard 
>> to realize in-band performance measurement if using optional field for 
>> flag bit, or large field with a counter or sequence number, or some kind 
>> of flow ID?  If not a single flag, how large do you see the field being?
>> 
>> of packets, the marking field is only used for performance measurement. 
>> The NVO3 packet with this marking flag don't need to be sent to control 
>> plane, it is different from OAM(ping/Trace) packet processing.
>> Thanks
>> weiguo
>> 
>> variation, may be viewed as one of passive performance measurement 
>> methods. But such marking should not alter, at least not significantly 
>> alter, treatment of data flow in the network. Because of that, I believe, 
>> OAM flag should not be used for marking as that will force punting marked 
>> packets from fast forwarding path to the control plane. But it might be 
>> good to have a field in NVO3 header that may be used for marking and not 
>> affect forwarding of packets if altered.
>> 
>> Regards,
>> Greg
>> 
>>> real time. When ingress NVE receives traffic from local TS, it gets 
>>> packet statistics, and mark(coloring) the OAM flag relying on local 
>>> policy when it performs NVO3 encapsulation. When egress NVEs receives the 
>>> traffic, it decapsulates NVO3 encapsulation, and gets packet statistics 
>>> with the real time OAM flag marking. By comparing the packet number of 
>>> ingress NVE and the sum of all egress NVEs, packet loss can be deduced. 
>>> This method can be applicable for both unicast and multicast traffic. 
>>> Local policy on ingress NVE is configured by operators or automatically 
>>> acquired from centralized orchestration.
>>> Thanks
>>> weiguo
>>>

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 11 Nov 2014 18:37:04 -0800

Hi Marc,
thank you for your thorough review and thoughtful comments.
How passive performance measurement may work discussed in IP Flow
Performance Measurement Framework
<https://datatracker.ietf.org/doc/draft-chen-ippm-coloring-based-ipfpm-framework/>and
IP Flow Performance Measurement Report
<http://tools.ietf.org/html/draft-chen-ippm-ipfpm-report-00>.

I still believe that "original OAM flag" is to be used for active OAM, e.g.
continuity check, proactive and on-demand, performance measurement. In some
way, the GAL in MPLS is that "original OAM flag". But active OAM, IMO,
should be complemented by use of passive measurement methods. Often these
viewed as reading counters, IPFIX. But marking is method that expands and
improves passive performance measurements through ability to correlate
measurements taken at individual nodes along a path of the flow.

Regards,
Greg
<https://datatracker.ietf.org/doc/draft-chen-ippm-coloring-based-ipfpm-framework/>

---

From: Haoweiguo <[email-masked]>
Date: Wed, 12 Nov 2014 03:04:04 +0000

Hi Tom,
Pls see inline with [weiguo].
Thanks
weiguo

________________________________________
发件人: Tom Herbert [[email-masked]]
发送时间: 2014年11月12日 8:22
收件人: Greg Mirsky
抄送: Larry Kreeger (kreeger); Haoweiguo; [email-masked]
主题: Re: [nvo3] Comments on NVO3 data plane requirements for OAM

[weiguo]: Yes,exactly. Current NVO3 OAM consideration only relates to out of band OAM, not piggy backed on data packet. So i would like to add a additional passive measurement method, i.e., in-band OAM.

 As

[weiguo]: For packet loss statistics purpose, no time stamp is needed. A bit in NVO3 header is enough for packet loss. The flag is used to separaring packets between different statistics period. For example, if statistic period is 10 seconds, first period packets the flag is set to 1 on ingress NVE, second period the flag is set to 0, third period the flag is set to 1 again, then repeat again and again, until the statistics behavior terminated.
Ingress NVE and egress NVE need to send their statistics for each period to a centralized point, the centralized point compares the statictics of packet number, the difference number is packet loss.

>> > of
>> > methods.

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 11 Nov 2014 19:11:36 -0800

Dear Tom and Weiguo,
I think what been referred as "out-of-band" usually been characterized as
Active OAM, i.e. injected test packets that used to check continuity,
verify connectivity or measure particular performance metric. "Out-of-band"
usually describes OAM, control or signaling flow that is not identical,
co-routed with the data flow. Obviously value of out-of-band OAM is
somewhat questionable and that is why ensuring active OAM being in-band,
IMO, is one of utmost important requirements.

Regards,
Greg

On Tue, Nov 11, 2014 at 7:04 PM, Haoweiguo <[email-masked]> wrote:

> [weiguo]: Yes,exactly. Current NVO3 OAM consideration only relates to out
> of band OAM, not piggy backed on data packet. So i would like to add a
> additional passive measurement method, i.e., in-band OAM.
>
>  As
> [weiguo]: For packet loss statistics purpose, no time stamp is needed. A
> bit in NVO3 header is enough for packet loss. The flag is used to
> separaring packets between different statistics period. For example, if
> statistic period is 10 seconds, first period packets the flag is set to 1
> on ingress NVE, second period the flag is set to 0, third period the flag
> is set to 1 again, then repeat again and again, until the statistics
> behavior terminated.
> Ingress NVE and egress NVE need to send their statistics for each period
> to a centralized point, the centralized point compares the statictics of
> packet number, the difference number is packet loss.
>
> > Regards,
> > Greg
> >
> >> >> Currently NVO3 data plane encapsulation only includes one OAM flag,
> it
> >> >> is
> >> >> used for Ping/Trace similar applications. This kind of OAM
> application
> >> >> is
> >> >> network failure occurs. There is another OAM requirements of real
> time
> >> >> OAM
> >> >> egress NVEs, packet loss can be deduced. This method can be
> applicable
> >> >> for

---

From: Tom Herbert <[email-masked]>
Date: Tue, 11 Nov 2014 19:24:08 -0800

> Pls see inline with [weiguo].

Can you just keep a running count of packets received on the tunnel
and return that periodically (I believe this is something like what
circuit breaker does)?

Tom

---

From: Haoweiguo <[email-masked]>
Date: Wed, 12 Nov 2014 08:40:07 +0000

Hi Greg and Tom,

Yes, we should allign with the term, out-of-band OAM (ping/trace)here normally is called Active OAM. Our new discussed OAM is passive OAM. Maybe some experts have better idea to name this new OAM using marking mechanism.

Thanks

weiguo

发件人: Greg Mirsky [[email-masked]]
发送时间: 2014年11月12日 11:11
收件人: Haoweiguo
抄送: Tom Herbert; Larry Kreeger (kreeger); [email-masked]
主题: Re: 答复: [nvo3] Comments on NVO3 data plane requirements for OAM

Dear Tom and Weiguo,
I think what been referred as "out-of-band" usually been characterized as Active OAM, i.e. injected test packets that used to check continuity, verify connectivity or measure particular performance metric. "Out-of-band" usually describes OAM, control or signaling flow that is not identical, co-routed with the data flow. Obviously value of out-of-band OAM is somewhat questionable and that is why ensuring active OAM being in-band, IMO, is one of utmost important requirements.

Regards,
Greg

On Tue, Nov 11, 2014 at 7:04 PM, Haoweiguo <[email-masked]<mailto:[email-masked]>> wrote:
Hi Tom,
Pls see inline with [weiguo].
Thanks
weiguo

________________________________________
发件人: Tom Herbert [[email-masked]<mailto:[email-masked]>]
发送时间: 2014年11月12日 8:22
收件人: Greg Mirsky
抄送: Larry Kreeger (kreeger); Haoweiguo; [email-masked]<mailto:[email-masked]>
主题: Re: [nvo3] Comments on NVO3 data plane requirements for OAM

 As

> Regards,
> Greg
>
> On Tue, Nov 11, 2014 at 1:42 PM, Tom Herbert <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> <[email-masked]<mailto:[email-masked]>> wrote:
>> > Hi Weiguo,
>> >

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 12 Nov 2014 00:43:56 -0800

Hello Greg and Weiguo,

ah, I see. Thanks for the links and the explanation!

While this is "OAM" maybe we can give it a more explicit name to avoid 
confusion? "Performance Measurement"? "Passive OAM"? Or maybe "Coloring"?

Anyway, so this is something different than what I thought. For a single bit 
I guess a full TLV is a bit overkill ;-)  So the data plane header needs a 
"flag field", okay. 

NVGRE, Geneve, GUE, VxLAN(-gpe) all have a flag/reserved field, shouldn't be 
a problem to add this flag, IMHO.

Regards, Marc

On Tue, 11 Nov 2014 18:37:04 -0800, Greg Mirsky wrote:
> Performance Measurement Framework and IP Flow Performance Measurement 
> Report.
>

---

From: Vero Zheng <[email-masked]>
Date: Wed, 12 Nov 2014 08:51:12 +0000

Hi Marc et.al,

I’m one of the co-authors/editors of the above-mentioned two drafts. Thanks for your interest in the marking mechanism.
By changing one or more bits of packets (header), data packets are marked into different blocks of markers without altering normal processing in the network.
No additional delimiting packet is needed and the performance can be measured in-service without the insertion of additional traffic.
The packet count/timestamp will be collected at the measurement point which correlated to certain blocks of marker and the loss/delay will be calculated.

BR,
Vero

From: nvo3 [mailto:[email-masked]] On Behalf Of Greg Mirsky
Sent: Wednesday, November 12, 2014 10:37 AM
To: Marc Binderberger
Cc: Haoweiguo; [email-masked]; Larry Kreeger
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hi Marc,
thank you for your thorough review and thoughtful comments.
How passive performance measurement may work discussed in IP Flow Performance Measurement Framework <https://datatracker.ietf.org/doc/draft-chen-ippm-coloring-based-ipfpm-framework/> and IP Flow Performance Measurement Report<http://tools.ietf.org/html/draft-chen-ippm-ipfpm-report-00>.
I still believe that "original OAM flag" is to be used for active OAM, e.g. continuity check, proactive and on-demand, performance measurement. In some way, the GAL in MPLS is that "original OAM flag". But active OAM, IMO, should be complemented by use of passive measurement methods. Often these viewed as reading counters, IPFIX. But marking is method that expands and improves passive performance measurements through ability to correlate measurements taken at individual nodes along a path of the flow.
Regards,
Greg

On Tue, Nov 11, 2014 at 6:24 PM, Marc Binderberger <[email-masked]<mailto:[email-masked]>> wrote:
Hello Greg and Weiguo,

Makes already 2 bits :-)

plane,

Regards, Marc

On Tue, 11 Nov 2014 16:04:30 -0800, Greg Mirsky wrote:
> On Tue, Nov 11, 2014 at 3:51 PM, Haoweiguo <[email-masked]<mailto:[email-masked]>> wrote:
>> Hi Larry,
>> 发件人: Larry Kreeger (kreeger) [[email-masked]<mailto:[email-masked]>]
>> 发送时间: 2014年11月12日 4:33

---

From: Haoweiguo <[email-masked]>
Date: Wed, 12 Nov 2014 08:53:46 +0000

Hi Marc,
Yes, a more explicit name is welcome for this "OAM".
About the reserved bits, i think two bits had better to be reserved in fixed position.
1 bit is used for packet loss detection. This bit usage has already been discussed in the thread.
1 bit is used for packet delay detection. This bit maybe has lower priority than the previous bit.
Thanks
weiguo

________________________________________
发件人: Marc Binderberger [[email-masked]]
发送时间: 2014年11月12日 16:43
收件人: Greg Mirsky; Haoweiguo
抄送: [email-masked]; Larry Kreeger
主题: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hello Greg and Weiguo,

Regards, Marc

---

From: Mach Chen <[email-masked]>
Date: Wed, 12 Nov 2014 08:55:38 +0000

Hi Greg and all,

Single bit is not sufficient if someone wants to perform loss and delay measurement  simultaneously, then two bits needed.

Best regards,
Mach

From: nvo3 [mailto:[email-masked]] On Behalf Of Greg Mirsky
Sent: Wednesday, November 12, 2014 8:05 AM
To: Haoweiguo
Cc: [email-masked]; Larry Kreeger (kreeger)
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Dear All,
agree with Weiguo, single bit flag in fixed position would be sufficient and HW-friendly.
Regards,
Greg

Hi Larry,

Thanks

weiguo

Hi Weiguo,

Thanks, Larry

Hi Greg,

I fully agree with you.

Thanks

weiguo

Thanks

weiguo

---

From: Tom Herbert <[email-masked]>
Date: Wed, 12 Nov 2014 01:05:30 -0800

On Wed, Nov 12, 2014 at 12:55 AM, Mach Chen <[email-masked]> wrote:
> Hi Greg and all,
>
>
>
> Single bit is not sufficient if someone wants to perform loss and delay
> measurement  simultaneously, then two bits needed.
>
Is that necessary? Can they share the same time quantum (as well as
other metrics maybe to be added later)? In all the protocols
mentioned, the reserved bits are a somewhat precious resource.

Tom

> realize in-band performance measurement if using optional field for marking.
> For other real time congestion control function, maybe more bits are needed.
>
> Thanks
>
> weiguo
>
> I maybe not clearly said in today’s NVO3 meeting, pls allow me to reiterate
> the OAM data plane requirements on the mail list.
>

---

From: Mach Chen <[email-masked]>
Date: Wed, 12 Nov 2014 09:34:52 +0000

Hi Tom,

> -----Original Message-----
> From: Tom Herbert [mailto:[email-masked]]
> Sent: Wednesday, November 12, 2014 5:06 PM
> To: Mach Chen
> Cc: Greg Mirsky; Haoweiguo; [email-masked]; Larry Kreeger (kreeger)
> > Single bit is not sufficient if someone wants to perform loss and
> > delay measurement  simultaneously, then two bits needed.
> >
> Is that necessary? Can they share the same time quantum (as well as other
> metrics maybe to be added later)? In all the protocols mentioned, the reserved
> bits are a somewhat precious resource.

Yes, it's necessary if there is ECMP. 

Given one bit is used for both loss and delay measurement, for loss measurement, it periodically set and clear the marking bit, a flow is divided into consecutive blocks, and then the counting and calculating are based on each block. This is fine for loss measurement. 

For delay measurement, it has to make sure the timestamps (collected at sender and receiver) are for the same packet. Presumably, the time when changing the marking bit is right time to get the timestamps. Since there is ECMP, the first packet of a block at the sender may probably different from the first packet at the receiver, thus it will get the mismatched timestamps to calculate the delay. 

Best regards,
Mach
> > For marking purpose, i think one bit maybe OK, fixed fields in NVO3
> > header is precious. I would like it is set in fixed field, rather than
> > in option field. Because chipset normally can't process optional
> > field, it is hard to realize in-band performance measurement if using optional
> field for marking.
> > single flag bit, or large field with a counter or sequence number, or
> > some kind of flow ID?  If not a single flag, how large do you see the field
> being?
> >
> >
> >
> > mandatory for all implementations, then it seems to fall into the
> > category of optional extensions.
> >
> >
> >
> > Thanks, Larry
> >
> >
> >
> > NVO3 data encapsulation has a field for marking and not affect
> > forwarding of packets, the marking field is only used for performance
> > measurement. The
> > NVO3 packet with this marking flag don't need to be sent to control
> measurement methods.
> > treatment of data flow in the network. Because of that, I believe, OAM
> > flag should not be used for marking as that will force punting marked
> > packets from fast forwarding path to the control plane. But it might
> > be good to have a field in NVO3 header that may be used for marking
> > and not affect forwarding of packets if altered.
> >
> > Regards,
> >
> > Greg
> >
> >
> >
> > Currently NVO3 data plane encapsulation only includes one OAM flag, it
> > is used for Ping/Trace similar applications. This kind of OAM
> > application is initiated by operators for network connectivity
> > verification, normally when network failure occurs. There is another
> > OAM requirements of real time OAM or synthesizing OAM. It can be used for
> packet loss detection in real time.
> > When ingress NVE receives traffic from local TS, it gets packet
> > statistics, and mark(coloring) the OAM flag relying on local policy
> > when it performs
> > NVO3 encapsulation, and gets packet statistics with the real time OAM
> > flag marking. By comparing the packet number of ingress NVE and the
> > sum of all egress NVEs, packet loss can be deduced. This method can be
> > applicable for both unicast and multicast traffic. Local policy on
> > ingress NVE is configured by operators or automatically acquired from
> > centralized orchestration.
> >
> > Thanks
> >
> > weiguo
> >
> >

---

From: Greg Mirsky <[email-masked]>
Date: Wed, 12 Nov 2014 01:47:12 -0800

Hi Marc,
I agree, marking is one of passive performance measurement methods.
Originally it was referred to as 'coloring' but later, in part because
among Ethernet folks 'coloring' describes flow policing and drop
eligibility marking process, we've changed to 'marking'.
But Mach corrected me and pointed to valid scenario when not one but two
bits required to concurrently measure loss and latency/jitter. So, we'll
settle with two bits for passive performance measurement. Can we refer to
it as Marker field? And it should have no role in routing/forwarding/qos
decision making.

Regards,
Greg

On Wed, Nov 12, 2014 at 12:43 AM, Marc Binderberger <[email-masked]> wrote:

> >> Really?  How is this working?  To do any processing of this real-time
> OAM
> >> you
> >> no difference from a larger fixed header. Still not sure what the
> chipset
> >> is
> >> supposed to process.
> >>
> >> If the NVO3 group thinks this kind of OAM is sort of a must then of
> course
> >> it
> >> What the (fixed?) base header should support is the principle mechanism
> -
> >> we
> >> header
> >> forwarding
> be
> >>>>> time OAM or synthesizing OAM. It can be used for packet loss
> detection
> >> in

---

From: Marc Binderberger <[email-masked]>
Date: Wed, 12 Nov 2014 02:11:13 -0800

Hello Mach,

so for delay measurement you use the color flag to mark a single packet, 
which helps the receiver to pick the right packet?  And repeat this every 
time period T ?

    ...000100000010000001000...

One question I still have is: why is the measurement done in the NVE header? 
The outer header is IP/IPv6, so couldn't we use the coloring for the IP/IPv6 
header, assuming this is defined?

Thanks & Regards,
Marc

On Wed, 12 Nov 2014 09:34:52 +0000, Mach Chen wrote:
> Given one bit is used for both loss and delay measurement, for loss 
> measurement, it periodically set and clear the marking bit, a flow is 
> divided into consecutive blocks, and then the counting and calculating are 
> based on each block. This is fine for loss measurement. 
> 
> For delay measurement, it has to make sure the timestamps (collected at 
> sender and receiver) are for the same packet. Presumably, the time when 
> changing the marking bit is right time to get the timestamps. Since there 
> is ECMP, the first packet of a block at the sender may probably different 
> from the first packet at the receiver, thus it will get the mismatched 
> timestamps to calculate the delay. 
> 
> Best regards,
> Mach
>> 
>> Tom
>> 
>>> 
>>> 
>>> Best regards,
>>> 
>>> Mach
>>> 
>>> 
>>>

---

From: Tom Herbert <[email-masked]>
Date: Wed, 12 Nov 2014 11:10:52 -0800

> which helps the receiver to pick the right packet?  And repeat this every
Is there there a draft or description of how this algorithm would
work? Seems like there would need to be quite a bot of synchronization
needed between end points (synchronized clocks, provisions to
correlate measurements correctly with lost packets, replicated
packets, etc.). Also, what is envisioned for range for the period?

Thanks,
Tom

---

From: Mach Chen <[email-masked]>
Date: Thu, 13 Nov 2014 01:06:13 +0000

Hi Marc,

> From: Marc Binderberger [mailto:[email-masked]]
> Sent: Wednesday, November 12, 2014 6:11 PM
> To: Mach Chen
> Cc: Tom Herbert; Greg Mirsky; Haoweiguo; [email-masked]; Larry Kreeger
> so for delay measurement you use the color flag to mark a single packet, which
> helps the receiver to pick the right packet?  And repeat this every time period
> T ?

Yes

Regarding NVE header of IP/IPv6 header, I am not sure which is better. One challenge for using IPv4/IPv6 header is that there is not too much bits that can be used for marking, especially for IPv4.

Best regards,
Mach

---

From: Mach Chen <[email-masked]>
Date: Thu, 13 Nov 2014 01:13:25 +0000

Hi Tom,

> Sent: Thursday, November 13, 2014 3:11 AM
> To: Marc Binderberger
> Cc: Mach Chen; Greg Mirsky; Haoweiguo; [email-masked]; Larry Kreeger
> > so for delay measurement you use the color flag to mark a single
> > packet, which helps the receiver to pick the right packet?  And repeat
> > this every time period T ?
> >
> Is there there a draft or description of how this algorithm would work? Seems
> like there would need to be quite a bot of synchronization needed between end
> points (synchronized clocks, provisions to correlate measurements correctly with
> lost packets, replicated packets, etc.). Also, what is envisioned for range for the
> period?

Here is a reference https://datatracker.ietf.org/doc/draft-chen-ippm-coloring-based-ipfpm-framework/.

Yes, it does need some synchronization. As for the range, it depends on two factors, one is the implementation limitation, the other the requirement of the operators. In the above reference, the suggested periods are 1s, 10s, 1min, 10min and 1h.

Best regards,
Mach
> > The outer header is IP/IPv6, so couldn't we use the coloring for the
> > IP/IPv6 header, assuming this is defined?
> >
> >
> > Thanks & Regards,
> > Marc
> >
> >
> >
> >> divided into consecutive blocks, and then the counting and
> >> calculating are based on each block. This is fine for loss measurement.
> >>
> >> For delay measurement, it has to make sure the timestamps (collected
> >> at sender and receiver) are for the same packet. Presumably, the time
> >> when changing the marking bit is right time to get the timestamps.
> >> Since there is ECMP, the first packet of a block at the sender may
> >> probably different from the first packet at the receiver, thus it
> >> will get the mismatched timestamps to calculate the delay.
> >>
> >> Best regards,
> >> Mach
> >>>
> >>> Tom
> >>>
> >>>>
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Mach
> >>>>
> >>>>
> >>>>
> >>>> header is precious. I would like it is set in fixed field, rather
> >>>> than in option field. Because chipset normally can't process
> >>>> optional field, it is hard to realize in-band performance
> >>>> measurement if using optional
> >>> field for marking.
> >>>> single flag bit, or large field with a counter or sequence number,
> >>>> or some kind of flow ID?  If not a single flag, how large do you
> >>>> see the field
> >>> being?
> >>>>
> >>>>
> >>>>
> >>>> forwarding of packets, the marking field is only used for
> >>>> performance measurement. The
> >>>> But such marking should not alter, at least not significantly
> >>>> alter, treatment of data flow in the network. Because of that, I
> >>>> believe, OAM flag should not be used for marking as that will force
> >>>> punting marked packets from fast forwarding path to the control
> >>>> plane. But it might be good to have a field in NVO3 header that may
> >>>> be used for marking and not affect forwarding of packets if altered.
> >>>>
> >>>> Regards,
> >>>>
> >>>> Greg
> >>>>
> >>>>
> >>>>
> >>>> it is used for Ping/Trace similar applications. This kind of OAM
> >>>> verification, normally when network failure occurs. There is
> >>>> another OAM requirements of real time OAM or synthesizing OAM. It
> >>>> can be used for
> >>>> NVO3 encapsulation, and gets packet statistics with the real time
> >>>> OAM flag marking. By comparing the packet number of ingress NVE and
> >>>> the sum of all egress NVEs, packet loss can be deduced. This method
> >>>> can be applicable for both unicast and multicast traffic. Local
> >>>> policy on ingress NVE is configured by operators or automatically

---

From: Haoweiguo <[email-masked]>
Date: Thu, 13 Nov 2014 05:57:25 +0000

Hi Mach & Marc,
Comments on the following question, pls see inline with [weiguo].

[weiguo]: IMO, i would perfer the marking bits in NVO3 header rather than in outer IP/IPV6 header. If we want to do performance measurements for only one tenants or some specified tenants, the marking field only should be in NVO3 header, this can be called overlay network performance measurements. IP/IPV6 header marking can be used for underlay network performance measurements, it can't discriminate different tenants traffic.

Thanks
weiguo

________________________________________
发件人: Mach Chen
发送时间: 2014年11月13日 9:06
收件人: Marc Binderberger
抄送: Tom Herbert; Greg Mirsky; Haoweiguo; [email-masked]; Larry Kreeger
主题: RE: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hi Marc,

Yes

Best regards,
Mach

---

From: Erik Nordmark <[email-masked]>
Date: Thu, 13 Nov 2014 17:27:10 -1000

On 11/11/14 1:51 PM, Haoweiguo wrote:
> field, it is hard to realize in-band performance measurement if using 
> optional field for marking. For other real time congestion control 
> function, maybe more bits are needed.
>

There is some work that describes how the ECN bits in the outer header 
can be used for tunnels.
See draft-briscoe-tsvwg-ecn-encap-guidelines

That provides one bit for in-band measurements over the tunnel.

   Erik
>
> Thanks
>
> weiguo
>
> ------------------------------------------------------------------------
> some kind of flow ID?  If not a single flag, how large do you see the 
> field being?
>
> From: Haoweiguo <[email-masked] <mailto:[email-masked]>>
> To: Greg Mirsky <[email-masked] <mailto:[email-masked]>>
> Cc: "[email-masked] <mailto:[email-masked]>" <[email-masked] 
> <mailto:[email-masked]>>
> The real time OAM is passive performance measurement methods. I would 
> like NVO3 data encapsulation has a field for marking and not affect 
> measurement. The NVO3 packet with this marking flag don't need to be 
> sent to control plane, it is different from OAM(ping/Trace) packet 
> processing.
>
> Thanks
>
> weiguo
>
> ------------------------------------------------------------------------
> *发件人:* Greg Mirsky [[email-masked] 
> <mailto:[email-masked]>]
> *发送时间:* 2014年11月12日 4:07
> *收件人:* Haoweiguo
> *抄送:* [email-masked] <mailto:[email-masked]>
> that, I believe, OAM flag should not be used for marking as that will 
> force punting marked packets from fast forwarding path to the control 
> plane. But it might be good to have a field in NVO3 header that may be 
> used for marking and not affect forwarding of packets if altered.
>
> Regards,
> Greg
>
>     Currently NVO3 data plane encapsulation only includes one OAM
>     flag, it is used for Ping/Trace similar applications. This kind of
>     OAM application is initiated by operators for network connectivity
>     can be used for packet loss detection in real time. When ingress
>     NVE receives traffic from local TS, it gets packet statistics, and
>     mark(coloring) the OAM flag relying on local policy when it
>     performs NVO3 encapsulation. When egress NVEs receives the
>     traffic, it decapsulates NVO3 encapsulation, and gets packet
>     statistics with the real time OAM flag marking. By comparing the
>     packet number of ingress NVE and the sum of all egress NVEs,
>     packet loss can be deduced. This method can be applicable for both
>     unicast and multicast traffic. Local policy on ingress NVE is
>     [email-masked] <mailto:[email-masked]>

---

From: Haoweiguo <[email-masked]>
Date: Fri, 14 Nov 2014 07:31:25 +0000

Hi Eric,
I would prefer the marking bit in NVO3 header, rather than in outer IP header.
This is for overlay network performance measurements, not for underlay network.
In ingress NVE, marking policy should support discrimination between different tenants,even support discrimation different applications of same tenant.
In this case, marking bit can only be set in NVO3 header. And just as Mach and Greg's description, two bits are necessary, one bit for packet loss detection, another bit for packet latency detection.
Thanks
weiguo

_____________________________
发件人: Erik Nordmark [[email-masked]]
发送时间: 2014年11月14日 11:27
收件人: Haoweiguo; Larry Kreeger (kreeger); Greg Mirsky
抄送: [email-masked]
主题: Re: [nvo3] 答复:  Comments on NVO3 data plane requirements for OAM

   Erik

---

From: Tom Herbert <[email-masked]>
Date: Fri, 14 Nov 2014 00:26:36 -0800

> Here is a reference https://datatracker.ietf.org/doc/draft-chen-ippm-coloring-based-ipfpm-framework/.

Thanks for the pointer. Regarding the need for synchronized clocks to
measure delay, I consulted our local NTP expert. The host clock jitter
we currently see in our network is currently usually greater than
one-way packet delay (in some cases much greater), so in his words:
"measuring one-way packet delays using host clocks is a lost cause".
Please take this as just one data point!

> Yes, it does need some synchronization. As for the range, it depends on two factors, one is the implementation limitation, the other the requirement of the operators. In the above reference, the su…
I think if we were implementing delay measurement in GUE, I would
advocate add a 64 bit optional field for timestamp, probably
containing source time stamp, and echoed timestamp for a flow (usec
resolution and similar in design TCP timestamp option). This easily
gives a precise RTT, and if clocks are precisely synchronized then one
way latency could be calculated also.

Thanks,
Tom

---

From: Mach Chen <[email-masked]>
Date: Fri, 14 Nov 2014 09:22:04 +0000

Hi Tom,

Please see my replies inline...

> Sent: Friday, November 14, 2014 4:27 PM
> To: Mach Chen
> Cc: Marc Binderberger; Greg Mirsky; Haoweiguo; [email-masked]; Larry Kreeger
> >> > packet, which helps the receiver to pick the right packet?  And
> >> > repeat this every time period T ?
> >> >
> >> work? Seems like there would need to be quite a bot of
> >> synchronization needed between end points (synchronized clocks,
> >> provisions to correlate measurements correctly with lost packets,
> >> replicated packets, etc.). Also, what is envisioned for range for the period?
> >
> > Here is a reference
> https://datatracker.ietf.org/doc/draft-chen-ippm-coloring-based-ipfpm-framew
> ork/.
> >
> 
> Thanks for the pointer. 

You're welcome!

> Regarding the need for synchronized clocks to measure
> delay, I consulted our local NTP expert. The host clock jitter we currently see in
> our network is currently usually greater than one-way packet delay (in some
> cases much greater), so in his words:

Sure, thanks for this data input.

> 
> > Yes, it does need some synchronization. As for the range, it depends on two
> factors, one is the implementation limitation, the other the requirement of the
> operators. In the above reference, the suggested periods are 1s, 10s, 1min,
> 10min and 1h.
> >
> I think if we were implementing delay measurement in GUE, I would advocate
> add a 64 bit optional field for timestamp, probably containing source time stamp,
> and echoed timestamp for a flow (usec resolution and similar in design TCP
> timestamp option). This easily gives a precise RTT, and if clocks are precisely
> synchronized then one way latency could be calculated also.

Yes, if GUE or other encaps can give such a generous field, that would greatly simplify delay measurement, not matter for NTT or one-way delay. 

Best regards,
Mach

> >> >>> On Wed, Nov 12, 2014 at 12:55 AM, Mach Chen
> >> >>> <[email-masked]>
> >> >>> wrote:
> >> >>>> Hi Greg and all,
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> Single bit is not sufficient if someone wants to perform loss
> >> >>>> and delay measurement  simultaneously, then two bits needed.
> >> >>>>
> >> >>> Is that necessary? Can they share the same time quantum (as well
> >> >>> as other metrics maybe to be added later)? In all the protocols
> >> >> Given one bit is used for both loss and delay measurement, for
> >> >> loss measurement, it periodically set and clear the marking bit, a
> >> >> flow is divided into consecutive blocks, and then the counting and
> >> >> For delay measurement, it has to make sure the timestamps
> >> >> (collected at sender and receiver) are for the same packet.
> >> >> Presumably, the time when changing the marking bit is right time to get the
> timestamps.
> >> >>>> On Tue, Nov 11, 2014 at 3:51 PM, Haoweiguo
> >> >>>> <[email-masked]>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi Larry,
> >> >>>>
> >> >>>> For marking purpose, i think one bit maybe OK, fixed fields in
> >> >>>> NVO3 header is precious. I would like it is set in fixed field,
> >> >>>> rather than in option field. Because chipset normally can't
> >> >>>> process optional field, it is hard to realize in-band
> >> >>>> performance measurement if using optional
> >> >>> field for marking.
> >> >>>> What do you envision this marking looking like?  e.g. is it just
> >> >>>> a single flag bit, or large field with a counter or sequence
> >> >>>> number, or some kind of flow ID?  If not a single flag, how
> >> >>>> large do you see the field
> >> >>> being?
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> mandatory for all implementations, then it seems to fall into
> >> >>>> the category of optional extensions.
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> Thanks, Larry
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> NVO3 packet with this marking flag don't need to be sent to
> >> >>>> control plane, it is different from OAM(ping/Trace) packet processing.
> >> >>>>
> >> >>>> Thanks
> >> >>>>
> >> >>>> weiguo
> >> >>>>
> >> >>>> facilitate measurement of some performance metric, whether loss
> >> >>>> or delay/delay variation, may be viewed as one of passive
> >> >>>> performance
> >> >>> measurement methods.
> >> >>>> believe, OAM flag should not be used for marking as that will
> >> >>>> force punting marked packets from fast forwarding path to the
> >> >>>> control plane. But it might be good to have a field in NVO3
> >> >>>> header that may be used for marking and not affect forwarding of
> packets if altered.
> >> >>>>
> >> >>>> Regards,
> >> >>>>
> >> >>>> Greg
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Tue, Nov 11, 2014 at 12:34 AM, Haoweiguo
> >> >>>> <[email-masked]>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi All,
> >> >>>>
> >> >>>> I maybe not clearly said in today’s NVO3 meeting, pls allow me
> >> >>>> to reiterate the OAM data plane requirements on the mail list.
> >> >>>>
> >> >>>> flag, it is used for Ping/Trace similar applications. This kind
> >> >>>> of OAM application is initiated by operators for network
> >> >>>> connectivity verification, normally when network failure occurs.
> >> >>>> There is another OAM requirements of real time OAM or
> >> >>>> synthesizing OAM. It can be used for
> >> >>>> statistics, and mark(coloring) the OAM flag relying on local
> >> >>>> policy when it performs
> >> >>>> NVO3 encapsulation, and gets packet statistics with the real
> >> >>>> time OAM flag marking. By comparing the packet number of ingress
> >> >>>> NVE and the sum of all egress NVEs, packet loss can be deduced.
> >> >>>> This method can be applicable for both unicast and multicast
> >> >>>> traffic. Local policy on ingress NVE is configured by operators
> >> >>>> or automatically acquired from centralized orchestration.
> >> >>>>
> >> >>>> Thanks
> >> >>>>
> >> >>>> weiguo
> >> >>>>
> >> >>>>

---

From: Vero Zheng <[email-masked]>
Date: Fri, 14 Nov 2014 09:47:46 +0000

Hi Tom,

Please see in-line.

BR, Vero

> From: nvo3 [mailto:[email-masked]] On Behalf Of Tom Herbert
> Cc: Greg Mirsky; Haoweiguo; Marc Binderberger; Larry Kreeger; [email-masked]
> Thanks for the pointer. Regarding the need for synchronized clocks to measure

[Vero] Thanks for this. What about the current experience with 1588v2 then?
> add a 64 bit optional field for timestamp, probably containing source time
> stamp, and echoed timestamp for a flow (usec resolution and similar in design
> TCP timestamp option). This easily gives a precise RTT, and if clocks are
> precisely synchronized then one way latency could be calculated also.
>
[Vero] If the source timestamp could be carried, it could also be used for packet loss calculation/correlation.

> >> >> Presumably, the time when changing the marking bit is right time to get
> the timestamps.
> >> >>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> requirements
> >> >>>> for OAM
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> Dear All,
> >> >>>> Subject: [nvo3] 答复: Comments on NVO3 data plane requirements
> for
> >> >>>> OAM
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> Hi Greg,
> >> >>>>
> >> >>>> I fully agree with you.
> >> >>>>

---

From: Jon Hudson <[email-masked]>
Date: Fri, 14 Nov 2014 11:09:14 -1000

One comment in line....

> On Nov 13, 2014, at 11:47 PM, Vero Zheng <[email-masked]> wrote:
> 
> Hi Tom,
> 
> Please see in-line.
> 
> BR, Vero
> 

<Jon> Thank you. As someone who has managed NTP more times and for more years than I care to admit, this is a very good datapoint to consider. NTP helps many understand that time is relative.

---

From: "Deepak Kumar (dekumar)" <[email-masked]>
Date: Sun, 16 Nov 2014 16:44:54 +0000

Hi,

Please see inline +++DK:

On 11/14/14 11:09 AM, "Jon Hudson" <[email-masked]> wrote:

>>>measure
><Jon> Thank you. As someone who has managed NTP more times and for more
>years than I care to admit, this is a very good datapoint to consider.
>NTP helps many understand that time is relative.

+++DK: As per our experience in carrier Ethernet we supported one way
delay and never found NTP useful even for our lab networks (I am referring
software based NTP NTPv3).
As mentioned below IEEE 1588v2 will vary based on equipment and operator
networks but in our testing we found it very precise if properly deployed.
IEEE 1588v2 is very precise if phy based timestamping is used. Even
timestamping at NP level provided great results for one way delay.

If we want to accurately measure two way delay we need 4 timestamp total
on receiver of frame (this is to avoid processing time that's taken for
reply by software as hardware can put timestamp at lower layer without
doing delay and jitter calculation).
For one way delay we will require 2 timestamp, so lower layer hardware can
timestamp before packet is punted to software.

As mentioned below I agree 8 byte IEEE 1588 timestamp is required.

We should also look for Synthetic OAM applicability for performance ('O'
bit can be overloaded to do both Fault and performance if OAM is defined
with different oam Subtype for Delay and Loss frames and it will not be
too deep hardware inspection) as that give large flexibility
(synthetic/real loss measurement, Availability/unavailability, on-demand
and pro-active performance) and can be run on all flows of ECMP.

Thanks,
Deepak
>>>advocate

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 17 Nov 2014 00:01:43 -0800

Hello Deepak et al.,

so this sounds like we need more than just a (2nd) bit for delay measurement. 
Seems we need an optional header extension or a TLV to carry all the 
information (timestamps, oam Subtype). Sounds definitely more than a 32/64bit 
header could carry (*).

The optional header extension, when done similar to GUE, has a fixed 
position. For the TLV this would be an additional requirement. This would 
allow for hardware-stamping.

Now if we introduce such an OAM extension header it could as well carry the 
"first" bit we discussed for packet loss measurement (?).

Regards, Marc

(*: at least all proposals so far have a base header that fits into 32/64 
bit, plus IP and potential UDP)

---

From: Greg Mirsky <[email-masked]>
Date: Mon, 17 Nov 2014 00:24:34 -0800

Hi Deepak,
very interesting and helpful to learn from your experience with
latency/jitter measurement.
But I think that there are different levels of details:

   - NVO3 packet format;
   - NVO3 OAM requirements and gap analysis.

For the former, I think OAM flag and two bit long marking field that can be
used in passive performance measurements are sufficient to support both
active and passive FM and PM.

As far as latency/jitter measurement, we can look into re-using work of
IPPM and MPLS WGs as much as possible. OWAMP/TWAMP, on one hand, and RFC
6374, on another, should be mapped to NVO3 OAM requirements, IMO. True, we
may find that support of both one-way and two-way PM methods required. I'd
just note, that one-way latency/jitter measurement does require clock
synchronization. Two-way can measure round-trip with or without good clock
synchronization between sender and reflector. If the former case, forward
and reverse latency/jitter as well as two-way can be measured and
calculated. If the latter - round trip only. Personally, I find one-sided
two-way PM more useful.

As far as timestamp format, approach of RFC 6374 seems as most flexible as
it allows use of NTP and truncated 1588v2 timestamps.

Regards,

Greg

---

From: Marc Binderberger <[email-masked]>
Date: Mon, 17 Nov 2014 01:06:02 -0800

Hello Weiguo,

I understand - and expected - your comment :-)

Let me ignore for a moment that a full OAM/performance probably requires a 
header extension (for timestamps etc), which means we could define this as an 
extension of the NVO3 overlay header anyway. Let's assume we talk about just 
one bit for packet loss.

As far as I understand the NVO3 header does not exists on it's own, it is 
tightly coupled with an IP (v4/v6) header. In other words they always come 
together. Then having the flag in the IP header would not affect the ability 
to measure per VNI.

But maybe I'm missing something (?).

Regards, Marc

On Thu, 13 Nov 2014 05:57:25 +0000, Haoweiguo wrote:
> Regarding NVE header of IP/IPv6 header, I am not sure which is better. One 
> challenge for using IPv4/IPv6 header is that there is not too much bits 
> that can be used for marking, especially for IPv4.
> 
> [weiguo]: IMO, i would perfer the marking bits in NVO3 header rather than 
> in outer IP/IPV6 header. If we want to do performance measurements for only 
> one tenants or some specified tenants, the marking field only should be in 
> NVO3 header, this can be called overlay network performance measurements. 
> IP/IPV6 header marking can be used for underlay network performance 
> measurements, it can't discriminate different tenants traffic.
> 
> Thanks
> weiguo
> 
> 
> 
>> which
> Regarding NVE header of IP/IPv6 header, I am not sure which is better. One 
> challenge for using IPv4/IPv6 header is that there is not too much bits 
> that can be used for marking, especially for IPv4.
> 
> Best regards,
> Mach
> 
>> 
>> 
>> Thanks & Regards,
>> Marc
>> 
>> 
>> 
>>>>> field
>>>>> like

---

From: Tom Herbert <[email-masked]>
Date: Mon, 17 Nov 2014 08:02:21 -0800

> allow for hardware-stamping.
The alternative is to do active delay measurement using request/reply.
We should be able to define the requirements so that an OAM message
corresponding to a flow which would be routed in exactly the same way
as a data message for the flow. Larry mentioned that we might even
want to put a "fake" packet header as the first part of the
encapsulated payload of an OAM message for instance.

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 17 Nov 2014 10:35:36 -0800

> way latency could be calculated also.

Tom,

Unless you are going to have an immediate echo packet, then you need 3 
timestamps in the packet:
1. timestamp when A sent packet to B
2. timestamp when B received packet from A
3. timestamp when B sent packet to A (different than #2 unless you send 
a immediate response)

A will also take a timestamp when it receives the packet from B, but 
that doesn't have to be included in the packet.

   Erik

---

From: Erik Nordmark <[email-masked]>
Date: Mon, 17 Nov 2014 10:40:00 -0800

> This is for overlay network performance measurements, not for underlay network.

My understanding is that you want to measure from the encapsulating NVE 
to the decapsulating NVE. Is that correct?

The outer IP+NVO3 headers travel from NVE to NVE, thus I don't 
understand why we should consider the fields in that outer header 
differently than the fields in the NVO3 header.

> In ingress NVE, marking policy should support discrimination between different tenants,even support discrimation different applications of same tenant.

Are you concerned about the DSCP bit? (Those are different than the ECN 
bits).

    Erik

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Mon, 17 Nov 2014 16:47:43 +0000

I think we are complicating OAM beyond what it is needed.

As far as packet encapsulation is concern, all what is needed is single bit. This bit is needed to prevent OAM packets leaking out from the domain.

Termination of OAM and processing of it happen based on the addressing in the packet.

E.g. if Address matches and OAM bit is set then it is an OAM packet addressed to the local MEP/MP.

Not other way around. Why? Because we want OAM to be as closely as possible follow the Data path.

If we need to have performance and delay measurements, we SHOULD NOT mutate the packet header.

Instead OAM specific extensions should be in the OAM shim.

As an example. You could have packet fragment (which is sometimes called flow entropy) and at the end of that you can have all of the stuff you need in the world of OAM.

Hope this clarify

Thanks
Tissa
-----Original Message-----
From: nvo3 [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Monday, November 17, 2014 8:02 AM
To: Marc Binderberger
Cc: Greg Mirsky; Mach Chen; Deepak Kumar (dekumar); [email-masked]; Haoweiguo; Larry Kreeger (kreeger); Vero Zheng; Jon Hudson
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

On Mon, Nov 17, 2014 at 12:01 AM, Marc Binderberger <[email-masked]> wrote:
> information (timestamps, oam Subtype). Sounds definitely more than a 
> 32/64bit header could carry (*).
>
> position. For the TLV this would be an additional requirement. This 
> would allow for hardware-stamping.
>
The alternative is to do active delay measurement using request/reply.
We should be able to define the requirements so that an OAM message corresponding to a flow which would be routed in exactly the same way as a data message for the flow. Larry mentioned that we might even want to put a "fake" packet header as the first part of the encapsulated payload of an OAM message for instance.

> Now if we introduce such an OAM extension header it could as well 
> carry the "first" bit we discussed for packet loss measurement (?).
>
>
> Regards, Marc
>
> (*: at least all proposals so far have a base header that fits into 
> 32/64 bit, plus IP and potential UDP)
>
>
>
>
>>>>>>> On Wed, Nov 12, 2014 at 2:11 AM, Marc Binderberger 
>>>>>>> <[email-masked]>
>>>>> wrote:
>>>>>>>> Hello Mach,
>>>>>>>>
>>>>>>>> so for delay measurement you use the color flag to mark a 
>>>>>>>> single packet, which helps the receiver to pick the right 
>>>>>>>> packet?  And repeat this every time period T ?
>>>>>>>>
>>>>>>> Is there there a draft or description of how this algorithm 
>>>>>>> would work? Seems like there would need to be quite a bot of 
>>>>>>> provisions to correlate measurements correctly with lost 
>>>>>>> packets, replicated packets, etc.). Also, what is envisioned for 
>>>>>>> range for the period?
>>>>>>
>>>>>> Here is a reference
>>>>>
>>>>> https://datatracker.ietf.org/doc/draft-chen-ippm-coloring-based-ip
>>>>> fpm-fr
>>>>> amew
>>>>> ork/.
>>>>>
>>>>> Thanks for the pointer. Regarding the need for synchronized clocks 
>>>>> to measure delay, I consulted our local NTP expert. The host clock 
>>>>> jitter we currently see in our network is currently usually 
>>>>> greater than one-way packet delay (in some cases much greater), so 
>>>>> in his words:
>>> <Jon> Thank you. As someone who has managed NTP more times and for 
>>> more years than I care to admit, this is a very good datapoint to consider.
>> delay and never found NTP useful even for our lab networks (I am 
>> referring software based NTP NTPv3).
>> As mentioned below IEEE 1588v2 will vary based on equipment and 
>> operator networks but in our testing we found it very precise if properly deployed.
>> If we want to accurately measure two way delay we need 4 timestamp 
>> total on receiver of frame (this is to avoid processing time that's 
>> taken for reply by software as hardware can put timestamp at lower 
>> layer without doing delay and jitter calculation).
>> For one way delay we will require 2 timestamp, so lower layer 
>> hardware can timestamp before packet is punted to software.
>>
>> bit can be overloaded to do both Fault and performance if OAM is 
>> defined with different oam Subtype for Delay and Loss frames and it 
>> will not be too deep hardware inspection) as that give large 
>> flexibility (synthetic/real loss measurement, 
>> Availability/unavailability, on-demand and pro-active performance) and can be run on all flows of ECMP.
>>
>> Thanks,
>> Deepak
>>>
>>>
>>>>
>>>>> factors, one is the implementation limitation, the other the 
>>>>> requirement of the operators. In the above reference, the 
>>>>> suggested periods are 1s, 10s, 1min, 10min and 1h.
>>>>> containing source time stamp, and echoed timestamp for a flow 
>>>>> (usec resolution and similar in design TCP timestamp option). This 
>>>>> easily gives a precise RTT, and if clocks are precisely 
>>>> [Vero] If the source timestamp could be carried, it could also be 
>>>> used for packet loss calculation/correlation.
>>>>
>>>>> Thanks,
>>>>> Tom
>>>>>
>>>>>> Best regards,
>>>>>> Mach
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Tom
>>>>>>>
>>>>>>>>
>>>>>>>> One question I still have is: why is the measurement done in 
>>>>>>>> the NVE
>>>>> header?
>>>>>>>>>> requirements for OAM
>>>>>>>>>>> Single bit is not sufficient if someone wants to perform 
>>>>>>>>>>> loss and delay measurement  simultaneously, then two bits needed.
>>>>>>>>>> Is that necessary? Can they share the same time quantum (as 
>>>>>>>>>> well as other metrics maybe to be added later)? In all the 
>>>>>>>>>> protocols mentioned, the reserved bits are a somewhat precious resource.
>>>>>>>>>
>>>>>>>>> loss measurement, it periodically set and clear the marking 
>>>>>>>>> bit, a flow is divided into consecutive blocks, and then the 
>>>>>>>>> counting and calculating are based on each block. This is fine 
>>>>>>>>> for loss measurement.
>>>>>>>>>
>>>>>>>>> Since there is ECMP, the first packet of a block at the sender 
>>>>>>>>> may probably different from the first packet at the receiver, 
>>>>>>>>> thus it will get the mismatched timestamps to calculate the delay.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Mach
>>>>>>>>>>
>>>>>>>>>> Tom
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>>
>>>>>>>>>>> Mach
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> NVO3 header is precious. I would like it is set in fixed 
>>>>>>>>>>> field, rather than in option field. Because chipset normally 
>>>>>>>>>>> can't process optional field, it is hard to realize in-band 
>>>>>>>>>>> What do you envision this marking looking like?  e.g. is it 
>>>>>>>>>>> just a single flag bit, or large field with a counter or 
>>>>>>>>>>> sequence number, or some kind of flow ID?  If not a single 
>>>>>>>>>>> flag, how large do you see the field
>>>>>>>>>> being?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If it is more than a flag (and I assume it would be), and is 
>>>>>>>>>>> not mandatory for all implementations, then it seems to fall 
>>>>>>>>>>> into the category of optional extensions.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks, Larry
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> NVO3 data encapsulation has a field for marking and not 
>>>>>>>>>>> affect forwarding of packets, the marking field is only used 
>>>>>>>>>>> for performance measurement. The
>>>>>>>>>>> marking groups of packets that belong to the particular flow 
>>>>>>>>>>> to facilitate measurement of some performance metric, 
>>>>>>>>>>> whether loss or delay/delay variation, may be viewed as one 
>>>>>>>>>>> of passive performance
>>>>>>>>>> measurement methods.
>>>>>>>>>>> But such marking should not alter, at least not 
>>>>>>>>>>> significantly alter, treatment of data flow in the network. 
>>>>>>>>>>> Because of that, I believe, OAM flag should not be used for 
>>>>>>>>>>> marking as that will force punting marked packets from fast 
>>>>>>>>>>> forwarding path to the control plane. But it might be good 
>>>>>>>>>>> to have a field in NVO3 header that may be used for marking 
>>>>>>>>>>> I maybe not clearly said in today’s NVO3 meeting, pls allow 
>>>>>>>>>>> me to reiterate the OAM data plane requirements on the mail list.
>>>>>>>>>>>
>>>>>>>>>>> Currently NVO3 data plane encapsulation only includes one 
>>>>>>>>>>> OAM flag, it is used for Ping/Trace similar applications. 
>>>>>>>>>>> This kind of OAM application is initiated by operators for 
>>>>>>>>>>> network connectivity verification, normally when network failure occurs.
>>>>>>>>>>> When ingress NVE receives traffic from local TS, it gets 
>>>>>>>>>>> packet statistics, and mark(coloring) the OAM flag relying 
>>>>>>>>>>> on local policy when it performs
>>>>>>>>>>> time OAM flag marking. By comparing the packet number of 
>>>>>>>>>>> traffic. Local policy on ingress NVE is configured by 
>>>>>>>>>>> operators or automatically acquired from centralized orchestration.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> weiguo
>>>>>>>>>>>
>>>>>>>>>>>

---

From: Greg Mirsky <[email-masked]>
Date: Mon, 17 Nov 2014 09:32:18 -0800

Dear All,
I would like to point that this discussion, though very interesting, may be
premature. I believe that the WG should first agree on NVO3 OAM
Requirements and conclude OAM gap analysis. I believe that NVO3 Operations,
Administration, and Maintenance Requirements
draft-ashwood-nvo3-oam-requirements-01 is good starting point and may be
considered for WG adoption. Once WG agrees on the OAM requirements we can
start the gap analysis.

Regards,
Greg

On Mon, Nov 17, 2014 at 8:47 AM, Tissa Senevirathne (tsenevir) <
[email-masked]> wrote:

> As far as packet encapsulation is concern, all what is needed is single
> bit. This bit is needed to prevent OAM packets leaking out from the domain.
>
> As an example. You could have packet fragment (which is sometimes called
> flow entropy) and at the end of that you can have all of the stuff you need
> in the world of OAM.
>
> Hope this clarify
>
> Thanks
> Tissa
> Cc: Greg Mirsky; Mach Chen; Deepak Kumar (dekumar); [email-masked];
> Haoweiguo; Larry Kreeger (kreeger); Vero Zheng; Jon Hudson
> corresponding to a flow which would be routed in exactly the same way as a
> data message for the flow. Larry mentioned that we might even want to put a
> "fake" packet header as the first part of the encapsulated payload of an
> OAM message for instance.
>

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Mon, 17 Nov 2014 17:41:23 +0000

I think discussion on what should be on the packet header can happen in parallel. –requirement document has been around over an year and I am sure interested people have digested and commented.

Keeping packet header discussions to later may be troublesome as some of these needed to be built in to hardware and once casted in sand (silicon) hard to change

From: Greg Mirsky [mailto:[email-masked]]
Sent: Monday, November 17, 2014 9:32 AM
To: Tissa Senevirathne (tsenevir)
Cc: Tom Herbert; Marc Binderberger; Mach Chen; Deepak Kumar (dekumar); [email-masked]; Haoweiguo; Larry Kreeger (kreeger); Vero Zheng; Jon Hudson
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Dear All,
I would like to point that this discussion, though very interesting, may be premature. I believe that the WG should first agree on NVO3 OAM Requirements and conclude OAM gap analysis. I believe that NVO3 Operations, Administration, and Maintenance Requirements draft-ashwood-nvo3-oam-requirements-01 is good starting point and may be considered for WG adoption. Once WG agrees on the OAM requirements we can start the gap analysis.
Regards,
Greg

On Mon, Nov 17, 2014 at 8:47 AM, Tissa Senevirathne (tsenevir) <[email-masked]<mailto:[email-masked]>> wrote:
I think we are complicating OAM beyond what it is needed.

Hope this clarify

Thanks
Tissa
-----Original Message-----
From: nvo3 [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Tom Herbert
Sent: Monday, November 17, 2014 8:02 AM
To: Marc Binderberger
Cc: Greg Mirsky; Mach Chen; Deepak Kumar (dekumar); [email-masked]<mailto:[email-masked]>; Haoweiguo; Larry Kreeger (kreeger); Vero Zheng; Jon Hudson
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

>> On 11/14/14 11:09 AM, "Jon Hudson" <[email-masked]<mailto:[email-masked]>> wrote:
>>
>>>
>>> One comment in line....
>>>
>>>> On Nov 13, 2014, at 11:47 PM, Vero Zheng <[email-masked]<mailto:[email-masked]>> wrote:
>>>>
>>>> Hi Tom,
>>>>
>>>> Please see in-line.
>>>>
>>>> BR, Vero
>>>>
>>>>> From: nvo3 [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Tom Herbert
>>>>> Cc: Greg Mirsky; Haoweiguo; Marc Binderberger; Larry Kreeger;
>>>>> On Wed, Nov 12, 2014 at 5:13 PM, Mach Chen <[email-masked]<mailto:[email-masked]>>
>>>>> wrote:
>>>>>> Hi Tom,
>>>>>>
>>>>>>> From: Tom Herbert [mailto:[email-masked]<mailto:[email-masked]>]
>>>>>>> Cc: Mach Chen; Greg Mirsky; Haoweiguo; [email-masked]<mailto:[email-masked]>; Larry
>>>>>>> Kreeger
>>>>>>> <[email-masked]<mailto:[email-masked]>>
>>>>> wrote:
>>>>>>>> Hello Mach,
>>>>>>>>
>>>>>>>>>> From: Tom Herbert [mailto:[email-masked]<mailto:[email-masked]>]
>>>>>>>>>> Cc: Greg Mirsky; Haoweiguo; [email-masked]<mailto:[email-masked]>; Larry Kreeger
>>>>>>>>>> (kreeger)
>>>>>>>>>> <[email-masked]<mailto:[email-masked]>>
>>>>>>>>>> wrote:
>>>>>>>>>>> Hi Greg and all,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From: nvo3 [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Greg
>>>>>>>>>>> Mirsky
>>>>>>>>>>> Cc: [email-masked]<mailto:[email-masked]>; Larry Kreeger (kreeger)
>>>>>>>>>>> <[email-masked]<mailto:[email-masked]>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Larry,
>>>>>>>>>>>
>>>>>>>>>>> <[email-masked]<mailto:[email-masked]>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>

---

From: Greg Mirsky <[email-masked]>
Date: Mon, 17 Nov 2014 09:58:44 -0800

Hi Tissa,
and those vendors implementing pre-standard ideas should fully understand
the risk of ending up on proprietary path.
The document I've pointed to been around and, IMO, can be called for WG
adoption.
Discussion of the header certainly can continue but, I believe, it should
be gated by work on OAM Requirements and Gap Analysis documents and their
progress.

Regards,
Greg

On Mon, Nov 17, 2014 at 9:41 AM, Tissa Senevirathne (tsenevir) <
[email-masked]> wrote:

>  I think discussion on what should be on the packet header can happen in
> parallel. –requirement document has been around over an year and I am sure
> interested people have digested and commented.
>
>
>
> Keeping packet header discussions to later may be troublesome as some of
> these needed to be built in to hardware and once casted in sand (silicon)
> hard to change
>
>
>
> *From:* Greg Mirsky [mailto:[email-masked]]
> *Sent:* Monday, November 17, 2014 9:32 AM
> *To:* Tissa Senevirathne (tsenevir)
> *Cc:* Tom Herbert; Marc Binderberger; Mach Chen; Deepak Kumar (dekumar);
> [email-masked]; Haoweiguo; Larry Kreeger (kreeger); Vero Zheng; Jon Hudson
> *Subject:* Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM
>
>
>
> Dear All,
>
> I would like to point that this discussion, though very interesting, may
> be premature. I believe that the WG should first agree on NVO3 OAM

---

From: "Deepak Kumar (dekumar)" <[email-masked]>
Date: Mon, 17 Nov 2014 20:39:05 +0000

I Agree with Tissa below. My Goal also was to point out that instead of
complicating the header, we can do OAM performance within OAM channel
itself and this is extensible and can be done in hardware which is why
mostly things are added in header.

Also, Operators keep asking for new OAM tools (Fault detection,
verification, isolation, Interworking, alarm, putting service in
maintenance and perform test)  and Performance tools, eg: (Delay/Jitter,
Actual Loss Measurement, Synthetic Loss, loopback signaling like TDM,
Generate frames to verify qos etc.) and so OAM Channel solution will be
extensible.

Thanks,
Deepak

On 11/17/14 8:47 AM, "Tissa Senevirathne (tsenevir)" <[email-masked]>
wrote:

>flow entropy) and at the end of that you can have all of the stuff you
>need in the world of OAM.
>
>Hope this clarify
>
>Thanks
>Tissa
>corresponding to a flow which would be routed in exactly the same way as
>a data message for the flow. Larry mentioned that we might even want to
>put a "fake" packet header as the first part of the encapsulated payload
>of an OAM message for instance.
>

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 18 Nov 2014 15:02:58 -0800

Hi Tapraj,
though I agree and support with idea of having OAM flag in NVO3 header I
have to point to:

   - absence of WG agreed upon OAM Requirements;
   - no gap analysis of tools for NVO3 OAM;
   - OAM flag does not help passive performance measurement marking method
   (two bit-long field for marking in fixed position).

I agree that PW VCCV and GAL/G-ACh can be viewed as MPLS identification of
OAM packet (though not necessarily OAM). But IP clearly doesn't have such
identification for OAM and that, in part, why in-band requirement for IP
OAM, both FM and Active PM, is not attainable (ECMP environment).

Regards,
Greg

On Tue, Nov 18, 2014 at 1:31 PM, Tapraj Singh <[email-masked]> wrote:

> Hi All,
>
>  I totally agree with the point made by Deepak and Tissa here.
> Our OAM should follow the data path for services as much as possible and
> all
> other protocol specific information should be in the OAM protocol specific
> TLVs.
>
> LAYER2 OAM
>
> In term of identify the OAM packet, first level of identification for L2
> OAM
> Should be the MAC address and send level of hierarchy should be the ether
> type or OUI.
> No other OAM Specific field should be allowed in the packet header.
>
>  Please note that L3 OAM and MPLS also follow the same principle.
>
> Thanks
> Tapraj
>
> On 11/17/14 12:39 PM, "Deepak Kumar (dekumar)" <[email-masked]> wrote:
>

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 18 Nov 2014 15:34:41 -0800

Hi Tapraj,
perhaps single hop but that is benign case for in-band problem. Multi-hop
BFD cannot ensure in-band because of common practice to hash ECMP on
5-tuple and the fact that multi-hop BFD uses distinct well-port number.
This is well-known and well-understood problem in IP OAM.

Regards,
Greg

On Tue, Nov 18, 2014 at 3:28 PM, Tapraj Singh <[email-masked]> wrote:

>  Hi Greg,
>  BFD for IP protocols also works in the same way. What are other IP OAM
> that
> We are referring here.
>
>    BFD on IPv4 & IPv6
>
> BFD
>
> Destination IP address
>
>
>
> Destination UDP port 3784 (for single hop) + Destination port 4784 for
> (multihop) (Note 1)
>
>  Thanks
> Tapraj
>
>   From: Greg Mirsky <[email-masked]>
> Date: Tuesday, November 18, 2014 3:02 PM
> To: Tapraj Singh <[email-masked]>
>    - OAM flag does not help passive performance measurement marking
>    method (two bit-long field for marking in fixed position).
>
>  I agree that PW VCCV and GAL/G-ACh can be viewed as MPLS identification
> of OAM packet (though not necessarily OAM).
>      But IP clearly doesn't have such identification for OAM and that, in
> part, why in-band requirement for IP OAM,
>
>    both FM and Active PM, is not attainable (ECMP environment).
>
>  Regards,
>  Greg
>
>> >On 11/17/14 8:47 AM, "Tissa Senevirathne (tsenevir)" <[email-masked]
>> >
>> >wrote:
>> >
>> >>Termination of OAM and processing of it happen based on the addressing
>> in
>> >>the packet.
>> >>
>> >>>> Availability/unavailability, on-demand and pro-active performance)
>> and
>> >>>>can be run on all flows of ECMP.
>> >>>>
>> >>>> Thanks,
>> >>>> Deepak
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>>>>>>>>>>> protocols mentioned, the reserved bits are a somewhat
>> precious
>> >>>>>>>>>>>>resource.
>> >>>>>>>>>>>

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Tue, 18 Nov 2014 23:54:08 +0000

Greg

I disagree with you on FM and PM cannot be achieved in ECMP environment. Significant amount of work has gone in to this area during TRILL OAM.  Please check the use of Flow entropy functionality proposed in NVO3 OAM.

https://tools.ietf.org/html/draft-tissa-nvo3-oam-fm-00

From: nvo3 [mailto:[email-masked]] On Behalf Of Greg Mirsky
Sent: Tuesday, November 18, 2014 3:03 PM
To: Tapraj Singh
Cc: [email-masked]
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hi Tapraj,
though I agree and support with idea of having OAM flag in NVO3 header I have to point to:

  *   absence of WG agreed upon OAM Requirements;
  *   no gap analysis of tools for NVO3 OAM;
  *   OAM flag does not help passive performance measurement marking method (two bit-long field for marking in fixed position).
I agree that PW VCCV and GAL/G-ACh can be viewed as MPLS identification of OAM packet (though not necessarily OAM). But IP clearly doesn't have such identification for OAM and that, in part, why in-band requirement for IP OAM, both FM and Active PM, is not attainable (ECMP environment).
Regards,
Greg

On Tue, Nov 18, 2014 at 1:31 PM, Tapraj Singh <[email-masked]<mailto:[email-masked]>> wrote:
Hi All,

LAYER2 OAM

Thanks
Tapraj

On 11/17/14 12:39 PM, "Deepak Kumar (dekumar)" <[email-masked]<mailto:[email-masked]>> wrote:

>On 11/17/14 8:47 AM, "Tissa Senevirathne (tsenevir)" <[email-masked]<mailto:[email-masked]>>
>wrote:
>
>>Cc: Greg Mirsky; Mach Chen; Deepak Kumar (dekumar); [email-masked]<mailto:[email-masked]>;

---

From: Greg Mirsky <[email-masked]>
Date: Tue, 18 Nov 2014 16:19:09 -0800

Hi Tissa,
I think you're misunderstanding or misinterpreting my POV. I'm not saying
that either FM or PM cannot be performed in IP ECMP environment. But I
believe that IP OAM has certain limitations like in case of in-band
requirement. Of course, if one uses tunnels in server layer and maps flows
into tunnels at the edge, then in-band comes for free. Another example that
comes to mind is use of MPLS Entropy label. But I think that such are not
the most generic scenarios for IP network.

Regards,
Greg

On Tue, Nov 18, 2014 at 3:54 PM, Tissa Senevirathne (tsenevir) <
[email-masked]> wrote:

>  Greg
>
>
>
> I disagree with you on FM and PM cannot be achieved in ECMP environment.
> Significant amount of work has gone in to this area during TRILL OAM.
> Please check the use of Flow entropy functionality proposed in NVO3 OAM.
>
>
>
> *From:* nvo3 [mailto:[email-masked]] *On Behalf Of *Greg Mirsky
> *Sent:* Tuesday, November 18, 2014 3:03 PM
> *To:* Tapraj Singh
> *Cc:* [email-masked]
> of OAM packet (though not necessarily OAM). But IP clearly doesn't have
> such identification for OAM and that, in part, why in-band requirement for
> IP OAM, both FM and Active PM, is not attainable (ECMP environment).
>
> Regards,
>
> Greg
>
>
>

---

From: Tom Herbert <[email-masked]>
Date: Tue, 18 Nov 2014 16:19:34 -0800

> https://tools.ietf.org/html/draft-tissa-nvo3-oam-fm-00
Tissa,

If I am reading this correctly, the OAM message would be composed of
the encapsulation header, followed by 128 bytes of which contains a
pseudo header for switching, followed by a self defining OAM message.
The OAM bit is only used at the receiver to distinguish data messages
for OAM messages for processing. Is this interpretation correct?

Thanks,
Tom

> absence of WG agreed upon OAM Requirements;
> no gap analysis of tools for NVO3 OAM;
> OAM flag does not help passive performance measurement marking method (two
> bit-long field for marking in fixed position).
>

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Wed, 19 Nov 2014 00:25:27 +0000

Hi Tom

Your interpretation is correct. The entropy allows OAM packets to follow the same path as the data packet.

As I noted earlier on in the thread, OAM processing would not kick in unless address matches the MEP/MIP. If address match MEP/MIP and OAM bit is set, then OAM processing begins.

-----Original Message-----
From: Tom Herbert [mailto:[email-masked]] 
Sent: Tuesday, November 18, 2014 4:20 PM
To: Tissa Senevirathne (tsenevir)
Cc: Greg Mirsky; Tapraj Singh; [email-masked]; Deepak Kumar (dekumar)
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Tissa,

If I am reading this correctly, the OAM message would be composed of the encapsulation header, followed by 128 bytes of which contains a pseudo header for switching, followed by a self defining OAM message.
The OAM bit is only used at the receiver to distinguish data messages for OAM messages for processing. Is this interpretation correct?

Thanks,
Tom

> absence of WG agreed upon OAM Requirements; no gap analysis of tools 
> for NVO3 OAM; OAM flag does not help passive performance measurement 
> marking method (two bit-long field for marking in fixed position).
>
> I agree that PW VCCV and GAL/G-ACh can be viewed as MPLS 
> identification of OAM packet (though not necessarily OAM). But IP 
> clearly doesn't have such identification for OAM and that, in part, 
> why in-band requirement for IP OAM, both FM and Active PM, is not attainable (ECMP environment).
>
> Regards,
>
> Greg
>
>
>
> Our OAM should follow the data path for services as much as possible 
> and all other protocol specific information should be in the OAM 
> protocol specific TLVs.
>
> LAYER2 OAM
>
> In term of identify the OAM packet, first level of identification for 
> L2 OAM Should be the MAC address and send level of hierarchy should be 
> the ether type or OUI.
>>I Agree with Tissa below. My Goal also was to point out that instead 
>>of complicating the header, we can do OAM performance within OAM 
>>channel itself and this is extensible and can be done in hardware 
>>which is why mostly things are added in header.
>>
>>maintenance and perform test)  and Performance tools, eg: 
>>(Delay/Jitter, Actual Loss Measurement, Synthetic Loss, loopback 
>>signaling like TDM, Generate frames to verify qos etc.) and so OAM 
>>Channel solution will be extensible.
>>
>>Thanks,
>>Deepak
>>
>>>As far as packet encapsulation is concern, all what is needed is 
>>>single bit. This bit is needed to prevent OAM packets leaking out 
>>>from the domain.
>>>
>>>As an example. You could have packet fragment (which is sometimes 
>>>called flow entropy) and at the end of that you can have all of the 
>>>stuff you need in the world of OAM.
>>>
>>>Hope this clarify
>>>
>>>Thanks
>>>Tissa
>>>> Seems we need an optional header extension or a TLV to carry all 
>>>>the  information (timestamps, oam Subtype). Sounds definitely more 
>>>>than a  32/64bit header could carry (*).
>>>>
>>>> The optional header extension, when done similar to GUE, has a 
>>>> fixed position. For the TLV this would be an additional 
>>>> requirement. This would allow for hardware-stamping.
>>>>
>>>>>>>> On Wed, Nov 12, 2014 at 5:13 PM, Mach Chen 
>>>>>>>> <[email-masked]>
>>>>>>>> wrote:
>>>>>>>>> Hi Tom,
>>>>>>>>>
>>>>>>>>>> synchronization needed between end points (synchronized 
>>>>>>>>>> clocks, provisions to correlate measurements correctly with 
>>>>>>>>>> lost packets, replicated packets, etc.). Also, what is 
>>>>>>>>>> envisioned for range for the period?
>>>>>>>>>
>>>>>>>>> Here is a reference
>>>>>>>>
>>>>>>>> Thanks for the pointer. Regarding the need for synchronized 
>>>>>>>>clocks  to measure delay, I consulted our local NTP expert. The 
>>>>>>>>host clock  jitter we currently see in our network is currently 
>>>>>>>>usually  greater than one-way packet delay (in some cases much 
>>>>>>>>greater), so  in his words:
>>>>>> <Jon> Thank you. As someone who has managed NTP more times and 
>>>>>>for  more years than I care to admit, this is a very good 
>>>>>>datapoint to consider.
>>>>> +++DK: As per our experience in carrier Ethernet we supported one 
>>>>> +++way
>>>>> IEEE 1588v2 is very precise if phy based timestamping is used. 
>>>>>Even  timestamping at NP level provided great results for one way delay.
>>>>>
>>>>> total on receiver of frame (this is to avoid processing time 
>>>>> that's taken for reply by software as hardware can put timestamp 
>>>>> at lower layer without doing delay and jitter calculation).
>>>>>and can be run on all flows of ECMP.
>>>>>
>>>>> Thanks,
>>>>> Deepak
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>> I think if we were implementing delay measurement in GUE, I 
>>>>>>>> would advocate add a 64 bit optional field for timestamp, 
>>>>>>>> probably containing source time stamp, and echoed timestamp for 
>>>>>>>> a flow (usec resolution and similar in design TCP timestamp 
>>>>>>>> option). This easily gives a precise RTT, and if clocks are 
>>>>>>> [Vero] If the source timestamp could be carried, it could also 
>>>>>>> be used for packet loss calculation/correlation.
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Tom
>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> Mach
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Tom
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> Is that necessary? Can they share the same time quantum 
>>>>>>>>>>>>>(as  well as other metrics maybe to be added later)? In all 
>>>>>>>>>>>>>the  protocols mentioned, the reserved bits are a somewhat 
>>>>>>>>>>>>>precious resource.
>>>>>>>>>>>>
>>>>>>>>>>>> Given one bit is used for both loss and delay measurement, 
>>>>>>>>>>>> for loss measurement, it periodically set and clear the 
>>>>>>>>>>>> marking bit, a flow is divided into consecutive blocks, and 
>>>>>>>>>>>> then the counting and calculating are based on each block. 
>>>>>>>>>>>> This is fine for loss measurement.
>>>>>>>>>>>>
>>>>>>>>>>>> Since there is ECMP, the first packet of a block at the 
>>>>>>>>>>>>sender  may probably different from the first packet at the 
>>>>>>>>>>>>receiver,  thus it will get the mismatched timestamps to 
>>>>>>>>>>>>calculate the delay.
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> Mach
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tom
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Mach
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> field, rather than in option field. Because chipset 
>>>>>>>>>>>>>> normally can't process optional field, it is hard to 
>>>>>>>>>>>>>> 主题: Re: [nvo3] Comments on NVO3 data plane requirements 
>>>>>>>>>>>>>> for
>>>>>>>> OAM
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Weiguo,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> What do you envision this marking looking like?  e.g. is 
>>>>>>>>>>>>>> it just a single flag bit, or large field with a counter 
>>>>>>>>>>>>>> or sequence number, or some kind of flow ID?  If not a 
>>>>>>>>>>>>>> single flag, how large do you see the field
>>>>>>>>>>>>> being?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If it is more than a flag (and I assume it would be), and 
>>>>>>>>>>>>>> is not mandatory for all implementations, then it seems 
>>>>>>>>>>>>>> to fall into the category of optional extensions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks, Larry
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> affect forwarding of packets, the marking field is only 
>>>>>>>>>>>>>> used for performance measurement. The
>>>>>>>>>>>>>> NVO3 packet with this marking flag don't need to be sent 
>>>>>>>>>>>>>> to control plane, it is different from OAM(ping/Trace) 
>>>>>>>>>>>>>> packet processing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> weiguo
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 主题: Re: [nvo3] Comments on NVO3 data plane requirements 
>>>>>>>>>>>>>> for
>>>>>>>> OAM
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Weiguo,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> marking groups of packets that belong to the particular 
>>>>>>>>>>>>>> flow to facilitate measurement of some performance 
>>>>>>>>>>>>>> metric, whether loss or delay/delay variation, may be 
>>>>>>>>>>>>>> viewed as one of passive performance
>>>>>>>>>>>>> measurement methods.
>>>>>>>>>>>>>> Because of that, I believe, OAM flag should not be used 
>>>>>>>>>>>>>> for marking as that will force punting marked packets 
>>>>>>>>>>>>>> from fast forwarding path to the control plane. But it 
>>>>>>>>>>>>>> might be good to have a field in NVO3 header that may be 
>>>>>>>>>>>>>> I maybe not clearly said in today’s NVO3 meeting, pls 
>>>>>>>>>>>>>>allow  me to reiterate the OAM data plane requirements on 
>>>>>>>>>>>>>>the mail list.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This kind of OAM application is initiated by operators 
>>>>>>>>>>>>>>for  network connectivity verification, normally when 
>>>>>>>>>>>>>>network failure occurs.
>>>>>>>>>>>>>> NVO3 encapsulation, and gets packet statistics with the 
>>>>>>>>>>>>>>real  time OAM flag marking. By comparing the packet 
>>>>>>>>>>>>>>number of  ingress NVE and the sum of all egress NVEs, 
>>>>>>>>>>>>>>packet loss can be deduced.
>>>>>>>>>>>>>> This method can be applicable for both unicast and 
>>>>>>>>>>>>>>multicast  traffic. Local policy on ingress NVE is 
>>>>>>>>>>>>>>configured by  operators or automatically acquired from

---

From: Mach Chen <[email-masked]>
Date: Wed, 19 Nov 2014 02:16:45 +0000

Dear Greg, Tissa and all,

Seems we are mixing in-band and out-of-band OAM here. IMHO, both in-band and out-of-band are needed and useful.

Most of the previous discussions (e.g., marking bit) of this thread are about in-band OAM, more specific it is about in-band PM (or passive PM).  I agree with Greg’s point here, a clear requirement would help to define the solution.

Best regards,
Mach

From: nvo3 [mailto:[email-masked]] On Behalf Of Greg Mirsky
Sent: Wednesday, November 19, 2014 8:19 AM
To: Tissa Senevirathne (tsenevir)
Cc: Tapraj Singh; Deepak Kumar (dekumar); [email-masked]
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hi Tissa,
I think you're misunderstanding or misinterpreting my POV. I'm not saying that either FM or PM cannot be performed in IP ECMP environment. But I believe that IP OAM has certain limitations like in case of in-band requirement. Of course, if one uses tunnels in server layer and maps flows into tunnels at the edge, then in-band comes for free. Another example that comes to mind is use of MPLS Entropy label. But I think that such are not the most generic scenarios for IP network.
Regards,
Greg

On Tue, Nov 18, 2014 at 3:54 PM, Tissa Senevirathne (tsenevir) <[email-masked]<mailto:[email-masked]>> wrote:
Greg

From: nvo3 [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Greg Mirsky
Sent: Tuesday, November 18, 2014 3:03 PM
To: Tapraj Singh
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hi Tapraj,

LAYER2 OAM

Thanks
Tapraj

---

From: Haoweiguo <[email-masked]>
Date: Wed, 19 Nov 2014 02:42:51 +0000

Hi Eric,

________________________________________
From: Erik Nordmark [[email-masked]]
Sent: Tuesday, November 18, 2014 2:40
To: Haoweiguo; Larry Kreeger (kreeger); Greg Mirsky
Cc: [email-masked]
Subject: Re: [nvo3] 答复:  答复:  Comments on NVO3 data plane requirements for OAM

[weiguo]: Right.

[weiguo]: Sorry, after i discussed with some experts, the 2 bits can be in outer IP header or NVO3 header, both methods can discriminate the flow of different tenants. But outer IP header has no extra space for these two bits, ECN bits had better not be multiplexed. So the two marking bits had better be included in NVO3 header.
 
    Erik

---

From: Haoweiguo <[email-masked]>
Date: Wed, 19 Nov 2014 03:50:11 +0000

Hi  Tissa,
Your solution is active OAM,  i think it is a basic and important solution in whole OAM framework.The disccussed thread is about passive OAM.
 Both active and passive OAM have its pros/cons, both have its usecases and scenarios.The regular method for passive OAM is to add marking bits in packet header, in NVO3 case, the marking bits had better be set in NVO3 header.But just as Greg said,currently it's unfortunate that there is no accepted OAM requirements, gap analysis, and etc in the WG. We hope this work could be progressed more quickly.
Thanks
weiguo
________________________________________
From: Tissa Senevirathne (tsenevir) [[email-masked]]
Sent: Wednesday, November 19, 2014 8:25
To: Tom Herbert
Cc: Greg Mirsky; Tapraj Singh; Deepak Kumar (dekumar); [email-masked]
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hi Tom

-----Original Message-----
From: Tom Herbert [mailto:[email-masked]]
Sent: Tuesday, November 18, 2014 4:20 PM
To: Tissa Senevirathne (tsenevir)
Cc: Greg Mirsky; Tapraj Singh; [email-masked]; Deepak Kumar (dekumar)
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Tissa,

Thanks,
Tom

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Wed, 19 Nov 2014 12:45:29 +0000

Hi Weiguo, Mach et,al

The discussion here is NVO3 data plane requirements for OAM. Like I have said earlier,  we do not need to complicate the Data Plane. Can you explain to me what difference it make to the data plane whether it is active/passive or some other means of OAM.

All what it needs to know is that the packet is an OAM packet and it is addressed to the local device,

-----Original Message-----
From: Haoweiguo [mailto:[email-masked]] 
Sent: Tuesday, November 18, 2014 7:50 PM
To: Tissa Senevirathne (tsenevir); Tom Herbert
Cc: Greg Mirsky; Tapraj Singh; Deepak Kumar (dekumar); [email-masked]
Subject: RE: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Hi  Tissa,

Hi Tom

-----Original Message-----
From: Tom Herbert [mailto:[email-masked]]
Sent: Tuesday, November 18, 2014 4:20 PM
To: Tissa Senevirathne (tsenevir)
Cc: Greg Mirsky; Tapraj Singh; [email-masked]; Deepak Kumar (dekumar)
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Tissa,

Thanks,
Tom

---

From: Mach Chen <[email-masked]>
Date: Thu, 20 Nov 2014 01:54:30 +0000

Hi Tissa,

Thanks for your response!

Please see my response inline...

> From: nvo3 [mailto:[email-masked]] On Behalf Of Tissa Senevirathne
> (tsenevir)
> Sent: Wednesday, November 19, 2014 8:45 PM
> To: Haoweiguo; Tom Herbert
> The discussion here is NVO3 data plane requirements for OAM. Like I have said

You are right, this discussion is about "NVO3 data plane requirements for OAM", but recently the focus is Performance Measurement (PM) requirement to NVO3 that is also one of the OAM functions.

> earlier,  we do not need to complicate the Data Plane. Can you explain to me

"Complicate/simple" is not the goal, the goal is to define a reasonable solution that can satisfy the requirement. That's why I agree with Greg that we should firstly make the agreement on the requirement.

> what difference it make to the data plane whether it is active/passive or some
> other means of OAM.

Active/passive is mainly regarding to PM which normally includes Active and Passive PM. 

Active PM measures the injected packets (e.g., OAM packets) to evaluate the performance of a path. Passive PM measures the performance of the real/live traffic of a path, it reflects the real performance of the path. For more detail about active/passive PM, you may refer to the material of IPPM WG. 

> All what it needs to know is that the packet is an OAM packet and it is addressed

What you are talking are just part of the OAM functions (e.g., CC, CV), for passive PM, OAM packets may not be needed.

Best regards,
Mach

> Your solution is active OAM,  i think it is a basic and important solution in whole
> OAM framework.The disccussed thread is about passive OAM.
>  Both active and passive OAM have its pros/cons, both have its usecases and
> scenarios.The regular method for passive OAM is to add marking bits in packet
> header, in NVO3 case, the marking bits had better be set in NVO3 header.But just
> as Greg said,currently it's unfortunate that there is no accepted OAM
> requirements, gap analysis, and etc in the WG. We hope this work could be
> progressed more quickly.
> Thanks
> weiguo
> As I noted earlier on in the thread, OAM processing would not kick in unless
> address matches the MEP/MIP. If address match MEP/MIP and OAM bit is set,
> then OAM processing begins.
> 
> If I am reading this correctly, the OAM message would be composed of the
> encapsulation header, followed by 128 bytes of which contains a pseudo header
> for switching, followed by a self defining OAM message.
> The OAM bit is only used at the receiver to distinguish data messages for OAM
> messages for processing. Is this interpretation correct?
> 
> Thanks,
> Tom
> 
> >
> >
> >
> >

---

From: Tom Herbert <[email-masked]>
Date: Thu, 20 Nov 2014 09:27:39 -0800

> "Complicate/simple" is not the goal, the goal is to define a reasonable solution that can satisfy the requirement. That's why I agree with Greg that we should firstly make the agreement on the requ…
Mach,

The nvo3 OAM requirements draft
(draft-ashwood-nvo3-oam-requirements-01) seems to already contain a
fairly comprehensive list of requirements. Particularly apropos to
this discussion are:

   R13) NVO3 OAM frames MUST be forwarded along the same path (i.e.,
   links (including LAG members) and nodes) as the NVO3 data frames.

   R16) NVO3 OAM should be extensible such that new functionality and
   information elements related to this functionality can be introduced
   in the future.

I believe that an an active OAM message format like Tissa describes
would meet these and most of the others in that draft.

If a passive mechanism is indeed required, then we need to consider
how to meet the extensibility requirement. I don't believe that
allocating two bit flags in the encapsulation header is at all an
extensible solution. The reserved header bits are too a precious
resource to be allocated for such a narrow purpose and for something
not required for protocol operation. As previously discussed in this
thread, using one bit to get one-way time delay measurements is not
even viable in a lot deployments-- in this case we probably need
timestamps to get RTT.

To support passive OAM support in GUE, I would probably propose to add
a generic optional "measurement" field. This would provide some number
of bits in the header that can be used for passive measurement
(possibly a few different sizes say 32, 64, 128 bits). The field can
be structured to allow different mechanisms (e.g. include timestamps
for RTT measurement). This also reduces the constraints on the
measurement techniques, for instance the marking technique might no
longer limited to use a single bit which should reduce the complexity
needed to deal with OOO or packet loss.

Tom

---

From: Haoweiguo <[email-masked]>
Date: Fri, 21 Nov 2014 01:39:19 +0000

Hi Tom,

________________________________________
From: Tom Herbert [[email-masked]]
Sent: Friday, November 21, 2014 1:27
To: Mach Chen
Cc: [email-masked]
Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

Mach,

[weiguo]: Yes, Tissa's solution can meet active OAM requirements.

If a passive mechanism is indeed required, then we need to consider
how to meet the extensibility requirement. I don't believe that
allocating two bit flags in the encapsulation header is at all an
extensible solution. The reserved header bits are too a precious
resource to be allocated for such a narrow purpose and for something
not required for protocol operation. 
[weiguo]: Although this purpose is narrow, but i think it is a important requirements, it's worthy to add two bits in the fixed field(or other applicable methods). IP header has no enough space for marking, so unfortunately IP traffic can't do performance measurements in real time. NVO3 is a new encapsulation, we should levarage this oppotunity to enhance functionality in contrast with IP header.
As previously discussed in this
thread, using one bit to get one-way time delay measurements is not
even viable in a lot deployments-- in this case we probably need
timestamps to get RTT.
[weiguo]: Performance measurements items include: Packet loss, one-way packet latency, two-way packet latency. One-way packet latency is a difficult problem because time syncronization accuracy issue. Both using marking bit method and carrying time stamp method need time syncronization, so the one-way packet latency is hard to measure accurately. But for the remaining two items, using marking bits method is OK, and there already some commertial deployment(IP header marking using ECN bits for IP traffic performance measurements) usecase in industry.

[weiguo]: I think the performance measurements should be hardware friendly, because most chipsets can't process option field, using option method will induce the hardware inability for the performance measurements. So i think using option field can't meet the passive OAM requirements.

Tom

>> >>>>>>>>>>>>>> ?件人: Larry Kreeger (kreeger) [[email-masked]]
>> >>>>>>>>>>>>>> ?送??: 2014年11月12日 4:33
>> >>>>>>>>>>>>>> 主?: Re: [nvo3] Comments on NVO3 data plane
>> requirements
>> >>>>>>>>>>>>>> for
>> >>>>>>>> OAM
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Hi Weiguo,
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> ?件人: Greg Mirsky [[email-masked]]
>> >>>>>>>>>>>>>> ?送??: 2014年11月12日 4:07
>> >>>>>>>>>>>>>> 收件人: Haoweiguo
>> >>>>>>>>>>>>>> 抄送: [email-masked]
>> >>>>>>>>>>>>>> 主?: Re: [nvo3] Comments on NVO3 data plane
>> requirements
>> >>>>>>>>>>>>>> for
>> >>>>>>>> OAM
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Hi Weiguo,
>> >>>>>>>>>>>>>>

---

From: Mach Chen <[email-masked]>
Date: Fri, 21 Nov 2014 03:02:52 +0000

Hi Tom,

> Sent: Friday, November 21, 2014 1:28 AM
> To: Mach Chen
> Cc: [email-masked]
> > You are right, this discussion is about "NVO3 data plane requirements for OAM",
> but recently the focus is Performance Measurement (PM) requirement to NVO3
> that is also one of the OAM functions.
> >
> > "Complicate/simple" is not the goal, the goal is to define a reasonable solution
> that can satisfy the requirement. That's why I agree with Greg that we should
> firstly make the agreement on the requirement.
> >
> Mach,
> 
> (draft-ashwood-nvo3-oam-requirements-01) seems to already contain a fairly
> comprehensive list of requirements. Particularly apropos to this discussion are:
> 
> I believe that an an active OAM message format like Tissa describes would meet
> these and most of the others in that draft.

There are also the following requirements listed in the draft:

   R7) NVO3 OAM MUST support measurement of per VNI frame loss between
   two NV Edge devices that support the same VNI within a given NVO3
   domain.

   R8) NVO3 OAM MUST support measurement of per VNI two-way frame delay
   between two NV edge devices that support the same VNI within a given
   NVO3 domain.

   R9) NVO3 OAM MUST support measurement of per VNI one-way frame delay
   between two NV Edge devices that support the same VNI within a given
   NVO3 domain.

   R10) NVO3 OAM MUST support measurement of per VNI frame delay
   variation between two NV Edge devices that support the same VNI
   within a given NVO3 domain.

> 
> If a passive mechanism is indeed required,

I personally think it is indeed required, and we also received such requirements from the operators.

> then we need to consider how to
> meet the extensibility requirement. I don't believe that allocating two bit flags in
> the encapsulation header is at all an extensible solution. The reserved header
> bits are too a precious resource to be allocated for such a narrow purpose and for

Looking through the bits in some headers, we could find that every bit in a header has its own purpose. It's better that one bit could be defined for as many usages as possible, but there is always tradeoff. 

As for the two bits for passive PM that include loss, one/two way delay, delay variation and throughput, I may not think this is a narrow purpose. And if you want, you may use the marking bit for some policies control. 

> something not required for protocol operation. As previously discussed in this
> thread, using one bit to get one-way time delay measurements is not even viable
> in a lot deployments-- in this case we probably need timestamps to get RTT.

For the deployments in question, the challenge for one way time delay is time synchronization and its accuracy. The solution of using one-bit for one-way delay is really workable, there have been some prototypes and experiments show that.

Since time synchronization is not needed for RTT, IMHO, measure RTT should be the easiest way to go. 

> 
> To support passive OAM support in GUE, I would probably propose to add a
> generic optional "measurement" field. This would provide some number of bits
> in the header that can be used for passive measurement (possibly a few
> different sizes say 32, 64, 128 bits). The field can be structured to allow different
> mechanisms (e.g. include timestamps for RTT measurement). This also reduces
> the constraints on the measurement techniques, for instance the marking
> technique might no longer limited to use a single bit which should reduce the
> complexity needed to deal with OOO or packet loss.

Even with the solution as above, seems there needs at least one bit(at the fix position of the header) that indicates there is an optional field exist. In the case we could have opportunity and "enough" reserved bits to allocate for the marking bits, I'd like to suggest allocating two bits for passive PM. 

In addition, I see the value of you proposed optional "measurement" field, it could be used to carry some correlation (e.g., block/period number) and timestamp information, then combine with the marking bit, it can greatly simplify the marking based solution.

Thanks,
Mach

> 
> Tom
> 
> >
> >> what difference it make to the data plane whether it is
> >> active/passive or some other means of OAM.
> >
> > Active PM measures the injected packets (e.g., OAM packets) to evaluate the
> performance of a path. Passive PM measures the performance of the real/live
> traffic of a path, it reflects the real performance of the path. For more detail
> about active/passive PM, you may refer to the material of IPPM WG.
> >
> >>
> > What you are talking are just part of the OAM functions (e.g., CC, CV), for
> passive PM, OAM packets may not be needed.
> >
> >
> > Best regards,
> > Mach
> >
> >>
> >> Your solution is active OAM,  i think it is a basic and important
> >> solution in whole OAM framework.The disccussed thread is about passive
> OAM.
> >>  Both active and passive OAM have its pros/cons, both have its
> >> usecases and scenarios.The regular method for passive OAM is to add
> >> marking bits in packet header, in NVO3 case, the marking bits had
> >> better be set in NVO3 header.But just as Greg said,currently it's
> >> unfortunate that there is no accepted OAM requirements, gap analysis,
> >> and etc in the WG. We hope this work could be progressed more quickly.
> >> Thanks
> >> weiguo
> >> As I noted earlier on in the thread, OAM processing would not kick in
> >> unless address matches the MEP/MIP. If address match MEP/MIP and OAM
> >> bit is set, then OAM processing begins.
> >>
> >> > absence of WG agreed upon OAM Requirements; no gap analysis of
> >> > tools for NVO3 OAM; OAM flag does not help passive performance
> >> > measurement marking method (two bit-long field for marking in fixed
> position).
> >> >
> >> > why in-band requirement for IP OAM, both FM and Active PM, is not
> >> > attainable
> >> (ECMP environment).
> >> >
> >> > Regards,
> >> >
> >> > Greg
> >> >
> >> >
> >> >
> >> > Our OAM should follow the data path for services as much as
> >> > possible and all other protocol specific information should be in
> >> > the OAM protocol specific TLVs.
> >> >
> >> > LAYER2 OAM
> >> >
> >> > L2 OAM Should be the MAC address and send level of hierarchy should
> >> > be the ether type or OUI.
> >> >>I Agree with Tissa below. My Goal also was to point out that
> >> >>instead of complicating the header, we can do OAM performance
> >> >>within OAM channel itself and this is extensible and can be done in
> >> >>hardware which is why mostly things are added in header.
> >> >>
> >> >>>called flow entropy) and at the end of that you can have all of
> >> >>>the stuff you need in the world of OAM.
> >> >>>
> >> >>>Hope this clarify
> >> >>>
> >> >>>Thanks
> >> >>>Tissa
> >> >>>On Mon, Nov 17, 2014 at 12:01 AM, Marc Binderberger
> >> >>><[email-masked]>
> >> >>>wrote:
> >> >>>> Hello Deepak et al.,
> >> >>>>
> >> >>>>the  information (timestamps, oam Subtype). Sounds definitely
> >> >>>>more than a  32/64bit header could carry (*).
> >> >>>>
> >> >>>We should be able to define the requirements so that an OAM
> >> >>>message corresponding to a flow which would be routed in exactly
> >> >>>the same way as a data message for the flow. Larry mentioned that
> >> >>>we might even want to put a "fake" packet header as the first part
> >> >>>of the encapsulated payload of an OAM message for instance.
> >> >>>
> >> >>>> Now if we introduce such an OAM extension header it could as
> >> >>>> well carry the "first" bit we discussed for packet loss measurement (?).
> >> >>>>
> >> >>>>
> >> >>>> Regards, Marc
> >> >>>>
> >> >>>>>>> On Nov 13, 2014, at 11:47 PM, Vero Zheng
> >> >>>>>>><[email-masked]>
> >> >>>>>>>wrote:
> >> >>>>>>>
> >> >>>>>>> Hi Tom,
> >> >>>>>>>
> >> >>>>>>> Please see in-line.
> >> >>>>>>>
> >> >>>>>>> BR, Vero
> >> >>>>>>>
> >> >>>>>>>>>> Is there there a draft or description of how this
> >> >>>>>>>>>> algorithm would work? Seems like there would need to be
> >> >>>>>>>>>> quite a bot of synchronization needed between end points
> >> >>>>>>>>>> (synchronized clocks, provisions to correlate measurements
> >> >>>>>>>>>> correctly with lost packets, replicated packets, etc.).
> >> >>>>>>>>>> Also, what is envisioned for range for the period?
> >> >>>>>>>>>
> >> >>>>>>>>> Here is a reference
> >> >>>>>>>>
> >> >>>>>>>> https://datatracker.ietf.org/doc/draft-chen-ippm-coloring-ba
> >> >>>>>>>> sed
> >> >>>>>>>> -ip
> >> >>>>>>>> fpm-fr
> >> >>>>>>>> amew
> >> >>>>>>>> ork/.
> >> >>>>>>>>
> >> >>>>>>>>clocks  to measure delay, I consulted our local NTP expert.
> >> >>>>>>>>The host clock  jitter we currently see in our network is
> >> >>>>>>>>currently usually  greater than one-way packet delay (in some
> >> >>>>> +++DK: As per our experience in carrier Ethernet we supported
> >> >>>>> +++one way
> >> >>>>> delay and never found NTP useful even for our lab networks (I
> >> >>>>>am referring software based NTP NTPv3).
> >> >>>>> If we want to accurately measure two way delay we need 4
> >> >>>>> timestamp total on receiver of frame (this is to avoid
> >> >>>>> processing time that's taken for reply by software as hardware
> >> >>>>> can put timestamp at lower layer without doing delay and jitter
> calculation).
> >> >>>>> bit can be overloaded to do both Fault and performance if OAM
> >> >>>>>is defined with different oam Subtype for Delay and Loss frames
> >> >>>>>and it will not be too deep hardware inspection) as that give
> >> >>>>>large flexibility (synthetic/real loss measurement,
> >> >>>>>Availability/unavailability, on-demand and pro-active
> >> >>>>>performance) and can be run on all flows of ECMP.
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>> Deepak
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>>
> >> >>>>>>> [Vero] Thanks for this. What about the current experience
> >> >>>>>>> with
> >> >>>>>>> 1588v2 then?
> >> >>>>>>>>
> >> >>>>>>>> probably containing source time stamp, and echoed timestamp
> >> >>>>>>>> for a flow (usec resolution and similar in design TCP
> >> >>>>>>>> timestamp option). This easily gives a precise RTT, and if
> >> >>>>>>>> clocks are precisely synchronized then one way latency could be
> calculated also.
> >> >>>>>>> [Vero] If the source timestamp could be carried, it could
> >> >>>>>>> also be used for packet loss calculation/correlation.
> >> >>>>>>>
> >> >>>>>>>> Thanks,
> >> >>>>>>>> Tom
> >> >>>>>>>>
> >> >>>>>>>>> Best regards,
> >> >>>>>>>>> Mach
> >> >>>>>>>>>>
> >> >>>>>>>>>> Thanks,
> >> >>>>>>>>>> Tom
> >> >>>>>>>>>>
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> One question I still have is: why is the measurement done
> >> >>>>>>>>>>> in the NVE
> >> >>>>>>>> header?
> >> >>>>>>>>>>>>> Cc: Greg Mirsky; Haoweiguo; [email-masked]; Larry
> >> >>>>>>>>>>>>> Kreeger
> >> >>>>>>>>>>>>> (kreeger)
> >> >>>>>>>>>>>>>> Single bit is not sufficient if someone wants to
> >> >>>>>>>>>>>>>>perform loss and delay measurement  simultaneously,
> >> >>>>>>>>>>>>>>then two bits needed.
> >> >>>>>>>>>>>>>(as  well as other metrics maybe to be added later)? In
> >> >>>>>>>>>>>>>all the  protocols mentioned, the reserved bits are a
> >> >>>>>>>>>>>>>somewhat precious resource.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Given one bit is used for both loss and delay
> >> >>>>>>>>>>>> measurement, for loss measurement, it periodically set
> >> >>>>>>>>>>>> and clear the marking bit, a flow is divided into
> >> >>>>>>>>>>>> consecutive blocks, and then the counting and calculating are
> based on each block.
> >> >>>>>>>>>>>> For delay measurement, it has to make sure the
> >> >>>>>>>>>>>> timestamps (collected at sender and receiver) are for the
> same packet.
> >> >>>>>>>>>>>>sender  may probably different from the first packet at
> >> >>>>>>>>>>>>the receiver,  thus it will get the mismatched timestamps
> >> >>>>>>>>>>>>to calculate the delay.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Best regards,
> >> >>>>>>>>>>>> Mach
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Tom
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Best regards,
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Mach
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> NVO3 header is precious. I would like it is set in
> >> >>>>>>>>>>>>>> fixed field, rather than in option field. Because
> >> >>>>>>>>>>>>>> chipset normally can't process optional field, it is
> >> >>>>>>>>>>>>>> hard to realize in-band performance measurement if
> >> >>>>>>>>>>>>>> using optional
> >> >>>>>>>>>>>>> field for marking.
> >> >>>>>>>>>>>>>> What do you envision this marking looking like?  e.g.
> >> >>>>>>>>>>>>>> is it just a single flag bit, or large field with a
> >> >>>>>>>>>>>>>> counter or sequence number, or some kind of flow ID?
> >> >>>>>>>>>>>>>> If not a single flag, how large do you see the field
> >> >>>>>>>>>>>>> being?
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> If it is more than a flag (and I assume it would be),
> >> >>>>>>>>>>>>>> and is not mandatory for all implementations, then it
> >> >>>>>>>>>>>>>> seems to fall into the category of optional extensions.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Thanks, Larry
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> NVO3 data encapsulation has a field for marking and
> >> >>>>>>>>>>>>>> not affect forwarding of packets, the marking field is
> >> >>>>>>>>>>>>>> only used for performance measurement. The
> >> >>>>>>>>>>>>>> NVO3 packet with this marking flag don't need to be
> >> >>>>>>>>>>>>>> sent to control plane, it is different from
> >> >>>>>>>>>>>>>> OAM(ping/Trace) packet processing.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Thanks
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> weiguo
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> marking groups of packets that belong to the
> >> >>>>>>>>>>>>>> particular flow to facilitate measurement of some
> >> >>>>>>>>>>>>>> performance metric, whether loss or delay/delay
> >> >>>>>>>>>>>>>> Because of that, I believe, OAM flag should not be
> >> >>>>>>>>>>>>>> used for marking as that will force punting marked
> >> >>>>>>>>>>>>>> packets from fast forwarding path to the control
> >> >>>>>>>>>>>>>> plane. But it might be good to have a field in NVO3
> >> >>>>>>>>>>>>>>allow  me to reiterate the OAM data plane requirements
> >> >>>>>>>>>>>>>>on the mail list.
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> Currently NVO3 data plane encapsulation only includes
> >> >>>>>>>>>>>>>>one OAM flag, it is used for Ping/Trace similar applications.
> >> >>>>>>>>>>>>>> When ingress NVE receives traffic from local TS, it
> >> >>>>>>>>>>>>>>gets packet statistics, and mark(coloring) the OAM flag
> >> >>>>>>>>>>>>>>relying on local policy when it performs
> >> >>>>>>>>>>>>>> NVO3 encapsulation, and gets packet statistics with
> >> >>>>>>>>>>>>>>the real  time OAM flag marking. By comparing the
> >> >>>>>>>>>>>>>>packet number of  ingress NVE and the sum of all egress
> >> >>>>>>>>>>>>>>NVEs, packet loss can be deduced.

---

From: "Deepak Kumar (dekumar)" <[email-masked]>
Date: Fri, 21 Nov 2014 06:34:22 +0000

Hi Mach,

Please see inline +++DK:

On 11/20/14 5:02 PM, "Mach Chen" <[email-masked]> wrote:

>>fairly
>>would meet
>Looking through the bits in some headers, we could find that every bit in
>a header has its own purpose. It's better that one bit could be defined
>for as many usages as possible, but there is always tradeoff.
>
>As for the two bits for passive PM that include loss, one/two way delay,
>delay variation and throughput, I may not think this is a narrow purpose.
>And if you want, you may use the marking bit for some policies control.
>
>>even viable
>For the deployments in question, the challenge for one way time delay is
>time synchronization and its accuracy. The solution of using one-bit for
>one-way delay is really workable, there have been some prototypes and
>experiments show that.
>
>Even with the solution as above, seems there needs at least one bit(at
>the fix position of the header) that indicates there is an optional field
>exist. In the case we could have opportunity and "enough" reserved bits
>to allocate for the marking bits, I'd like to suggest allocating two bits
>for passive PM. 
>
>In addition, I see the value of you proposed optional "measurement"
>field, it could be used to carry some correlation (e.g., block/period
>number) and timestamp information, then combine with the marking bit, it
>can greatly simplify the marking based solution.

+++DK:
I think adding information regarding measurement field, block, period,
etc. is not required in data path as more information reduces the mtu, and
this can easily be added if required by TLV to OAM functionality with new
subtype (as this is control or configuration functionality).
Also even passive oam loss measurement solution to calculating loss is not
accurate as packets can arrive late outside the measuring blocks. Even in
that case to get accurate measurement instead of ipfix method, better to
use OAM to exchange these marked packet counters on both ends and do loss
measurement between two consecutive loss measurement replies.

For loss measurement, why we have to count traffic for marked packets only
and not maintain counters per flow?

Thanks,
Deepak
>> > Active/passive is mainly regarding to PM which normally includes
>>Active and
>> Passive PM.
>> >
>> >> >>>>>Even  timestamping at NP level provided great results for one
>>way
>> delay.
>> >> >>>>>

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 20 Nov 2014 22:50:31 -0800

Hi Deepak, et. al,
the problem is with TLVs is that, as Mach noted, these are not HW-friendly.
And timestamping, that is necessary for Delay Measurement, whether by
active or passive measurement method, should be done consistently.
Processing packet with variable format cannot ensure that while checking
fixed field do. Hence the suggestion to have marking field in fixed
position.
Again, I believe that even though discussion of a solution is interesting,
we need agree on requirements first. The mentioned earlier draft is very
start and, in my opinion, can be WG document as-is. No one needs overloaded
and over-engineered format but ending up with one that cannot support
essential functionality is equally bad result of our efforts.

Regards,
Greg

On Thu, Nov 20, 2014 at 10:34 PM, Deepak Kumar (dekumar) <[email-masked]>
wrote:

> >> >> >>>>>>>>>>>>>> From: nvo3 [mailto:[email-masked]] On Behalf
> Of
> >> >> >>>>>>>>>>>>>> Greg Mirsky

---

From: Mach Chen <[email-masked]>
Date: Fri, 21 Nov 2014 08:37:21 +0000

Hi Deepak,

> >number) and timestamp information, then combine with the marking bit,
> >it can greatly simplify the marking based solution.
> 
> +++DK:
> I think adding information regarding measurement field, block, period, etc. is not
> required in data path as more information reduces the mtu, and this can easily be
> added if required by TLV to OAM functionality with new subtype (as this is control
> or configuration functionality).

I am talking about two things here:

1) the fixed marking bits, I think they are necessary for passive PM;

2) the correlation information, timestamps, counters, they could be communicated either through in-band or out-or-band, each way has its pros and cons;

> accurate as packets can arrive late outside the measuring blocks. Even in that

If it only depends on the marking bit and the measuring period is set to a very small interval, indeed, that will affect the accuracy. But from engineering point view, an operator and an implementation will not set (or support) to a very aggressive period. And if the packets can carry some correlation information (e.g., the block/period number), then the accuracy should be no problem. 

In theory, you are right, if the delay of the packets of block exceed a threshold (e.g., a block period), the packets may be mis-counted into another block. 

> case to get accurate measurement instead of ipfix method, better to use OAM to
> exchange these marked packet counters on both ends and do loss measurement
> between two consecutive loss measurement replies.

I am fine with either way for communicating the counters and timestamps. 

> 
> For loss measurement, why we have to count traffic for marked packets only and
> not maintain counters per flow?

I'm not sure what's your question here. 

To calculate the packet loss, counters maintenance (no matter at where) is necessary, it depends on the specific implementation. 

Best regards,
Mach

> From: Deepak Kumar (dekumar) [mailto:[email-masked]]
> Sent: Friday, November 21, 2014 2:34 PM
> To: Mach Chen; Tom Herbert
> Cc: [email-masked]
> >> > You are right, this discussion is about "NVO3 data plane
> >> > requirements
> >>for OAM",
> >> that can satisfy the requirement. That's why I agree with Greg that
> >>we should  firstly make the agreement on the requirement.
> >> >
> >> Mach,
> >>
> >> meet the extensibility requirement. I don't believe that allocating
> >>two bit flags in  the encapsulation header is at all an extensible
> >>solution. The reserved header  bits are too a precious resource to be
> >>allocated for such a narrow purpose and for
> >
> >Looking through the bits in some headers, we could find that every bit
> >in a header has its own purpose. It's better that one bit could be
> >defined for as many usages as possible, but there is always tradeoff.
> >
> >As for the two bits for passive PM that include loss, one/two way
> >delay, delay variation and throughput, I may not think this is a narrow purpose.
> >> something not required for protocol operation. As previously
> >>discussed in this  thread, using one bit to get one-way time delay
> >>measurements is not even viable  in a lot deployments-- in this case
> >>we probably need timestamps to get RTT.
> >
> >For the deployments in question, the challenge for one way time delay
> >is time synchronization and its accuracy. The solution of using one-bit
> >for one-way delay is really workable, there have been some prototypes
> >and experiments show that.
> >
> >> To support passive OAM support in GUE, I would probably propose to
> >>add a  generic optional "measurement" field. This would provide some
> >>number of bits  in the header that can be used for passive measurement
> >the fix position of the header) that indicates there is an optional
> >field exist. In the case we could have opportunity and "enough"
> >reserved bits to allocate for the marking bits, I'd like to suggest
> >allocating two bits for passive PM.
> >
> >number) and timestamp information, then combine with the marking bit,
> >it can greatly simplify the marking based solution.
> 
> +++DK:
> I think adding information regarding measurement field, block, period, etc. is not
> required in data path as more information reduces the mtu, and this can easily be
> added if required by TLV to OAM functionality with new subtype (as this is control
> or configuration functionality).
> accurate as packets can arrive late outside the measuring blocks. Even in that
> case to get accurate measurement instead of ipfix method, better to use OAM to
> exchange these marked packet counters on both ends and do loss measurement
> between two consecutive loss measurement replies.
> 
> For loss measurement, why we have to count traffic for marked packets only and
> not maintain counters per flow?
> 
> Thanks,
> Deepak
> >
> >Thanks,
> >Mach
> >
> >>
> >> Tom
> >>
> >> >
> >> performance of a path. Passive PM measures the performance of the
> >>real/live  traffic of a path, it reflects the real performance of the
> >>path. For more detail  about active/passive PM, you may refer to the
> >>material of IPPM WG.
> >> >
> >> >>
> >> >> usecases and scenarios.The regular method for passive OAM is to
> >> >> add marking bits in packet header, in NVO3 case, the marking bits
> >> >> had better be set in NVO3 header.But just as Greg said,currently
> >> >> it's unfortunate that there is no accepted OAM requirements, gap
> >> >> analysis, and etc in the WG. We hope this work could be progressed
> >> >> more
> >>quickly.
> >> >> Thanks
> >> >> weiguo
> >> >> As I noted earlier on in the thread, OAM processing would not kick
> >> >> in unless address matches the MEP/MIP. If address match MEP/MIP
> >> >> and OAM bit is set, then OAM processing begins.
> >> >>
> >> >> If I am reading this correctly, the OAM message would be composed
> >> >> of the encapsulation header, followed by 128 bytes of which
> >> >> contains a pseudo header for switching, followed by a self defining OAM
> message.
> >> >> The OAM bit is only used at the receiver to distinguish data
> >> >> messages for OAM messages for processing. Is this interpretation correct?
> >> >>
> >> >> Thanks,
> >> >> Tom
> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > identification of OAM packet (though not necessarily OAM). But
> >> >> > IP clearly doesn't have such identification for OAM and that, in
> >> >> > part, why in-band requirement for IP OAM, both FM and Active PM,
> >> >> > is not attainable
> >> >> (ECMP environment).
> >> >> >
> >> >> > Regards,
> >> >> >
> >> >> > Greg
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Tue, Nov 18, 2014 at 1:31 PM, Tapraj Singh
> >> >> > <[email-masked]>
> >>wrote:
> >> >> >
> >> >> > Hi All,
> >> >> >
> >> >> > possible and all other protocol specific information should be
> >> >> > in the OAM protocol specific TLVs.
> >> >> >
> >> >> > LAYER2 OAM
> >> >> >
> >> >> > L2 OAM Should be the MAC address and send level of hierarchy
> >> >> > should be the ether type or OUI.
> >> >> > On 11/17/14 12:39 PM, "Deepak Kumar (dekumar)"
> >> >> > <[email-masked]>
> >> >> wrote:
> >> >> >
> >> >> >>within OAM channel itself and this is extensible and can be done
> >> >> >>in hardware which is why mostly things are added in header.
> >> >> >>
> >> >> >>signaling like TDM, Generate frames to verify qos etc.) and so
> >> >> >>OAM Channel solution will be extensible.
> >> >> >>
> >> >> >>Thanks,
> >> >> >>Deepak
> >> >> >>
> >> >> >>>As far as packet encapsulation is concern, all what is needed
> >> >> >>>is single bit. This bit is needed to prevent OAM packets
> >> >> >>>leaking out from the domain.
> >> >> >>>
> >> >> >>>As an example. You could have packet fragment (which is
> >> >> >>>sometimes called flow entropy) and at the end of that you can
> >> >> >>>have all of the stuff you need in the world of OAM.
> >> >> >>>
> >> >> >>>Hope this clarify
> >> >> >>>
> >> >> >>>Thanks
> >> >> >>>Tissa
> >> >> >>>Cc: Greg Mirsky; Mach Chen; Deepak Kumar (dekumar);
> >> >> >>>> Seems we need an optional header extension or a TLV to carry
> >> >> >>>>all the  information (timestamps, oam Subtype). Sounds
> >> >> >>>>definitely more than a  32/64bit header could carry (*).
> >> >> >>>>
> >> >> >>>> The optional header extension, when done similar to GUE, has
> >> >> >>>> a fixed position. For the TLV this would be an additional
> >> >> >>>message corresponding to a flow which would be routed in
> >> >> >>>exactly the same way as a data message for the flow. Larry
> >> >> >>>mentioned that we might even want to put a "fake" packet header
> >> >> >>>as the first part of the encapsulated payload of an OAM message for
> instance.
> >> >> >>>
> >> >> >>>>>>>>>>> so for delay measurement you use the color flag to
> >> >> >>>>>>>>>>> mark a single packet, which helps the receiver to pick
> >> >> >>>>>>>>>>> the right packet?  And repeat this every time period T ?
> >> >> >>>>>>>>>>>
> >> >> >>>>>>>>>> quite a bot of synchronization needed between end
> >> >> >>>>>>>>>> points (synchronized clocks, provisions to correlate
> >> >> >>>>>>>>>> measurements correctly with lost packets, replicated packets,
> etc.).
> >> >> >>>>>>>> Thanks for the pointer. Regarding the need for
> >> >> >>>>>>>>synchronized clocks  to measure delay, I consulted our local
> NTP expert.
> >> >> >>>>>>>>currently usually  greater than one-way packet delay (in
> >> >> >>>>>>>>some cases much greater), so  in his words:
> >> >> >>>>> +++DK: As per our experience in carrier Ethernet we
> >> >> >>>>> +++supported one way
> >> >> >>>>> delay and never found NTP useful even for our lab networks
> >> >> >>>>>(I am referring software based NTP NTPv3).
> >> >> >>>>> As mentioned below IEEE 1588v2 will vary based on equipment
> >> >> >>>>>and operator networks but in our testing we found it very
> >> >> >>>>>precise if properly deployed.
> >> >> >>>>> processing time that's taken for reply by software as
> >> >> >>>>> hardware can put timestamp at lower layer without doing
> >> >> >>>>> delay and jitter
> >> calculation).
> >> >> >>>>> For one way delay we will require 2 timestamp, so lower
> >> >> >>>>> layer hardware can timestamp before packet is punted to software.
> >> >> >>>>>
> >> >> >>>>> bit can be overloaded to do both Fault and performance if
> >> >> >>>>>OAM is defined with different oam Subtype for Delay and Loss
> >> >> >>>>>frames and it will not be too deep hardware inspection) as
> >> >> >>>>>that give large flexibility (synthetic/real loss measurement,
> >> >> >>>>>>>> factors, one is the implementation limitation, the other
> >> >> >>>>>>>> the requirement of the operators. In the above reference,
> >> >> >>>>>>>> the suggested periods are 1s, 10s, 1min, 10min and 1h.
> >> >> >>>>>>>> I think if we were implementing delay measurement in GUE,
> >> >> >>>>>>>> I would advocate add a 64 bit optional field for
> >> >> >>>>>>>> timestamp, probably containing source time stamp, and
> >> >> >>>>>>>> echoed timestamp for a flow (usec resolution and similar
> >> >> >>>>>>>> in design TCP timestamp option). This easily gives a
> >> >> >>>>>>>> precise RTT, and if clocks are precisely synchronized
> >> >> >>>>>>>> then one way latency
> >>could be
> >> calculated also.
> >> >> >>>>>>>>>>> One question I still have is: why is the measurement
> >> >> >>>>>>>>>>> done in the NVE
> >> >> >>>>>>>> header?
> >> >> >>>>>>>>>>>>> Is that necessary? Can they share the same time
> >> >> >>>>>>>>>>>>>quantum (as  well as other metrics maybe to be added
> >> >> >>>>>>>>>>>>>later)? In all the  protocols mentioned, the reserved
> >> >> >>>>>>>>>>>> measurement, for loss measurement, it periodically
> >> >> >>>>>>>>>>>> set and clear the marking bit, a flow is divided into
> >> >> >>>>>>>>>>>> Since there is ECMP, the first packet of a block at
> >> >> >>>>>>>>>>>>the sender  may probably different from the first
> >> >> >>>>>>>>>>>>packet at the receiver,  thus it will get the
> >> >> >>>>>>>>>>>>mismatched timestamps to calculate the delay.
> >> >> >>>>>>>>>>>>
> >> >> >>>>>>>>>>>> Best regards,
> >> >> >>>>>>>>>>>> Mach
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>> Tom
> >> >> >>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Best regards,
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Mach
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Of Greg Mirsky
> >> >> >>>>>>>>>>>>>> agree with Weiguo, single bit flag in fixed
> >> >> >>>>>>>>>>>>>> position would be sufficient and HW-friendly.
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Regards,
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Greg
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> chipset normally can't process optional field, it
> >> >> >>>>>>>>>>>>>> is hard to realize in-band performance measurement
> >> >> >>>>>>>>>>>>>> if using optional
> >> >> >>>>>>>>>>>>> field for marking.
> >> >> >>>>>>>>>>>>>> If it is more than a flag (and I assume it would
> >> >> >>>>>>>>>>>>>> be), and is not mandatory for all implementations,
> >> >> >>>>>>>>>>>>>> then it seems to fall into the category of optional
> >>extensions.
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Thanks, Larry
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> not affect forwarding of packets, the marking field
> >> >> >>>>>>>>>>>>>> is only used for performance measurement. The
> >> >> >>>>>>>>>>>>>> variation, may be viewed as one of passive
> >> >> >>>>>>>>>>>>>> performance
> >> >> >>>>>>>>>>>>> measurement methods.
> >> >> >>>>>>>>>>>>>> I maybe not clearly said in today’s NVO3 meeting,
> >> >> >>>>>>>>>>>>>>pls allow  me to reiterate the OAM data plane
> >> >> >>>>>>>>>>>>>>requirements on the mail list.
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Currently NVO3 data plane encapsulation only
> >> >> >>>>>>>>>>>>>>includes one OAM flag, it is used for Ping/Trace
> >> >> >>>>>>>>>>>>>>similar
> >>applications.
> >> >> >>>>>>>>>>>>>> This kind of OAM application is initiated by
> >> >> >>>>>>>>>>>>>>operators for  network connectivity verification,
> >> >> >>>>>>>>>>>>>>normally when network failure occurs.
> >> >> >>>>>>>>>>>>>> There is another OAM requirements of real time OAM
> >> >> >>>>>>>>>>>>>>gets packet statistics, and mark(coloring) the OAM
> >> >> >>>>>>>>>>>>>>flag relying on local policy when it performs
> >> >> >>>>>>>>>>>>>>packet number of  ingress NVE and the sum of all
> >> >> >>>>>>>>>>>>>>egress NVEs, packet loss can be deduced.
> >> >> >>>>>>>>>>>>>>configured by  operators or automatically acquired
> >> >> >>>>>>>>>>>>>>from centralized orchestration.
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> Thanks
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>> weiguo
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>
> >> >> >>>>>>>>>>>>>>

---

From: Tom Herbert <[email-masked]>
Date: Mon, 24 Nov 2014 11:13:27 -0800

On Fri, Nov 21, 2014 at 12:37 AM, Mach Chen <[email-masked]> wrote:
> If it only depends on the marking bit and the measuring period is set to a very small interval, indeed, that will affect the accuracy. But from engineering point view, an operator and an implementation will not set (or support) to a very aggressive period.

I don't think this would be true in our data center. The very reason
we would enable a passive mechanism is for getting accurate
measurements of high granularity. For instance, if we want to use this
as real time feedback for congestion control we need fine grained
information. Also, as part of debugging a customers problems it may
come down to us being able to identify specific packets that are being
dropped or experiencing unusual latency, I don't see how marking with
a couple of bits is sufficient for that. We already have this deployed
in the pre-NV world (mostly provided by TCP), in an NV world there are
many cases we won't have visibility into the customer's protocol so
we'll need to find alternative methods (which likely results in
annotating packets).

> And if the packets can carry some correlation information (e.g., the block/period number), then the accuracy should be no problem.
>

---

From: Mach Chen <[email-masked]>
Date: Tue, 25 Nov 2014 01:32:00 +0000

Hi Tom,

> Sent: Tuesday, November 25, 2014 3:13 AM
> To: Mach Chen
> Cc: Deepak Kumar (dekumar); [email-masked]
> > > >number) and timestamp information, then combine with the marking
> > > >bit, it can greatly simplify the marking based solution.
> > >
> > > +++DK:
> > > I think adding information regarding measurement field, block,
> > > period, etc. is not required in data path as more information
> > > reduces the mtu, and this can easily be added if required by TLV to
> > > OAM functionality with new subtype (as this is control or configuration
> functionality).
> >
> > 2) the correlation information, timestamps, counters, they could be
> > communicated either through in-band or out-or-band, each way has its
> > pros and cons;
> >
> > > Also even passive oam loss measurement solution to calculating loss
> > > is not accurate as packets can arrive late outside the measuring
> > > blocks. Even in that
> >
> > If it only depends on the marking bit and the measuring period is set to a very
> small interval, indeed, that will affect the accuracy. But from engineering point
> view, an operator and an implementation will not set (or support) to a very
> aggressive period.
> 
> I don't think this would be true in our data center. The very reason we would
> enable a passive mechanism is for getting accurate measurements of high
> granularity. For instance, if we want to use this as real time feedback for
> congestion control we need fine grained information. 

OK, let's put the solution part aside for a while. Could you please share what's your passive PM requirement here? For example, the granularity, the period interval, etc. I think these are important inputs for choosing the appropriate solution. 

Thanks,
Mach

> Also, as part of debugging a
> customers problems it may come down to us being able to identify specific
> packets that are being dropped or experiencing unusual latency, I don't see how
> marking with a couple of bits is sufficient for that. We already have this deployed
> in the pre-NV world (mostly provided by TCP), in an NV world there are many
> cases we won't have visibility into the customer's protocol so we'll need to find
> alternative methods (which likely results in annotating packets).

> 
> > And if the packets can carry some correlation information (e.g., the
> block/period number), then the accuracy should be no problem.
> >
> > In theory, you are right, if the delay of the packets of block exceed a threshold
> (e.g., a block period), the packets may be mis-counted into another block.
> >
> > > case to get accurate measurement instead of ipfix method, better to
> > > use OAM to exchange these marked packet counters on both ends and do
> > > loss measurement between two consecutive loss measurement replies.
> >
> > > For loss measurement, why we have to count traffic for marked
> > > packets only and not maintain counters per flow?
> >
> > To calculate the packet loss, counters maintenance (no matter at where) is
> necessary, it depends on the specific implementation.
> >
> > Best regards,
> > Mach
> >
> >
> > > >> but recently the focus is Performance Measurement (PM)
> > > >>requirement to
> > > >>NVO3
> > > >> that can satisfy the requirement. That's why I agree with Greg
> > > >>that we should  firstly make the agreement on the requirement.
> > > >> >
> > > >> Mach,
> > > >>
> > > >> (draft-ashwood-nvo3-oam-requirements-01) seems to already contain
> > > >>a fairly  comprehensive list of requirements. Particularly apropos
> > > >>to this discussion are:
> > > >>
> > > >> I believe that an an active OAM message format like Tissa
> > > >>describes would meet  these and most of the others in that draft.
> > > >
> > > >> meet the extensibility requirement. I don't believe that
> > > >>resource to be allocated for such a narrow purpose and for
> > > >
> > > >Looking through the bits in some headers, we could find that every
> > > >bit in a header has its own purpose. It's better that one bit could
> > > >be defined for as many usages as possible, but there is always tradeoff.
> > > >
> > > >>measurements is not even viable  in a lot deployments-- in this
> > > >>case we probably need timestamps to get RTT.
> > > >
> > > >For the deployments in question, the challenge for one way time
> > > >delay is time synchronization and its accuracy. The solution of
> > > >using one-bit for one-way delay is really workable, there have been
> > > >some prototypes and experiments show that.
> > > >
> > > >> To support passive OAM support in GUE, I would probably propose
> > > >>to add a  generic optional "measurement" field. This would provide
> > > >>some number of bits  in the header that can be used for passive
> > > >>measurement (possibly a few  different sizes say 32, 64, 128
> > > >>bits). The field can be structured to allow different  mechanisms
> > > >>(e.g. include timestamps for RTT measurement). This also reduces
> > > >>the constraints on the measurement techniques, for instance the
> > > >>marking  technique might no longer limited to use a single bit
> > > >>which should reduce the  complexity needed to deal with OOO or packet
> loss.
> > > >
> > > >Even with the solution as above, seems there needs at least one
> > > >bit(at the fix position of the header) that indicates there is an
> > > >optional field exist. In the case we could have opportunity and "enough"
> > > >number) and timestamp information, then combine with the marking
> > > >bit, it can greatly simplify the marking based solution.
> > >
> > > +++DK:
> > > I think adding information regarding measurement field, block,
> > > period, etc. is not required in data path as more information
> > > reduces the mtu, and this can easily be added if required by TLV to
> > > OAM functionality with new subtype (as this is control or configuration
> functionality).
> > > Also even passive oam loss measurement solution to calculating loss
> > > is not accurate as packets can arrive late outside the measuring
> > > blocks. Even in that case to get accurate measurement instead of
> > > ipfix method, better to use OAM to exchange these marked packet
> > > counters on both ends and do loss measurement between two consecutive
> loss measurement replies.
> > >
> > > For loss measurement, why we have to count traffic for marked
> > > packets only and not maintain counters per flow?
> > >
> > > Thanks,
> > > Deepak
> > > >
> > > >Thanks,
> > > >Mach
> > > >
> > > >>
> > > >> Tom
> > > >>
> > > >> >
> > > >>real/live  traffic of a path, it reflects the real performance of
> > > >>the path. For more detail  about active/passive PM, you may refer
> > > >>to the material of IPPM WG.
> > > >> >
> > > >> >>
> > > >> > What you are talking are just part of the OAM functions (e.g.,
> > > >> > CC,
> > > >>CV), for
> > > >> >> Your solution is active OAM,  i think it is a basic and
> > > >> >> important solution in whole OAM framework.The disccussed
> > > >> >> thread is about
> > > >>passive
> > > >> OAM.
> > > >> >> usecases and scenarios.The regular method for passive OAM is
> > > >> >> to add marking bits in packet header, in NVO3 case, the
> > > >> >> marking bits had better be set in NVO3 header.But just as Greg
> > > >> >> said,currently it's unfortunate that there is no accepted OAM
> > > >> >> requirements, gap analysis, and etc in the WG. We hope this
> > > >> >> work could be progressed more
> > > >>quickly.
> > > >> >> Thanks
> > > >> >> weiguo
> > > >> >> Your interpretation is correct. The entropy allows OAM packets
> > > >> >> to follow the same path as the data packet.
> > > >> >>
> > > >> >> As I noted earlier on in the thread, OAM processing would not
> > > >> >> kick in unless address matches the MEP/MIP. If address match
> > > >> >> MEP/MIP and OAM bit is set, then OAM processing begins.
> > > >> >>
> > > >> >> > Significant amount of work has gone in to this area during
> > > >> >> > TRILL
> > > >>OAM.
> > > >> >> > Please check the use of Flow entropy functionality proposed
> > > >> >> > in
> > > >>NVO3 OAM.
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> If I am reading this correctly, the OAM message would be
> > > >> >> composed of the encapsulation header, followed by 128 bytes of
> > > >> >> which contains a pseudo header for switching, followed by a
> > > >> >> self defining OAM
> > > message.
> > > >> >> > absence of WG agreed upon OAM Requirements; no gap analysis
> > > >> >> > of tools for NVO3 OAM; OAM flag does not help passive
> > > >> >> > performance measurement marking method (two bit-long field
> > > >> >> > for marking in fixed
> > > >> position).
> > > >> >> >
> > > >> >> > identification of OAM packet (though not necessarily OAM).
> > > >> >> > But IP clearly doesn't have such identification for OAM and
> > > >> >> > that, in part, why in-band requirement for IP OAM, both FM
> > > >> >> > and Active PM, is not attainable
> > > >> >> (ECMP environment).
> > > >> >> >
> > > >> >> > Regards,
> > > >> >> >
> > > >> >> > Greg
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > possible and all other protocol specific information should
> > > >> >> > be in the OAM protocol specific TLVs.
> > > >> >> >
> > > >> >> > LAYER2 OAM
> > > >> >> >
> > > >> >> >>instead of complicating the header, we can do OAM
> > > >> >> >>performance within OAM channel itself and this is extensible
> > > >> >> >>and can be done in hardware which is why mostly things are added
> in header.
> > > >> >> >>
> > > >> >> >>Also, Operators keep asking for new OAM tools (Fault
> > > >> >> >>detection, verification, isolation, Interworking, alarm,
> > > >> >> >>putting service in maintenance and perform test)  and Performance
> tools, eg:
> > > >> >> >>(Delay/Jitter, Actual Loss Measurement, Synthetic Loss,
> > > >> >> >>loopback signaling like TDM, Generate frames to verify qos
> > > >> >> >>etc.) and so OAM Channel solution will be extensible.
> > > >> >> >>
> > > >> >> >>Thanks,
> > > >> >> >>Deepak
> > > >> >> >>
> > > >> >> >>>As far as packet encapsulation is concern, all what is
> > > >> >> >>>needed is single bit. This bit is needed to prevent OAM
> > > >> >> >>>packets leaking out from the domain.
> > > >> >> >>>
> > > >> >> >>>E.g. if Address matches and OAM bit is set then it is an
> > > >> >> >>>OAM packet addressed to the local MEP/MP.
> > > >> >> >>>
> > > >> >> >>>Not other way around. Why? Because we want OAM to be as
> > > >> >> >>>closely as possible follow the Data path.
> > > >> >> >>>
> > > >> >> >>>sometimes called flow entropy) and at the end of that you
> > > >> >> >>>can have all of the stuff you need in the world of OAM.
> > > >> >> >>>
> > > >> >> >>>Hope this clarify
> > > >> >> >>>
> > > >> >> >>>Thanks
> > > >> >> >>>Tissa
> > > >> >> >>>> Seems we need an optional header extension or a TLV to
> > > >> >> >>>>carry all the  information (timestamps, oam Subtype).
> > > >> >> >>>>Sounds definitely more than a  32/64bit header could carry (*).
> > > >> >> >>>>
> > > >> >> >>>> The optional header extension, when done similar to GUE,
> > > >> >> >>>> has a fixed position. For the TLV this would be an
> > > >> >> >>>> additional requirement. This would allow for hardware-stamping.
> > > >> >> >>>>
> > > >> >> >>>mentioned that we might even want to put a "fake" packet
> > > >> >> >>>header as the first part of the encapsulated payload of an
> > > >> >> >>>OAM message for
> > > instance.
> > > >> >> >>>
> > > >> >> >>>> Now if we introduce such an OAM extension header it could
> > > >> >> >>>> as well carry the "first" bit we discussed for packet
> > > >> >> >>>> loss
> > > >>measurement (?).
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Regards, Marc
> > > >> >> >>>>
> > > >> >> >>>> On Sun, 16 Nov 2014 16:44:54 +0000, Deepak Kumar
> > > >> >> >>>> (dekumar)
> > > >>wrote:
> > > >> >> >>>>> Hi,
> > > >> >> >>>>>
> > > >> >> >>>>> Please see inline +++DK:
> > > >> >> >>>>>
> > > >> >> >>>>> On 11/14/14 11:09 AM, "Jon Hudson"
> > > >> >> >>>>> <[email-masked]>
> > > >>wrote:
> > > >> >> >>>>>
> > > >> >> >>>>>>
> > > >> >> >>>>>> One comment in line....
> > > >> >> >>>>>>
> > > >> >> >>>>>>>> Of Tom Herbert
> > > >> >> >>>>>>>>>>> mark a single packet, which helps the receiver to
> > > >> >> >>>>>>>>>>> pick the right packet?  And repeat this every time
> period T ?
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>> algorithm would work? Seems like there would need
> > > >> >> >>>>>>>>>> to be quite a bot of synchronization needed between
> > > >> >> >>>>>>>>>> end points (synchronized clocks, provisions to
> > > >> >> >>>>>>>>>> correlate measurements correctly with lost packets,
> > > >> >> >>>>>>>>>> replicated packets,
> > > etc.).
> > > >> >> >>>>>>>> https://datatracker.ietf.org/doc/draft-chen-ippm-colo
> > > >> >> >>>>>>>> ring
> > > >> >> >>>>>>>> -ba
> > > >> >> >>>>>>>> sed
> > > >> >> >>>>>>>> -ip
> > > >> >> >>>>>>>> fpm-fr
> > > >> >> >>>>>>>> amew
> > > >> >> >>>>>>>> ork/.
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>>The host clock  jitter we currently see in our network
> > > >> >> >>>>>>>>is currently usually  greater than one-way packet
> > > >> >> >>>>>>>>delay (in some cases much greater), so  in his words:
> > > >> >> >>>>>> <Jon> Thank you. As someone who has managed NTP more
> > > >> >> >>>>>> times
> > > >> and
> > > >> >> >>>>>>for  more years than I care to admit, this is a very
> > > >> >> >>>>>>good datapoint to consider.
> > > >> >> >>>>> delay and never found NTP useful even for our lab
> > > >> >> >>>>>networks (I am referring software based NTP NTPv3).
> > > >> >> >>>>> As mentioned below IEEE 1588v2 will vary based on
> > > >> >> >>>>>equipment and operator networks but in our testing we
> > > >> >> >>>>>found it very precise if properly deployed.
> > > >> >> >>>>> bit can be overloaded to do both Fault and performance
> > > >> >> >>>>>if OAM is defined with different oam Subtype for Delay
> > > >> >> >>>>>and Loss frames and it will not be too deep hardware
> > > >> >> >>>>>inspection) as that give large flexibility
> > > >> >> >>>>>(synthetic/real loss measurement,
> > > >> >> >>>>>>> [Vero] Thanks for this. What about the current
> > > >> >> >>>>>>> experience with
> > > >> >> >>>>>>> 1588v2 then?
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>> factors, one is the implementation limitation, the
> > > >> >> >>>>>>>> other the requirement of the operators. In the above
> > > >> >> >>>>>>>> reference, the suggested periods are 1s, 10s, 1min, 10min
> and 1h.
> > > >> >> >>>>>>>> I think if we were implementing delay measurement in
> > > >> >> >>>>>>>> GUE, I would advocate add a 64 bit optional field for
> > > >> >> >>>>>>>> echoed timestamp for a flow (usec resolution and
> > > >> >> >>>>>>>> similar in design TCP timestamp option). This easily
> > > >> >> >>>>>>>> gives a precise RTT, and if clocks are precisely
> > > >> >> >>>>>>>> synchronized then one way latency
> > > >>could be
> > > >> calculated also.
> > > >> >> >>>>>>> [Vero] If the source timestamp could be carried, it
> > > >> >> >>>>>>> could also be used for packet loss calculation/correlation.
> > > >> >> >>>>>>>
> > > >> >> >>>>>>>> Thanks,
> > > >> >> >>>>>>>> Tom
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>>> Best regards,
> > > >> >> >>>>>>>>> Mach
> > > >> >> >>>>>>>>>>
> > > >> >> >>>>>>>>>> Thanks,
> > > >> >> >>>>>>>>>> Tom
> > > >> >> >>>>>>>>>>
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>> One question I still have is: why is the
> > > >> >> >>>>>>>>>>> measurement done in the NVE
> > > >> >> >>>>>>>> header?
> > > >> >> >>>>>>>>>>>>>> Single bit is not sufficient if someone wants
> > > >> >> >>>>>>>>>>>>>>to perform loss and delay measurement
> > > >> >> >>>>>>>>>>>>>>simultaneously, then two bits needed.
> > > >> >> >>>>>>>>>>>>>quantum (as  well as other metrics maybe to be
> > > >> >> >>>>>>>>>>>>>added later)? In all the  protocols mentioned,
> > > >> >> >>>>>>>>>>>>>the reserved bits are a somewhat precious resource.
> > > >> >> >>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>> measurement, for loss measurement, it
> > > >> >> >>>>>>>>>>>> periodically set and clear the marking bit, a
> > > >>calculating are
> > > >> >> >>>>>>>>>>>> timestamps (collected at sender and receiver) are
> > > >> >> >>>>>>>>>>>> for
> > > >>the
> > > >> same packet.
> > > >> >> >>>>>>>>>>>> Since there is ECMP, the first packet of a block
> > > >> >> >>>>>>>>>>>>at the sender  may probably different from the
> > > >> >> >>>>>>>>>>>>first packet at the receiver,  thus it will get
> > > >> >> >>>>>>>>>>>>the mismatched timestamps to calculate the delay.
> > > >> >> >>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>> Best regards,
> > > >> >> >>>>>>>>>>>> Mach
> > > >> >> >>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>> Tom
> > > >> >> >>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Best regards,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Mach
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> NVO3 header is precious. I would like it is set
> > > >> >> >>>>>>>>>>>>>> in fixed field, rather than in option field.
> > > >> >> >>>>>>>>>>>>>> Because chipset normally can't process optional
> > > >> >> >>>>>>>>>>>>>> field, it is hard to realize in-band
> > > >> >> >>>>>>>>>>>>>> is it just a single flag bit, or large field
> > > >> >> >>>>>>>>>>>>>> with a counter or sequence number, or some kind of
> flow ID?
> > > >> >> >>>>>>>>>>>>>> If it is more than a flag (and I assume it
> > > >> >> >>>>>>>>>>>>>> would be), and is not mandatory for all
> > > >> >> >>>>>>>>>>>>>> implementations, then it seems to fall into the
> > > >> >> >>>>>>>>>>>>>> category of optional
> > > >>extensions.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Thanks, Larry
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Subject: [nvo3] 答复: Comments on NVO3 data
> plane
> > > >> >> >>>>>>>>>>>>>> requirements
> > > >> >> >>>>>>>> for
> > > >> >> >>>>>>>>>>>>>> OAM
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Hi Greg,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> I fully agree with you.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> NVO3 data encapsulation has a field for marking
> > > >> >> >>>>>>>>>>>>>> and not affect forwarding of packets, the
> > > >> >> >>>>>>>>>>>>>> marking field is only used for performance
> > > >> >> >>>>>>>>>>>>>> measurement. The
> > > >> >> >>>>>>>>>>>>>> NVO3 packet with this marking flag don't need
> > > >> >> >>>>>>>>>>>>>> to be sent to control plane, it is different
> > > >> >> >>>>>>>>>>>>>> from
> > > >> >> >>>>>>>>>>>>>> particular flow to facilitate measurement of
> > > >> >> >>>>>>>>>>>>>> some performance metric, whether loss or
> > > >> >> >>>>>>>>>>>>>> significantly alter, treatment of data flow in
> > > >> >> >>>>>>>>>>>>>> the
> > > >>network.
> > > >> >> >>>>>>>>>>>>>> Because of that, I believe, OAM flag should not
> > > >> >> >>>>>>>>>>>>>> be used for marking as that will force punting
> > > >> >> >>>>>>>>>>>>>> marked packets from fast forwarding path to the
> > > >> >> >>>>>>>>>>>>>> control plane. But it might be good to have a
> > > >> >> >>>>>>>>>>>>>> field in NVO3 header that may be used for
> > > >> >> >>>>>>>>>>>>>> I maybe not clearly said in today’s NVO3
> > > >> >> >>>>>>>>>>>>>>meeting, pls allow  me to reiterate the OAM data
> > > >> >> >>>>>>>>>>>>>>plane requirements on the mail list.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>operators for  network connectivity
> > > >> >> >>>>>>>>>>>>>>verification, normally when network failure occurs.
> > > >> >> >>>>>>>>>>>>>> There is another OAM requirements of real time
> > > >> >> >>>>>>>>>>>>>>OAM or synthesizing OAM. It can be used for
> > > >> >> >>>>>>>>>>>>>> When ingress NVE receives traffic from local
> > > >> >> >>>>>>>>>>>>>>TS, it gets packet statistics, and
> > > >> >> >>>>>>>>>>>>>>mark(coloring) the OAM flag relying on local
> > > >> >> >>>>>>>>>>>>>>policy when it performs
> > > >> >> >>>>>>>>>>>>>> NVO3 encapsulation, and gets packet statistics
> > > >> >> >>>>>>>>>>>>>>with the real  time OAM flag marking. By
> > > >> >> >>>>>>>>>>>>>>comparing the packet number of  ingress NVE and
> > > >> >> >>>>>>>>>>>>>>the sum of all egress NVEs, packet loss can be
> deduced.
> > > >> >> >>>>>>>>>>>>>> This method can be applicable for both unicast
> > > >> >> >>>>>>>>>>>>>>and multicast  traffic. Local policy on ingress
> > > >> >> >>>>>>>>>>>>>>NVE is configured by  operators or automatically
> > > >> >> >>>>>>>>>>>>>> nvo3 mailing list [email-masked]