Subject: Re: [TICTOC] draft-ietf-mpls-loss-delay Timestamp

From: "Shahram Davari" <[email-masked]>
Date: Thu, 3 Mar 2011 14:12:13 -0800

Agree.

Shahram

________________________________
From: Ron Cohen <[email-masked]>
To: Shahram Davari
Cc: [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Sent: Thu Mar 03 13:35:11 2011
Subject: Re: [TICTOC] draft-ietf-mpls-loss-delay Timestamp

Hi Shahram, I guess adding a requirement to take timestamp rollover into account would do. If there is a strong reason to also leave NTP timestamp as an option, a note explaining the ramification of leap seconds should be added. I suggest that a continuous timescale (i.e. PTP) would be selected in the draft as the default/must option. Best, Ron

On Thu, Mar 3, 2011 at 9:45 PM, Shahram Davari <[email-masked]<mailto:[email-masked]>> wrote:
Hi Ron,

Although the PTPv2 is 80 bytes, but I don’t see a need to use all 80 bytes for delay measurement. The upper 2^32 seconds covers almost 136 years, which should be good enough to measure delay. Also lets’ be compatible with Y.1731 format since that HW already exist.

Thx
Shahram

From: Ron Cohen [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Thursday, March 03, 2011 11:22 AM
To: [email-masked]<mailto:[email-masked]>
Cc: Shahram Davari; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [TICTOC] draft-ietf-mpls-loss-delay Timestamp

Hi,

I apologize in advance if my comments below where already been discussed.

The draft specify using PTPv1 timestamp format and not PTPv2 timestamp format. The seconds part of the PTPv2 timestamp is 48bits, and not 32bits as written in the draft. NTP timestamps will roll over in 2036. PTPv2 timestamps do not rollover.

I don't think its advisable to include timestamps that rolls over ~20 years from now.

In addition, NTP timestamp, unlike PTP timestamp, 'jumps' in leap seconds events. This makes calculation of time differences hard, and would render some measurements ambiguous. It is much easier to use continuous timescale like PTP.

In my opinion changing the timestamp format to PTPv2 timestamp format (i.e. 48bit/32bit sec/ns) should be the right way to go. I would also remove the NTP timestamp format option to avoid leap seconds confusion and 2036 rollover events.

Best,
Ron

On Thu, Mar 3, 2011 at 8:26 PM, <[email-masked]<mailto:[email-masked]>> wrote:
My feeling is 1588 is the right answer in the long run since it is already
coupled to the hardware and widely deployed in networks using the "packet
clock derived time" in precise time and frequency delivery.

Pat

            "Shahram Davari"
            <davari@broadcom.
            com>                                                       To
            Sent by:                  "[email-masked]<mailto:[email-masked]>"
            tictoc-bounces@ie         <[email-masked]<mailto:[email-masked]>>,
            tf.org<http://tf.org>                    "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
                                                                       cc
                                      "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
            03/03/2011 10:05                                      Subject
            AM                        Re: [TICTOC]
                                      draft-ietf-mpls-loss-delay
                                      Timestamp

Hi Stewart,

Accurate delay measurement requires the Timestamp to be done in HW. It is
true that most routers support NTP today, but the majority of those are
really maintained in Software and AFAIK most HW based timestamps
implementations are 1588 based and really the industry is shifting toward
1588 and SyncE for network synchronization. So I support 1588 as being the
default.

Thanks,
Shahram

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of
Stewart Bryant
Sent: Thursday, March 03, 2011 8:07 AM
To: [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [TICTOC] draft-ietf-mpls-loss-delay Timestamp

We have received a LC comment on draft-ietf-mpls-loss-delay concerning the
default timestamp.

In the first version of the draft we proposed NTP, but following initial
comments from the MPLS-TP community we changed to IEEE1588. This
requirement for IEEE1588 can be traced back to the choice of using IEEE1588
in Y.1731.

We have now received a LC request to change the default back to NTP.

NTP is the "natural" choice for an IETF protocol. NTP is specified in IETF
and is used in other MPLS protocols such as LSP ping. It is implemented on
almost every host and every router. However in general the implementations
provides a relatively low precision timestamp, and the NTP time
distribution infrastructure operates on a best effort basis. Thus even a
good client implementation would normally have a relatively low quality
path to the server, which would result in a low quality of timestamp. NTP
could be made to work to higher accuracy, but defacto upgrading NTP and
providing high quality NTP paths to the time servers is not getting much
attention. Computing timestamp differences is easier with NTP.

On the other hand IEEE1588 has defacto become the two way time tranfer
protocol for precision applications. IEEE1588 only provides high quality
time in well engineered networks with some form of hop by hop assistance.
It is not widely implemented other than in equipment targeted to specific
markets, although one of those markets is in mobile backhaul applications
where we expect to see significant initial deployment of MPLS-TP. Computing
timestamp differences is harder with IEEE1588

The loss-delay work is targeting to be able to do one way packet delay
measurement, so it does need a higher quality of timestamp than is required
in general purpose network instrumentation.

Converting from IEEE1588 to NTP is not trivial, since they use different
epochs, and different representations of sub-second time. In a
"traditional" NTP implementation the time error due to on the fly
conversion is likely to be small compared to the time error in the time
synchronization system, but an IEEE1588 system would need hardware
conversion to maintain the accuracy for an NTP timestamp.

So the question arises, should we make IEEE1588 or NTP the default for
draft-ietf-mpls-loss-delay. Much as I would like to suggest that we should
go back to NTP for consistency with other IETF protocols, I have difficulty
reconciling this with the situation in network deployments and thus suggest
that we continue to use IEEE1588.

What is the opinion of the working group?

Stewart (speaking as a draft editor)
_______________________________________________
TICTOC mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tictoc

_______________________________________________
TICTOC mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tictoc

---

From: Stewart Bryant <[email-masked]>
Date: Thu, 03 Mar 2011 22:48:47 +0000

I do not think that rollover or leap seconds are much of a problem in 
this application. We are measuring the transit time of a packet across a 
network. That is less than a second. Rollover or leap second errors mean 
that we make a mistake once in every 136 years in the first case and 
maybe never in the case of leap seconds. I am sure that it would be 
acceptable to simply state that for that for that one second at some 
time in the future we cannot make that measurement.

BTW we are only going to use 32 bits of seconds and 32 bits of sub 
seconds in the 1588 case, and even then most of the sub second bits will 
be ignored. I don't thing that we need to be measuring time of flight 
over 8" of  cable.

Stewart

On 03/03/2011 22:12, Shahram Davari wrote:
> Agree.
>
> Shahram
>
> ------------------------------------------------------------------------
> *From*: Ron Cohen <[email-masked]>
> *To*: Shahram Davari
> *Cc*: [email-masked] <[email-masked]>; [email-masked] 
> <[email-masked]>; [email-masked] <[email-masked]>; 
> [email-masked] <[email-masked]>
> *Sent*: Thu Mar 03 13:35:11 2011
> *Subject*: Re: [TICTOC] draft-ietf-mpls-loss-delay Timestamp
>
> Hi Shahram, I guess adding a requirement to take timestamp rollover 
> into account would do. If there is a strong reason to also leave NTP 
> timestamp as an option, a note explaining the ramification of leap 
> seconds should be added. I suggest that a continuous timescale (i.e. 
> PTP) would be selected in the draft as the default/must option. Best, Ron
>
>     Although the PTPv2 is 80 bytes, but I don’t see a need to use all
>     80 bytes for delay measurement. The upper 2^32 seconds covers
>     almost 136 years, which should be good enough to measure delay.
>     Also lets’ be compatible with Y.1731 format since that HW already
>     exist.
>
>     Thx
>
>     Shahram
>
>     *From:*Ron Cohen [mailto:[email-masked]
>     <mailto:[email-masked]>]
>     *Sent:* Thursday, March 03, 2011 11:22 AM
>     *To:* [email-masked] <mailto:[email-masked]>
>     *Cc:* Shahram Davari; [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>;
>     [email-masked] <mailto:[email-masked]>
>     *Subject:* Re: [TICTOC] draft-ietf-mpls-loss-delay Timestamp
>
>     Hi,
>
>     The draft specify using PTPv1 timestamp format and not PTPv2
>     timestamp format. The seconds part of the PTPv2 timestamp is
>     48bits, and not 32bits as written in the draft. NTP timestamps
>     will roll over in 2036. PTPv2 timestamps do not rollover.
>
>     In addition, NTP timestamp, unlike PTP timestamp, 'jumps' in leap
>     seconds events. This makes calculation of time differences hard,
>     and would render some measurements ambiguous. It is much easier to
>     use continuous timescale like PTP.
>
>     In my opinion changing the timestamp format to PTPv2 timestamp
>     format (i.e. 48bit/32bit sec/ns) should be the right way to go. I
>     would also remove the NTP timestamp format option to avoid leap
>     seconds confusion and 2036 rollover events.
>
>     Best,
>     Ron
>
>                 com>                                                  
>         To
>     tf.org <http://tf.org>                    "[email-masked]
>     <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
>                                                                      
>          cc
>                                           "[email-masked]
>     <mailto:[email-masked]>" <[email-masked] <mailto:[email-masked]>>
>                 03/03/2011 10:05                                    
>      Subject
>                 AM                        Re: [TICTOC]
>
>     From: [email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>]
>     On Behalf Of
>     Stewart Bryant
>     To: [email-masked] <mailto:[email-masked]>
>     Cc: [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>
> mpls mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/mpls

-- 
For corporate legal information go to:

http://www.cisco.com/web/about/doing_business/legal/cri/index.html

---

From: Alexander Vainshtein <[email-masked]>
Date: Fri, 4 Mar 2011 13:25:01 +0200

Stewart, Shahram and all,
Here is my proposal based on reading this thread and some thinking.

Instead of trying to specify the mandatory default for the timestamp format, let's mandate something else:

 1.  Ability to convert the timestamps from PTP to NTP for 1-way delay measurements.
    *   The computational load associated with this operation could result in relatively low rate of 1-way DM messages (say 1 pps?) in line with the general direction of the draft which says that loss and delay measurement should not stretch the network resources.
    *   The direction (PTP to NTP) of conversion has been chosen because computation of differences is simpler for NTP timestamps.
 2.  Ability to compute differences between pairs of timestamps in the same format both for NTP and PTP timestamps for 2-way delay measurements. This operation does not require conversion from one format into another.

Neither of the two items above carries with it any special HW requirements. Hence, IMHO this approach, if accepted, would promote backward compatibility with the installed LSR base while allowing new equipment to benefit from the HW-based PTP-oriented timestamping mechanisms.

Below you can find  the chain of reasoning that has led me to this proposal (you may skip this part if you are not interested:-).

 1.  I tend to agree with Stewart: neither rollover nor leap seconds are really relevant for the delay measurement application. (I did not think so yesterday:-().
 2.  AFAIK (and I have discussed in a personal email exchange with the authors of the draft), the required accuracy of delay measurements is not specified - neither in the draft itself (intentionally) nor in other relevant standards.
    *   This thread really confirms that - the best I see is a somewhat vague  reference to "expectations of the Transport network operators". Did I miss something here?
    *   Without these requirements references to comparative "accuracy of the timestamps"  for NTP vs. PTP in this thread do not make too much sense to me
 3.  The available accuracy of delay measurements depends on many factors in addition to the timestamp resolution. E.g., we all know that one-way delay measurements require ToD synchronization between two endpoints. It is also widely known that, e.g., TDD-based mobile application require ~1 microsecond accuracy of ToD synchronization. Hence I conclude that the achievable accuracy of 1-way delay measurements will be probably in the microseconds range, no matter which protocol and what kind of timestamps we use (as long as the granularity of the timestamps is in microseconds). Hence I agree with Stewart - quite a few  the bits in the nanoseconds part of the PTP timestamp can be actually ignored
 4.  The draft we discuss here is NOT an MPLS-TP draft. It is a generic MPLS draft with an accompanying MPLS-TP document. IMO imposing a TP-derived requirement on non-TP equipment doesn't make too much sense
 5.  The draft inherently supports the situation when the requester and the responder use different timestamping formats:
    *   For 2-way DM only ability to compute the difference between two NTP timestamps and two PTP ones is required
    *   For 1-way DM conversion to a single format (e.g., NTP) is required. While this may be computationally heavy, it is not an operation that has to be done in HW or even in hard real-time. If, say, you send the delay measurement packets once a second, the originator would have 1" to do this "heavy work" - lots of cycles in a modern CPU!  BTW, one of the issues *not addressed* in the draft is the rate at which delay measurement messages are supposed to be sent
    *   The installed base of LSRs will probably benefit from using SW-generated NTP timestamps (with whatever accuracy can be achieved)
    *   New equipment that supports HW timestamping for PTP "close to PHY" will benefit from using PTP timestamps.

My 2c,
     Sasha

________________________________
From: [email-masked] [[email-masked]] On Behalf Of Stewart Bryant [[email-masked]]
Sent: Friday, March 04, 2011 12:48 AM
To: Shahram Davari
Cc: [email-masked]'; [email-masked]'; [email-masked]'; [email-masked]'; [email-masked]'
Subject: Re: [mpls] [TICTOC] draft-ietf-mpls-loss-delay Timestamp

I do not think that rollover or leap seconds are much of a problem in this application. We are measuring the transit time of a packet across a network. That is less than a second. Rollover or leap second errors mean that we make a mistake once in every 136 years in the first case and maybe never in the case of leap seconds. I am sure that it would be acceptable to simply state that for that for that one second at some time in the future we cannot make that measurement.

BTW we are only going to use 32 bits of seconds and 32 bits of sub seconds in the 1588 case, and even then most of the sub second bits will be ignored. I don't thing that we need to be measuring time of flight over 8" of  cable.

Stewart

Shahram

________________________________
From: Ron Cohen <[email-masked]><mailto:[email-masked]>
To: Shahram Davari
Cc: [email-masked]<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>
Sent: Thu Mar 03 13:35:11 2011
Subject: Re: [TICTOC] draft-ietf-mpls-loss-delay Timestamp

Thx
Shahram

Hi,

Best,
Ron

Pat

            "Shahram Davari"
            <davari@broadcom.
            com>                                                       To

Hi Stewart,

Thanks,
Shahram

_______________________________________________
mpls mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/mpls

---

From: Alexander Vainshtein <[email-masked]>
Date: Fri, 4 Mar 2011 19:05:08 +0200

Stewart, Dan and all,
I have a question regarding the draft which, IMHO, to some extent is related to the ability to use HW-assisted PTP timestamping.

The question is, at which moment the DM message receives its Tx timestamp: before it is queued for transmission or after it has been extracted from the queue and is ready to be sent on the appropriate physical media?

I believe I've asked a similar question in the past with regard to direct loss measurement. However, I did not find any reference to this issue in the draft.

In the case of delay measurements, the following points should be taken into consideration IMO:

 1.  Is delay incurred by queuing in the node that originates the packet measured as part of the E2E delay?
    *    From the naive point of view, this delay is neither better nor worse than delay incurred by queuing in the intermediate nodes, so it should be taken into account
    *   One known application for delay measurements is PDV estimation for TDM PWs, since their jitter buffers must be configured in such a way as to accommodate maximal PDV
    *   I would assume that SW-based implementations would always use this type of timestamping
 2.  When it comes to PTP, there are valid reasons to put the timestamps as close as possible to actual egress and ingress, especially in the case of hop-by-hop operation. This is because the timestamps are used to estimate offset between ToD clocks in the ToD nodes. As a consequence, PTP-capable HW will most probably be able to insert the timestamps after egress queuing in order to reduce PDV.
 3.  As a consequence, I can easily imagine a situation when a (presumably, more accurate) HW-based implementation would provide inconsistent information about E2E PDV (if congestion in end nodes is a major contributor to PDV), while a SW-based implementation would yield a much more realistic picture. In particular, setting jitter buffers of TDM PWs in accordance with PDV estimated using HW timestamps may be non-operational.
 4.  As a consequence, PTP-capable HW may be not usable for E2E delay measurements if it cannot provide the timestamps in the appropriate  points of the data path.

My 2c,

     Sasha

________________________________
From: Alexander Vainshtein
Sent: Friday, March 04, 2011 1:25 PM
To: [email-masked]; Shahram Davari
Cc: [email-masked]'; [email-masked]'; [email-masked]'; [email-masked]'; [email-masked]'; 'Dan Frost'; Martin Greenberg; Jacob Ruthstein; Robert Rennison; Idan Kaspit
Subject: RE: [mpls] [TICTOC] draft-ietf-mpls-loss-delay Timestamp

My 2c,
     Sasha

Stewart

Shahram

Thx
Shahram

Hi,

Best,
Ron

Pat

            "Shahram Davari"
            <davari@broadcom.
            com>                                                       To

Hi Stewart,

Thanks,
Shahram

---

From: Dan Frost <[email-masked]>
Date: Fri, 4 Mar 2011 17:16:30 +0000

Hi Sasha,

On Fri, Mar 04, 2011 at 07:05:08PM +0200, Alexander Vainshtein wrote:
> Stewart, Dan and all,
> I have a question regarding the draft which, IMHO, to some extent is
> related to the ability to use HW-assisted PTP timestamping.
> 
> The question is, at which moment the DM message receives its Tx
> timestamp: before it is queued for transmission or after it has been
> extracted from the queue and is ready to be sent on the appropriate
> physical media?
> 
> I believe I've asked a similar question in the past with regard to
> direct loss measurement. However, I did not find any reference to this
> issue in the draft.

On this topic please see the last paragraph of Section 2.7.2.  Since
this draft is intended to be general enough to cover a wide variety of
situations, I'm inclined to avoid saying more than is said there.

Cheers,
-d

---

From: Alexander Vainshtein <[email-masked]>
Date: Sat, 5 Mar 2011 22:40:15 +0200

Dan,
Lots of thanks for a prompt response. I've looked up Section 2.7.2 as you've suggested. 

I think that placing the text that deals with location of the loss and delay measurement points in the section on QoS is misleading.
Would you consider splitting it into a dedicated section?

As for the position you've taking: I understand the intention to make the draft generic enough and not to prescribe specific location of the measurement points. However, leaving this point open carries with it serious interoperability issues. As a minimum, I would suggest replacing SHOULD in this text with MUST.

My 2c,
     Sasha

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Dan Frost [[email-masked]]
Sent: Friday, March 04, 2011 7:16 PM
To: Alexander Vainshtein
Cc: [email-masked]; [email-masked]
Subject: Re: [mpls] [TICTOC] draft-ietf-mpls-loss-delay Timestamp

Hi Sasha,

Cheers,
-d

---

From: Dan Frost <[email-masked]>
Date: Mon, 7 Mar 2011 12:53:32 +0000

On Sat, Mar 05, 2011 at 10:40:15PM +0200, Alexander Vainshtein wrote:
> I think that placing the text that deals with location of the loss and
> delay measurement points in the section on QoS is misleading.  Would
> you consider splitting it into a dedicated section?

Sure.

> As for the position you've taking: I understand the intention to make
> the draft generic enough and not to prescribe specific location of the
> measurement points. However, leaving this point open carries with it
> serious interoperability issues. As a minimum, I would suggest
> replacing SHOULD in this text with MUST.

Agreed.

-d

> My 2c, Sasha