Subject: Re: draft-elkins-v6ops-ipv6-pdm-recommended-usage-00

From: joel jaeggli <[email-masked]>
Date: Wed, 28 Aug 2013 03:17:42 -0700

On 8/27/13 5:48 AM, Nalini Elkins wrote:
> Joel,
>
> The synchronization and granularity of the timestamps for our proposal is one of the two outstanding issues.   I will take up the other issue (is this proposal useful for the Internet?) in a separate thread.
>
> We have also had some discussions with the folks in the IPPM group, so I am copying that working group on this thread.
>
> To restate the issues as I understand them (please correct me if I am wrong):
>
> 1.   Timestamp correlation depends on both ends being synchronized using a standard mechanism.
> 2.   This is a question of 'best practices'.
> 3.   Is the timestamp of sufficient quality or granularity to be useful.
>
> The answers to 1. and 2.  is yes.   Of course.   If people find this kind of data to be of use, then they will synchronize.  It is my understanding that activities other than those proposed by us require time synchronization.   So, it is quite likely that people may be doing this already.
>
> The answer to 3. is an interesting one.   I will cite a paper from Dr. Joachim Fabini.  It is available at:
>
> http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6553584
>
> The significant passage is on page 7:
>
> "Mobile client and server are both accurately timesynchronized using EM-406A GPS modules with PPS support.    The RS232 level conversion required for this GPS module is implemented by a SparkFun 8334 GPS evaluation board
> which has been extended by the PPS circuit as proposed in [8]. The Network Time Protocol Daemon (ntp) synchronizes both endpoints accurately with global time using the evaluation board’s level-converted PPS and GPS time signals accessed by the kernel using native RS232 serial interfaces. Tests over several weeks have shown that this setup can synchronize the system clock accurately down to 5μs to global time. Even if deployed in nonoptimum conditions (indoor, 1st ﬂoor, street view with 6-ﬂoor opposite building), the clock accuracy is better than 50μs." 
network devices are by-in-large not built with embedded GPS.
 
By way of demonstration of the level of measuerment error I expect.
Juniper RPM can do asic based timestamping on measurement traffic
between two devices. I have a path, the length of which does not vary by
much (it's a wave-length between san jose and ashburn virginia). the
routers derive time from a clustered of peered stratum 2 servers which
in turn are clients of stratum-1 sources that are traceable to NIST. 

in the following graph:

http://i.imgur.com/AkDRs9D.png

the top line is the RTT as measured by the sending router.

the green line is the difference between the timestamp applied by the
reciveing router in ashburn and the timestamp applied by the sending
router, and the redline is the differnce between the ashburn router
sending timestamp and the timestamp genrated when the packet is recieved
again in san jose.

the vertical axis is thousands of microseconds.

I'm farily comfortable that my NTP and my devices clocks aren't totaly
busted. I'm also comfortable with the idea that there's substantially
higher variation  out there when you get out a of a single admistrative
domain.
> I believe that 5 - 50 microseconds of accuracy is sufficient for triage.  In our proposal, we want three times: inbound network path, the server processing time, or the outbound network path.  I believe that for accurate triage, we will be able to say that the problem is in the inbound network path, the server, or the outbound network path even if the clocks vary by 50 microseconds.  In my experience, the differences between the 3 calculations which we want vary by quite a bit more than 50 microseconds.   I actually have quite a bit of data available to prove this from running diagnostics with a number of companies and over the Internet in the last few years and I will do a paper (when I get a free weekend!).   I will discuss my findings with the IPPM group as I believe this to be of interest to them.
>
> For those who may be new to this discussion, they may care to look at our proposals:
>
> http://tools.ietf.org/html/draft-elkins-6man-ipv6-pdm-dest-option-00
>
>
> http://tools.ietf.org/html/draft-elkins-v6ops-ipv6-end-to-end-rt-needed-00
>
>
> http://tools.ietf.org/html/draft-elkins-v6ops-ipv6-packet-sequence-needed-00
>
>
> http://www.ietf.org/id/draft-elkins-v6ops-ipv6-pdm-recommended-usage-00.txt
>
>
> Thanks,
>
> Nalini Elkins
> Inside Products, Inc.
> (831) 659-8360
> www.insidethestack.com
>
>
>
> ________________________________
> From: joel jaeggli <[email-masked]>
> To: Nalini Elkins <[email-masked]>; IPv6 Ops WG <[email-masked]> 
> Sent: Thursday, August 1, 2013 6:45 AM
> Subject: draft-elkins-v6ops-ipv6-pdm-recommended-usage-00
>
>
> Since I ran into you in the hall and the dicussion turned to ntp...
>
> I'll try and be succinct, and as a non-expert in the time field this
> should be taken with a grain of salt.
>
> The generic utility of a high-resultion time-stamp is imho dodgey
> outside of situations where the clocks are deliberately syncronized and
> traceable to a common standard whether the protocol used for this is ntp
> or ieee 1588 (or since this is the ietf PTPV2) . While this is tractable
> for devices in a single span of control, the agruement that ntp might be
> suffcient to make this timstamp useful generically between two
> aribitratry devices where this functionality may need to be enabled is
> imho a hard one to assert.
>
> It is a set of operational practice and discipline moreso than the
> choice of technology that allows for a high-resolution timestamp to have
> sufficient precision to be useful between hosts.
>
> joel 
>

---

From: Nalini Elkins <[email-masked]>
Date: Wed, 28 Aug 2013 03:53:11 -0700 (PDT)

Joel,

>>network devices are by-in-large not built with embedded GPS.

We are not talking about network devices.   Our header is a Destination Options header which ONLY end hosts need to be concerned with.   Your data is very interesting but it is not what we are talking about.

What is of concern is what actual end hosts can do.  This is a conversation that I will be starting with the OS vendors.    That is why Joachim's paper was of interest.

Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

________________________________
From: joel jaeggli <[email-masked]>
To: Nalini Elkins <[email-masked]>; IPv6 Ops WG <[email-masked]>; "[email-masked]" <[email-masked]> 
Cc: ALFRED C (AL) <[email-masked]>; "[email-masked]" <[email-masked]>; "[email-masked]" <[email-masked]> 
Sent: Wednesday, August 28, 2013 3:17 AM
Subject: Re: draft-elkins-v6ops-ipv6-pdm-recommended-usage-00

in the following graph:

---

From: Nalini Elkins <[email-masked]>
Date: Thu, 29 Aug 2013 05:47:35 -0700 (PDT)

Please take a look at:

http://www.ripe.net/data-tools/stats/ttm/test-traffic-measurement-service

Note that what they provide is:

    * NTP-server, the test-boxes can act a stratum 1 server for the machines on your network, providing time-stamps with an accuracy of 10 microseconds.

Thanks,

Nalini Elkins
Inside Products, Inc.
(831) 659-8360
www.insidethestack.com

---

From: joel jaeggli <[email-masked]>
Date: Thu, 29 Aug 2013 10:48:26 -0700

>     * NTP-server, the test-boxes can act a stratum 1 server for the machines on your network, providing time-stamps with an accuracy of 10 microseconds.
With respect to the IETF 87 presentation/discussion I think is is a
dicussion of the quality of time and it relationship to the general
utility of of pdm optional header, not the availability of specific
point solutions.

With respect to the design of TTM the claimed timestamp accuracy is
associated with packets emitted by the TTM not other devices associated
with them which is course subject to the same general constraints other
ntp deployments.

---

From: Nalini Elkins <[email-masked]>
Date: Thu, 29 Aug 2013 18:36:14 -0700 (PDT)

>utility of of pdm optional header, not the availability of specific

Let me back up and see if I can restate your basic concern in simple words so that I am sure that I understand you.

I believe that you are concerned that the timestamps provided by NTP will not be accurate enough if the timestamps are taken from more than one administrative domain.   Our PDM proposal depends on fairly reliable timestamps.  So, if the timestamps are not reliable, then the PDM is not very useful.

Please let me know if I have understood you.

Thanks,
Nalini

---

From: joel jaeggli <[email-masked]>
Date: Fri, 30 Aug 2013 11:45:55 -0700

On 8/29/13 6:36 PM, Nalini Elkins wrote:
> >    * NTP-server, the test-boxes can act a stratum 1 server for the
> machines on your network, providing time-stamps with an accuracy of 10
> microseconds.
> Let me back up and see if I can restate your basic concern in simple
> words so that I am sure that I understand you.
>
> I believe that you are concerned that the timestamps provided by NTP
> will not be accurate enough if the timestamps are taken from more than
> one administrative domain.
The clock in the computer is providing the timestamp, ntp is
syncronizing the clock in the computer to another clock.
>   Our PDM proposal depends on fairly reliable timestamps.
It depends on the accuracy of clocks generating the timestamps with
respect to each other.
>  So, if the timestamps are not reliable, then the PDM is not very useful.
If a diagnostic result is dependant on low number of ms or usec level
timestamp level accuracy and you don't have that, then yes. A related
question is how do you know when you do or don't? if you have long
timeseries data for two devices you may have evidence of the magnitude
of the eror. If packets arrive from the future with respect to your
clock that's a nice and gross indicator. Looking at a packet trace in
the past, the information availble to reconstruct the state of the
clocks is only the timestamps, baring external sources of that information.

---

From: Nalini Elkins <[email-masked]>
Date: Sat, 31 Aug 2013 05:19:18 -0700 (PDT)

From: joel jaeggli <[email-masked]>
To: Nalini Elkins <[email-masked]>; "MORTON JR., ALFRED C (AL)" <[email-masked]>; IPv6 Ops WG <[email-masked]>; "[email-masked]" <[email-masked]> 
Cc: "[email-masked]" <[email-masked]>; "[email-masked]" <[email-masked]> 
Sent: Friday, August 30, 2013 11:45 AM
Subject: Re: draft-elkins-v6ops-ipv6-pdm-recommended-usage-00

>clocks is only the timestamps, baring external sources of that information.

The basic issue here goes far beyond our PDM header.   Accurate time synchronization is necessary for much data processing today.   Let me quote from an IBM Redbook:

"In the information technology world, time synchronization has become a critical component for managing the correct order of the events in distributed applications (transaction processing, message logging), especially for audit and legal purposes." http://www.redbooks.ibm.com/redbooks/pdfs/sg247280.pdf (section 1.1)

If you are arguing that time synchronization cannot be done to the millisecond or microsecond level, then a great deal of data processing would flat out not work.   Let me point you to something called 'parallel sysplex' which is a way of putting together very large computer systems http://www-03.ibm.com/systems/z/advantages/pso/bsvsps.html  These kind of extremely expensive systems are in most large data centers today.  They require synchronization to the microsecond level.  They do it using NTP.

One of the members of my team has implemented NTP with a consortium of 120 separate organizations (all DIFFERENT administrative domains) and they have synched up to 20 - 50 milliseconds.  If anyone would like, they can contact us offline and we can share the implementation documentation.  If you would like, I can get from them the sources for timing that they use.   Multiple MASTER Clocks geographically dispersed for either accuracy and/or backup-contingency purposes may be needed.   By the way, this was done several years ago.

As far as our PDM header is concerned, what we are looking for MOST is to do triage.   That is to say which of the three: inbound network time, server processing time, and outbound network time is consistently quite large.   What happens in real life is that problems happen with response time to a major region or bank branch, and we need to quickly (hopefully VERY quickly) get the right team to work solving the problem.   So, what is required is that the differences in those times be consistently greater than any problems with time synchronization.   To give an example:

Assume: time synchronization varies at 50 milliseconds.

Inbound network time = 100 milliseconds
Server time = 500 milliseconds
Outbound network time = 200 milliseconds

The point of this game is to say which is the failing component.  If you add or subtract 50 milliseconds from the above numbers, it won't matter.  It will still be the server time.

If you think that we will have problems if all times are close together, (ex. all times = 40 milliseconds), then yes.  And, it doesn't matter.  What we are looking for in doing diagnostics and triage are, basically, the outliers (or those at the 95% to be a bit more precise). I am looking for which response times are among the worst.  That is where the problems are likely to be.   In those transactions, in my experience, I have NEVER found transactions where all numbers are close together.   

Definitely for trending and capacity planning purposes, we want to look at all response times.  But, I believe that time synchronization to the level that is required can be achieved today.   I have cited numerous examples of organizations doing so.

If our PDM proposal is accepted, it will still be 5-6 years before it is in use because operating systems need to be changed to use it.   I expect that time synchronization will become even better over time.   Notice how much more prevalent GPS is from even a few years ago.