Subject: (Fwd) Excessive data staleness for ifMIB and interfaces get-re

From: [email-masked]
Date: Mon, 23 Aug 1999 09:27:04 -500

<color><param>0100,0100,0100</param>Hi,

</color>So far there have been no substantive responses to the query I'm forwarding.

What is the position of the ifMIB working group on SNMP data accuracy?

In particular, is varying data staleness, for instance in the 10s of seconds 
range, consistent with existing and proposed IETF standards?  What are the 
current expectations with respect to acceptable and unacceptable levels of 
data staleness? Where are these expectations clearly set forth?

Are we entering a new era where technology limitations are going to undermine 
the usefulness of SNMP (and many current practices) for monitoring network 
performance?  Or is there agreement that it should be possible to return 
accurate data (say to within N timeticks of a sysUpTime returned in the same 
response - N small, hopefully 1), at all current bandwidths - and as 
bandwidths ramp up over the next several years?

Carl Holmberg

GTE Internetworking<color><param>0100,0100,0100</param>

------- Forwarded Message Follows -------

</color>From:           	<color><param>0000,0000,8000</param>[email-masked]

</color>Organization:   	<color><param>0000,0000,8000</param>Instantiations

</color>To:             	<color><param>0000,0000,8000</param>[email-masked]

</color>Date sent:      	<color><param>0000,0000,8000</param>Wed, 14 Jul 1999 07:40:56 -500

<bold></color>Subject:        	<color><param>0000,0000,8000</param>Excessive data staleness for ifMIB and interfaces get-responses

</bold></color>Send reply to:  	<color><param>0000,0000,8000</param>[email-masked]

</color>Copies to:      	<color><param>0000,0000,8000</param>[email-masked], [email-masked], "Bert Wijnen" <<[email-masked]>

</color>Priority:       	<color><param>0000,0000,8000</param>normal

</color>Hi,

(This is a second attempt - the first did not show

up on the if-mib mailing list.)

I want to bring attention to a get-response data staleness problem

that may have implications for if-mib (and possibly other) MIB

specifications.

This problem has been seen recently for ifTable and ifMIB OIDs in

several router HW/SW bases from the same major vendor.  Data

illustrating the problem is provided at the end of this message.  The

level of data staleness seen in this data breaks with many years of

experience.

In summary, ifTable get-responses can contain data that is sometimes

at least 10 seconds old with respect to the sysUpTime.0 returned in

the same response.  In addition, there is a time-granularity effect on

the order of 20-30 seconds for ifHC counters in the ifMIB.  The exact

syndrome seems to be subject to some variation based on the HW/SW

version and options in use.

One consequence is that, in order to keep relative error effects to

desirable levels, rate computations require counter differences taken

over (unreasonably) long time periods.

For instance, in order to bound the relative error of computed In or

Out Octets/S to no more than +/- 1%, based on 32-bit counter

differences, 1000 second time periods are required.  But this exceeds

the time to wrap 32-bit Octet counters on T3 and higher-speed

interfaces.  Going directly to 64-bit ifMIB HC counters is even worse,

likely requiring 2000-3000 second time periods to insure that relative

error effects are << 1%.

These effects greatly interfere with using SNMP data for supporting a

variety of management, operations and provisioning practices in a

manner that is consistent with past practices.  Even data returned

using 60 second polling periods are very noisy.  Furthermore, special

studies that would use high-frequency SNMP polling (for instance,

using snmpdelta with the default 1-second polling period) are rendered

essentially useless.

Ideally, to meet all of these needs, the time-accuracy of returned

data should be at least as good as sysUpTime.0 (one timetick or 10

mS).  Even a one-second granularity can be a problem, as this would

require 100-second periods for keeping computed rate errors under 1%.

One hundred seconds exceeds the ifOctet counter wrap time on OC12s

and remains a very long time for special study purposes.

These recent SNMP data staleness problems raise some issues that may

reflect on the usefulness of implementations of the interfaces MIB

and the ifMIB, and hence may refect on these (and other) MIBs:

1.  Is there anything technically infeasible about implementing the

    existing ifTable and ifMIB for higher speed interfaces that calls

    into question the usefulness of these MIBS as we go forward?

2.  What does the SNMP standards community expect with respect to data

    accuracy in relationship to the time the get-responses are generated?

3.  Do existing standards provide enough guidance regarding the

    desired/required accuracy of data with respect to the time that

    get-responses are generated?

I couldn't find any text in the RFCs I examined that explicitly sets

data timeliness expectations, or that defines procedures which define

unacceptable data staleness for get-responses.  It is not even clear,

from existing text, that sequential get-responses for the same object

should yield values corresponding to non-decreasing times (although

going backwards in time would seem to violate a generic 'usefulness'

criterion).

Is there any IETF work that directly addresses data staleness in

get-responses, particularly with respect to interface statistics?

Thanks,

Carl Holmberg

GTE Internetworking

####################################################################

#  SAMPLE DATA demonstrating 10-30 second time granularity effects

#  (and hence corresponding minimum staleness) in get-response data.

####################################################################

The data below provides an example of the staleness problem.

The ifInOctet and ifOutOctet counter movements exhibit underlying

8-10-second periodicities.  They seem to have multiple feeds (since

they continue to move every second or so).

The 64-bit ifMIB HC counters are clearly being updated from the 32-bit

counters about once every 20 seconds, with no movement between

updates.  Detailed observations show that when the 64-bit counter

jumps, its lower 32 bit portion is very close to, but may lag slightly

behind, the corresponding 32-bit counter.

The movements of ifTable and corresponding ifMIB HC counter are not

exactly synchronized since the number of Counter32 jumps that are

included in each Counter64 jump can vary.  (See record numbers 65 and

127 for cases where three ifInOctet Counter32 jumps are included in

the corresponding Counter64 jump instead of the typical two jumps.)

This data is not an artifact of measurement tools - various methods

show a similar problem.  These data staleness problems first came to

attention when standard tools began reporting excessive interval

utilizations.

# All observations are for the same ifIndex, same device, taken

# sequentially July 8.  Link capacity is approximately 75M Octets/S,

# although individual interval observations may exceed that figure.

#

# The polling period for the data below averages at about one poll per

# 1.8 seconds.  All related counters for the same interface were

# bundled together in single get-requests, including bracketing

# sysUpTime.0 requests (a few packet counters are omitted from the

# data below).  The responses to the first and last sysUpTime.0

# request never differed by more than 1 timetick.  The RTT (including

# polling host service time, using the polling host's 1-second clock)

# averaged << 1 second, and was never more than 3 seconds.

#

#  LEGEND:       (all differences are for this-record less previous)

#

#         Cnt   - Record sequence number

#        ESec   - (Elapsed target sysUpTime from first record)/100.0

#      DTicks   - Difference in target sysUpTime.0

#    DifInOct   - Difference between ifInOctets

#  DifHCInOct   - Difference between ifHCInOctets

#   DifOutOct   - Difference between ifOutOctets

# DifHCOutOct   - Difference between ifHCOutOctets

# Cnt    ESec DTicks  DifInOct   DifHCInOct  DifOutOct  DifHCOutOct

   2     2.44    244  85437296            0  116558817            0

   3     4.91    247      4982            0      59270            0

   4     7.19    228      3007            0       2837            0

   5     9.29    210      1702            0       2500            0

   6    12.50    321  85035162    259200948  115654171    232498433

   7    14.44    194       687            0        767            0

   8    16.80    236      1752            0       1790            0

   9    19.55    275      2383            0  104709036            0

  10    21.77    222  92029508            0     326483            0

  11    24.26    249      2691            0      49600            0

  12    26.92    266      4010            0       9795            0

  13    29.42    250  84921258            0  121995986            0

  14    31.93    251      4073            0     212681            0

  15    34.38    245      3393    176968357      46253    227351435

  16    37.38    300      3143            0       3795            0

  17    39.32    194  85085457            0  107392416            0

  18    41.80    248      2029            0     220953            0

  19    43.27    147      2436            0       2920            0

  20    45.47    220       529            0        696            0

  21    47.59    212  90503438            0   79928249            0

  22    49.96    237      4355            0   37761942            0

  23    52.48    252      6598    175611041      10993    225324748

  24    54.70    222      6266            0      10202            0

  25    56.39    169  85704443            0   77449148            0

  26    58.78    239      2160            0   36586369            0

  27    61.51    273      4196            0      43960            0

  28    63.40    189      5172            0       1593            0

  29    66.28    288 100311296            0   86589212            0

  30    68.14    186     11644            0      52532            0

  31    70.21    207      8689            0   37110899            0

  32    72.19    198     10195    186063819      10824    237854531

  33    74.10    191     12004            0      12338            0

  34    76.22    212  91630849            0  110869966            0

  35    78.03    181      2291            0     280069            0

  36    80.01    198      2341            0       1744            0

  37    81.81    180      2647            0       3088            0

  38    83.40    159      1848            0       1554            0

  39    86.11    271  99181955            0  117320701            0

  40    88.30    219      7298            0     346207            0

  41    90.23    193      2517            0       2777            0

  42    92.00    177      5855            0       2891            0

  43    93.86    186      1950    190850083   68094828    228841842

  44    95.34    148  97447716            0   36521668            0

  45    96.94    160      1614            0     203957            0

  46    98.84    190       874            0       1505            0

  47   100.43    159       806            0       1441            0

  48   102.01    158      6095            0   76610750            0

  49   103.70    169  96765818            0   35070207            0

  50   105.50    180      2746            0     248562            0

  51   107.54    204      2989            0       3215            0

  52   109.20    166      1411            0       1690            0

  53   111.12    192      2226            0       2441            0

  54   112.99    187  90409626    194234452  121980121    338739866

  55   114.68    169      3153            0     349114            0

  56   116.54    186      3537            0      37359            0

  57   118.24    170      2190            0       3083            0

  58   119.83    159      5461            0       5240            0

  59   121.66    183  94402806            0   79246119            0

  60   123.55    189       815            0   36374863            0

  61   125.34    179      1474            0      47267            0

  62   127.40    206      2733            0       3329            0

  63   128.99    159      3629            0       3745            0

  64   130.59    160       962            0  118431836            0

  65   132.74    215  96177384    281013108     229105    234503743

  66   134.23    149       320            0      37457            0

  67   135.77    154      1098            0       1545            0

  68   137.17    140      4157            0       7551            0

  69   138.66    149       749            0        851            0

  70   140.12    146      1119            0  114943117            0

  71   141.70    158 100325589            0       1343            0

  72   143.31    161      1256            0     322583            0

  73   144.78    147      8288            0       1031            0

  74   146.75    197      2838            0      19236            0

  75   148.41    166      3879            0  115058964            0

  76   149.99    158      1795            0       1938            0

  77   151.56    157  97302340            0     249951            0

  78   153.05    149      1125    197654602       1161    230874042

  79   154.58    153      1448            0       1424            0

  80   155.98    140      1244            0       1492            0

  81   157.48    150      2791            0  111263333            0

  82   158.90    142       825            0        535            0

  83   160.30    140  96558808            0       1667            0

  84   161.86    156       422            0     217464            0

  85   163.42    156     13150            0       4032            0

  86   164.86    144     10219            0      19935            0

  87   166.36    150      2999            0  121332176            0

  88   168.01    165     18147            0      11349            0

  89   169.67    166  93169487            0     191008            0

  90   171.30    163      3246            0      33860            0

  91   173.02    172      1231    189783756       1131    233079045

  92   174.42    140      2017            0       2098            0

  93   175.88    146      2620            0  110537943            0

  94   177.38    150  98115993            0       1181            0

  95   179.14    176      2582            0       2719            0

  96   180.89    175      2700            0     259126            0

  97   182.54    165      3755            0       2828            0

  98   184.29    175      2927            0  122625366            0

  99   186.51    222  86854665            0       6494            0

 100   188.34    183      3105            0      17534            0

 101   190.36    202      2670            0     682148            0

 102   191.93    157      1944            0       2050            0

 103   193.40    147      2813    184995853   36074394    234140370

 104   194.94    154      1985            0   90957206            0

 105   196.35    141  96209791            0        702            0

 106   197.77    142      2388            0     592704            0

 107   199.19    142       890            0        921            0

 108   200.94    175      1756            0       1792            0

 109   202.58    164      2817            0       1879            0

 110   204.09    151      1077            0  123095940            0

 111   206.46    237  83485766            0      39570            0

 112   208.44    198      4602            0     540079            0

 113   209.95    151      5148            0       5532            0

 114   211.50    155      2546            0       2758            0

 115   213.08    158      1875    179722379  118856631    370169677

 116   214.72    164  88197567            0      51563            0

 117   216.31    159      2926            0     830304            0

 118   218.22    191      1020            0       1120            0

 119   219.86    164       789            0        565            0

 120   221.42    156       661            0  113695354            0

 121   223.06    164  87165089            0       2209            0

 122   224.56    150       529            0      43557            0

 123   226.15    159      2336            0     851661            0

 124   227.89    174       696            0        733            0

 125   229.71    182       534            0        871            0

 126   231.63    192  95477966            0  117173867            0

 127   233.44    181      1854    270851302      34286    232652395

 128   235.26    182      6310            0     605218            0

 129   236.89    163      3643            0       3073            0

 130   238.70    181      6263            0       1497            0

 131   240.57    187      4690            0  117734972            0

 132   242.19    162 105011921            0       3777            0

 133   244.01    182      2495            0     620288            0

 134   245.72    171      1791            0       4408            0

 135   247.50    178      2730            0       2219            0

 136   249.33    183      3665            0  115229432            0

 137   251.27    194 106923176            0       7698            0

 138   252.98    171      2478    211970434     604878    234248030

 139   254.46    148       878            0       2490            0

 140   256.24    178      1698            0       2993            0

 141   258.26    202      2351            0   81365388            0

 142   260.22    196      1748            0   36698037            0

 143   262.14    192  97213174            0     519684            0

 144   263.95    181      3730            0       3404            0

 145   265.66    171      2505            0   75034576            0

 146   267.36    170      2188            0       2433            0

 147   269.15    179  89330910            0   33440217            0

 148   271.08    193      5229            0       5857            0

 149   272.86    178      2109    186565192     658239    228330775

---

From: "David T. Perkins" <[email-masked]>
Date: Mon, 23 Aug 1999 11:47:48 -0700

HI,

The ammount of "staleness" that is acceptable depends on the use of the
data. For many applications a "staleness" of upto 15 minutes is workable.

For your usage, a 10 second "staleness" doesn't seem to be workable.

This issue has not really been discussed before that I can remember,
except possibly when working on the RFCs containing the DS1 and DS3
MIB modules.

Regards,
/david t. perkins

---

From: sdurham <[email-masked]>
Date: Mon, 23 Aug 1999 13:21:49 -0700

I can't speak for the working group, but i can toss in my own 2 cents on the
issue :

Given the increasing linespeeds, and increasing dependence on ifmib counters, i
would say a time discrepancy between ifInOctets.x and ifInUcastPkts.x for
instance, would be a bad thing.  All counters on the same interface should be
consistant.  A correlary to this would suggest all subinterface counters on
said interface should all be at the same time instant.

But does this mean they must be accurate at the value of sysUpTime reported w/
the response pdu?  My take would be no.  They need to be consistant with
themselves, and perchance a new timestamp object would help, to know when the
data snapshot was taken.

From there, what would be an acceptable delta between consecutive data
snapshots? This would be a case wherein valid deltas may depend on linespeed. 
For instance, a 10second delay between snapshots may not be a big issue for a
slow serial link.  But using that same value with an OC48 is something the
user/customer should consider unacceptable.

> and remains a very long time for special study purposes. 

one confusion i have here - why are you using ifxOctet counters on an OC12
link?  shouldn't this be using the HC counters, and thus avoid the wrap issue
w/ a one second counter update, and 100 sec nms polling period?  (assuming of
course the HC counters behave better than the ones you've shown below).

>     into question the usefulness of these MIBS as we go forward? 

afaik, no, there isn't a technical problem here.  Rather it all suggests
design/cost tradeoffs.  For instance - it takes less cpu drag to gather counter
stats if you have a 10 second update period, vs a 10ms update period.  But i
don't see anything here that better hardware/software couldn't solve, if the
vendors are willing to put $$ into it.

What worries me abit more, and i haven't done a lick of math to see if this is
just a foolish nightmare on my part - but what happens when your interfaces are
so fast that you can't possibly do an snmp poll quickly enough to gather timely
stats, and catch errors before you've dropped millions of packets on the
floor.  That's the breakdown point i worry about.  The earlier problem is
easily solved by forking out the $$ to slap 64bit hardware counters on the
interface source.  But what happens when you hit the performance wall of a
generic snmp engine, and you just can't pass pdus around fast enough for the
stats you want to gather in a timely manner?  ...just alittel something that
wakes me up at 3am sometimes...

sandra

---

From: "C. M. Heard/VVNET, Inc." <[email-masked]>
Date: Thu, 26 Aug 1999 21:55:12 -0700 (PDT)

On Mon, 23 Aug 1999, sdurham wrote:

> design/cost tradeoffs.  For instance - it takes less cpu drag to gather
> counter stats if you have a 10 second update period, vs a 10ms update period.
> But i don't see anything here that better hardware/software couldn't solve,
> if the vendors are willing to put $$ into it.

I agree with this.  The excessive staleness problem that Mr. Holmberg has
observed is a problem with the implementation, not with the MIB design
or with SNMP.  In the product I'm working on, which is an SNMP-controlled
test set, we snapshot interface-related counters every 10 msec and have
taken pains to ensure that all the counters returned in response to a
single SNMP request always come from the same snapshot.  Granted, this
can be expensive in terms of CPU time, and may not be justified in all
applications;  but it is not impossible to achieve.

Mike
--
C. M. Heard/VVNET, Inc.
[email-masked]