Subject: [OPSAWG] discuss on draft-ietf-opsawg-syslog-msg-mib-06.txt

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 28 Aug 2009 15:06:15 +0200

Hi,

during the IESG review of draft-ietf-opsawg-syslog-msg-mib-06.txt,
Adrian Farrel raised a DISCUSS concerning the syslogMsgTable that we
have not managed to clear yet and where we seek input from the WG.

The syslogMsgTable is index by syslogMsgIndex, an unsigned index
number that is increases when entries are added and which rolls over
if the index number space has been exceeded. This means that the
entries in the table are (ignoring roll overs here) in the order they
were inserted (which is not necessarily the timestamp order of the
syslog events). This supports the following "tail -f" use case where a
management application initially reads the table, remembers the end of
the table and during the next poll starts at the last remembered index
position. This is useful to quickly check for additions to the
table. This approach also works with index roll overs unless the table
rolls over multiple times during a polling cycle.

Adrian Farrel asks in his DISCUSS whether there is not also a use case
to read the table in index order, that is oldest table entry to most
recent table entry, according to table insertion order. Right now,
this only works until the first index rollover occurs since after the
first index rollover, top table entries might be more recent than
bottom table entries. This can be fixed for example by adding an
additional scalar object reporting the index of the oldest entry in
the table.

As the editor of the document, I did not feel entitled to make a
decision on this issue. After talking with Scott Bradner, we believe
the WG needs to decide whether the ability to quickly find the oldest
entry in the table (according to the insertion time) is a feature the
MIB module needs to support. An important detail to consider here is
that the table insertion order is not necessarily syslog timestamp
order.

/js

PS: Adrian, please correct me if my summary of your DISCUSS is
    incorrect or unclear.

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>

---

From: WashamFan <[email-masked]>
Date: Fri, 28 Aug 2009 22:05:02 +0800

Hi,

----- Original Message -----
From: Juergen Schoenwaelder <[email-masked]>
Date: Friday, August 28, 2009 9:07 pm
Subject: [OPSAWG] discuss on draft-ietf-opsawg-syslog-msg-mib-06.txt
To: [email-masked]
Cc: Adrian Farrel <[email-masked]>

>  additional scalar object reporting the index of the oldest entry in

How about introduce a column as a rollover counter to the table?

washam
 
>  Fax:   +49 421 200 3103         <
>  _______________________________________________
>  OPSAWG mailing list
>  [email-masked]
>  https://www.ietf.org/mailman/listinfo/opsawg
>

---

From: "David Harrington" <[email-masked]>
Date: Fri, 28 Aug 2009 10:59:37 -0400

Hi,

The SNMP approach has a major guideline:
keep the agent simple; move complexity to the manager.

I think that different applications will want different orderings, and
that the applications should sort based on their preference rather
than forcing this onto the agent. I do not believe there is
significant operational benefit from knowing insertion order.

Do entries ever get removed from this table? Does rollover just
overwrite starting at the beginning of the table? Is there ever a case
where the newer entries get interlaced with older entries?

I am not sure I see the benefits of knowing where the oldest entry is.
I think most uses will want the whole table so they can sort it by
whatever criteria it needs. I can see a few edge cases (like having
oldest=0 might imply no rollover has occurred), but nothing
compelling. In the case of wanting to only get updated rows, I would
think knowing where the most recent entry is might be more useful than
knowing where the oldest entry is, so you know where to stop reading,
even if there has been no rollover. This seems to add complexity to
the agent that might not be needed. The manager can simply look at
each getNext retrieval to see if it already has that entry. But a
scalar is fairly simple to add, so I wouldn't object if it got added.

dbh

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Juergen Schoenwaelder
> Sent: Friday, August 28, 2009 9:06 AM
> To: [email-masked]
> Cc: Adrian Farrel

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 28 Aug 2009 18:35:11 +0200

> How about introduce a column as a rollover counter to the table?

To solve which problem? It does not give me the oldest entry without
reading the table - so no improvement related to Adrian Farrel's
discuss.

/js

---

From: "David Harrington" <[email-masked]>
Date: Fri, 28 Aug 2009 20:02:19 -0400

Hi, 

> 2.
> Yes. And this is what swings it for me.
> I am just left wondering whether an Agent is more loaded by a 
> bulk read of 
> 10,000 entries or maintaining a pointer into the table.
> 
> Clear up this last point, and I'm done.
> 
> Cheers,
> Adrian

Well, I think you have two cases - the initial read and update reads.
I might be surprised to see 10,000 in an initial read, but I
presumably want to get them all on the initial read, so if there are
10,000, then that's what I'll get, whether there is a scalar or not.
Knowing the oldest entry doesn't really help me on the initial read.

Now we'll deal with update reads. I would be surprised to see 10,000
entries in an update. But presuming they're there and I want them all,
I can bulk read them at 100 each time and check for whether I have
duplicates of entries I retrieved earlier (or whether the timestamp
changes significantly or whatever optimization I meet choose to use).
I can stop requesting more once I detect a duplicate. If the table is
actually that large that there are 10,000 update entries in the table,
a bulk read of 100 does not seem like much of a problem, and it would
be nicer to the CPU of the agent device. Worst case, I read 100
duplicate entries, but compared to an update of 10,000 entries I think
this is insignificant. How complex you want to make the algorithm can
be decided by the manager.

I am not certain how a scalar to the oldest helps that much. A pointer
to the latest entry would seem more useful to recognize where to stop.
Of course, the pointer to the oldest is probably 1 greater than the
pointer to the newest. I just don't think it makes that much
difference if you are talking 10,000 entries in an update. 

As I said I don't object to having a scalar. But I don't think it's
going to matter much in your scenario. If you're really concerned
about the load on the agent, remember that if there are 10000 update
entries, the scalar needs to be updated 10,000 times. And for the
manager to know the value of the scalar, it will have to do reads of
the scalar as well as of the table entries.

If you really want to reduce the load from bulk reads, then you might
want to provide filtering INDEXES on common fields, such as facility
(only get the ones for the email server), and severity (only retrieve
the critical errors). I think this might be more useful at reducing
retrieval load than the proposed scalar because these are commonly
used filters.

dbh

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 29 Aug 2009 20:41:49 +0200

On Fri, Aug 28, 2009 at 11:49:50PM +0200, Adrian Farrel wrote:
 
> Two minor points:
> 
> 1.
> > Do entries ever get removed from this table?
> Not as I understand it. The index is monotonic increasing and wraps at the 
> maximum value (defined as the lesser of syslogMsgTableMaxSize and 
> 4294967295).

No, it wraps at 4294967295 always. What makes you believe it maps at
min(syslogMsgTableMaxSize, 4294967295)?

> > Does rollover just overwrite starting at the beginning of the table?
> Yes.

Only in the case where the agent has storage for 4294967295 table
entries, something I consider unlikely implementations will support.

> I am just left wondering whether an Agent is more loaded by a bulk read of 

If you only need the oldest entry in table insertion time, such an
index is of course handy. The question is what practical value it has
to be able to retrieve this oldest entry quickly.

/js

---

From: Adrian Farrel <[email-masked]>
Date: Fri, 28 Aug 2009 22:49:50 +0100

David,

Thanks for that input. It almost closes the issue for me.

Two minor points:

1.
Yes.
> Is there ever a case where the newer entries get interlaced with older 
> entries?
Not unless someone screws up.

2.

Cheers,
Adrian

----- Original Message ----- 
From: "David Harrington" <[email-masked]>
To: "'Juergen Schoenwaelder'" <[email-masked]>; 
[email-masked]
Cc: "'Adrian Farrel'" <[email-masked]>
Sent: Friday, August 28, 2009 3:59 PM
Subject: RE: [OPSAWG] discuss on draft-ietf-opsawg-syslog-msg-mib-06.txt

---

From: Adrian Farrel <[email-masked]>
Date: Sat, 29 Aug 2009 19:55:26 +0100

> min(syslogMsgTableMaxSize, 4294967295)?

I'm very sorry. When I wrote this I went back to check on the email thread, 
and copied from my original question and not from your answer. (That's a 
poor excuse, I know.)

So, this impacts Dave's question; Entries *do* get deleted from the table as 
the valid index window slides upwards.

> to be able to retrieve this oldest entry quickly.

If we agree that the management station should read and store the table, 
parse it, and then do whatever it wants to. I agree: none.

Adrian