Subject: Re: [Netconf] LC on subscribed-notifications-10

From: "Eric Voit (evoit)" <[email-masked]>
Date: Tue, 3 Jul 2018 21:04:21 +0000

<Eric13>

From: Kent Watsen, June 28, 2018 11:19 AM

Please look for <Kent12> below.

 <Kent6> okay, I think I got it this time.  Having a *configurable* replay-start-time is so confusing.  Is it really worth having?

<Eric7>   Yes it is worth having.

(a) In many environments, reboot is very infrequent.  Without configurable start time, an operator setting up a configured subscription would not have the ability to designate what to send.  It could only send the full log (at whatever size).

(b) on-publisher security or troubleshooting diagnostics might identify a breach or some event where streaming recent historical event records is a MUST.  As a result, it might want to stream a subset of event records off a box going back in time to potential events which might have been evidence or contributing factors.

<Kent7> Let me come at this another way.  Assume we drop all support for *configurable* replay-start-time.  As such, configured subscriptions always start with the next-generated event (no replay at all).   This covers most use-cases, right?   For those receivers that really wanted the older logs, can't they just do a dynamic subscription to collect them, same as we've been discussing above?

<Eric8> Some reasons this might not always be practical:

(a) IoT devices just might want to passively listen to event streams of Telemetry.  (I.e., this would force configured receivers to support dynamic subscriptions.)

(b) This forces complexity onto applications which only ever need to track what has happened since boot.  (E.g., per above, continuous Integrity Measurement Architecture (IMA) boot log streaming and evaluation.)

(c) Publisher access permissions for who can use the establish-subscription RPC might have to be expanded to include lots of configured receivers.  This might open up a vector to control plane DDoS.  Right now the access permissions would just have to allow the receiver read access to the event records.

(d) A publisher may choose to firewall classes of receivers (or locations of receivers) into a listen-only mode without the ability to establish subscriptions.

<Kent8> This response seems to address the "can't they just do a dynamic subscription" aspect of my comment, but doesn't really address the "why is it important" (I paraphrase) part.  My contention is that the concept of a *configurable* replay-start-time seems confusing and of low value.   I acknowledge that there is some value, but it seems like the value is limited to a one-time start-up optimization that can be alternatively addressed by a dynamic subscription to fetch earlier events (assuming it's allowed, per your points b-d).   Additionally, FWIW, I've never seen such a feature implemented before, and logging mechanisms have been around for decades, so this makes me think that this is something that probably isn't worth having.

<Eric9> As you point out, the why "can't they just do a dynamic subscription" is covered, and we shouldn’t always assume away (b)-(d) as they can matter in some scenarios.  So if we want to support the use case of streaming log entries made after boot, but before the transport session is available, the only alternative I see is to have a configured replay-flag rather than a configuring a start-time.  Are you ok with a flag instead?  Or do you have an alternative suggestion?

<Kent9> see below.

In terms of using this configured replay capability, Cisco’s Integrity Verification application

https://www.cisco.com/c/dam/en/us/td/docs/cloud-systems-management/application-policy-infrastructure-controller-enterprise-module/1-5-x/integrity_verification/user-guide/Cisco_Integrity_Verification_Application_APIC-EM_User_Guide_1_5_0_x.pdf<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cisco.com_c_dam_en_us_td_docs_cloud-2Dsystems-2Dmanagement_application-2Dpolicy-2Dinfrastructure-2Dcontroller-2Denterprise-2Dmodule_1-2D5-2Dx_integrity-5Fverification_user-2Dguide_Cisco-5FIntegrity-5FVerification-5FApplication-5FAPIC-2DEM-5FUser-5FGuide-5F1-5F5-5F0-5Fx.pdf&d=DwMGaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=YLzifR1978kb_hHj64ZtYbrlHE2fJaofeSKu9OAFQXg&s=Vc8m5WAJJE8YkQIpZuxlnVTgAtVKQZ-n0dyoRKX3Eao&e=>

does do a shell access event log fetch of the full log after boot, and then just does incremental fetch the deltas of the log (based on log line numbers).  This application is interested in configured subscriptions subsequent to boot for this purpose.  So such incremental streaming of portions of syslog after boot seems like a typical/common need to me.

<Kent9> it might be typical/common desire, but it's still once in the lifetime of the configured subscription.  It seems like, if the device supports dynamic subscriptions, after receiving subscription-started, the client could a) pause the configured subscription, b) use a dynamic subscript to fetch the missing logs, and then c) resume the flow of logs from the configured subscriptions.

<Eric10> Your proposal still precludes (b)-(d) above.   In addition for your step a), there is no RPC or action which allows the event records from a configured (or dynamic) subscription to be paused.  The solution also adds complexity into the client to recognize that early events might be missing, to issue an establish-subscription, and then to tie the results of the independent subscriptions together.

<Kent10> pausing can be implemented by the receiver not reading any more from the TCP socket, or something else.

<Eric11> There is no mechanism for a receiver to pause a single subscription without pausing other subscriptions on the TCP session (as subscriptions typically would share a common TCP.)

<Kent11> Different "receivers" of different configured subscriptions pointing to the same underlying netconf or restconf call-home connection?

<Eric12> Yes

<Kent12> Ack.  So, *if* we were to do this, the client would either have to pause all the subscriptions, or do a dynamic fetch in parallel.  Hmmm, given that we're talking about the *configured* replay-start-time, which kicks in after a reboot, all the subscriptions would be restarted simultaneously (right?), so maybe this isn't a big issue?

<Eric13> Per this thread, this is now a configured-replay empty object (rather than a start-time).  This empty object simply tells the publisher to push off all events retained from a stream since reboot.  And yes, all configured subscriptions will restart simultaneously.  But without this feature, receivers will not see events from boot to transport establishment.  And without this feature, two different configured receivers might get different initial events as the transport might not be brought up simultaneously.

How is it any more complex for the client/receiver than the following in the SN draft already?

   When a receiver of a configured subscription gets a new

   "subscription-started" message for a known subscription where it is

   already consuming events, the receiver SHOULD retrieve any event

   records generated since the last event record was received.  This can

   be accomplish by establishing a separate dynamic replay subscription

   with the same filtering criteria with the publisher", assuming the

   publisher supports the "replay" feature.

<Eric11> It is the same general process.  But it turns the SHOULD into a MUST for applications which need to know the events since boot.  It also doesn’t deliver the events in order to the application, delaying application event analysis.

<Kent11> here's another question that might be good to raise to the WG level.   Please be sure to capture my general concern and also the availability of this workaround.  Thanks.

<Eric12>  You are welcome to take the question to the WG level.  I have no desire to waste people’s time with such an obvious question:

- The current solution does not add the extra complexity described above for configured subscription replay.

- The current solution supports deployment scenarios (b)-(d) above.

- The current solution has far less implementation complexity and error reconciliation states for the client.

<Kent12> No Eric, you are the Editor.  We can take this to Montreal if you prefer.   We need more opinions to break the standoff, and I don't think folks are watching this thread.  Please try to present the tradeoffs in a fair manner.   BTW, a==b and c==d, AFAICT.  a/b seems true and c/d also seems true, but 1) it is already a SHOULD for the receiver to do a dynamic fetch for already started subscriptions (see quoted paragraph above), so having another SHOULD for newly started configured subscription doesn't threaten of a-d any more than already, 2) it seems really weird to have persistent configuration that only gets used once in the lifetime of a configured subscription, 3) it is good to remove frivolous features, and 4) the current text in draft is confusing about replay-start-time.   #4 is what started this fork in the thread, but rather than fix the text, I'm thinking it might be better to remove the feature.

<Eric13> Montreal it is.

Eric

Kent  // contributor

---

From: Kent Watsen <[email-masked]>
Date: Thu, 5 Jul 2018 22:19:15 +0000

<Kent13>

<Eric12> Yes

<Kent13> I'm unsure how this addresses my concern that replaying is unnecessary for configured subscriptions…I somehow thought that it might be related since it's listed here.   Separately, I'm unclear about what this change does, I know you provide some explanation above, but you can say it differently or provide an example or two to illustrate what you mean?   Does this change introduce the problem you mentioned before about duplicates events being sent?  BTW, the description statement on replay-start-time seems incorrect, with the node now being config false, how can it be used to "trigger" anything?

Kent // contributer

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Fri, 6 Jul 2018 00:06:59 +0000

<Eric14>

From: Kent Watsen, July 5, 2018 6:19 PM

<Eric12> Yes

<Kent13> I'm unsure how this addresses my concern that replaying is unnecessary for configured subscriptions…I somehow thought that it might be related since it's listed here.   Separately, I'm unclear about what this change does, I know you provide some explanation above, but you can say it differently or provide an example or two to illustrate what you mean?   Does this change introduce the problem you mentioned before about duplicates events being sent?

<Eric14>  Here is a link to the slides which I have put together for IETF 102 which hopefully hits your request above:

https://github.com/netconf-wg/rfc5277bis/blob/master/draft-configured-replay-slides-ietf102.pdf

BTW, the description statement on replay-start-time seems incorrect, with the node now being config false, how can it be used to "trigger" anything?

<Eric14>  replay-start-time exists as part of the establish-subscription RPC.   The RPC is where the triggering occurs.

Eric

Kent // contributer

---

From: Kent Watsen <[email-masked]>
Date: Fri, 6 Jul 2018 21:04:42 +0000

<Kent14>

 

<Kent14> not really.  The switch to "configured-replay" seems orthogonal, likely due to it being perceived as easier than specifying an exact time.  Separately, as a comment on the slides, perhaps list both Pros and Cons?  I provided 4 perceived benefits in a recent email…

<Kent14> right, and that's fine, but it's not okay for configured subscriptions.   Subscription-policy uses subscription-policy-dynamic, the description needs to be refined, like it was for the subscription-started notification.

Kent // contributer

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Fri, 6 Jul 2018 23:13:45 +0000

> From: Kent Watsen, July 6, 2018 5:05 PM
> 
> <Kent14>
> 
> <Eric13> Per this thread, this is now a configured-replay empty object (rather
> than a start-time).  This empty object simply tells the publisher to push off all
> events retained from a stream since reboot.  And yes, all configured
> subscriptions will restart simultaneously.  But without this feature, receivers
> will not see events from boot to transport establishment.  And without this
> feature, two different configured receivers might get different initial events as
> the transport might not be brought up simultaneously.
> 
> <Kent13> I'm unsure how this addresses my concern that replaying is
> unnecessary for configured subscriptions…I somehow thought that it might be
> related since it's listed here.   Separately, I'm unclear about what this change
> does, I know you provide some explanation above, but you can say it
> differently or provide an example or two to illustrate what you mean?   Does
> this change introduce the problem you mentioned before about duplicates
> events being sent?
> 
> <Kent14> not really.  The switch to "configured-replay" seems orthogonal,
> likely due to it being perceived as easier than specifying an exact time.

<Eric15> It is a flag.   It just means that the replay starts with the earliest event after reboot.

> Separately, as a comment on the slides, perhaps list both Pros and Cons?  

I put PRO and CON on slide 2 of:
https://github.com/netconf-wg/rfc5277bis/blob/master/draft-configured-replay-slides-ietf102.pdf 
If you disagree with any bullet, let me know.

> I provided 4 perceived benefits in a recent email…

<Eric15>  Pulling the four items which I believe you are referring to:

Kent #1: it is already a SHOULD for the receiver to do a dynamic fetch for already started subscriptions (see quoted paragraph above), so having another SHOULD for newly started configured subscription doesn't threaten of a-d any more than already, 
Eric response #1: Please reread the provided slide.  I tried to reframe (a)-(d) and provide a summary all the other points from our long thread.  For example, asserting that the two SHOULDs can be safely coupled is incorrect.  Per my Option 2, bullet 4, sub-bullet 2, can be are cases where dynamic subscriptions are not needed on the receiver.  

Kent #2: it seems really weird to have persistent configuration that only gets used once in the lifetime of a configured subscription
Eric response #2: This is not a benefit.   Beyond that, a flag that say when to begin is not a bad thing.  And there is plenty of configuration which is used only once. E.g., stop-time, synch-on-start.

(Kent #3) it is good to remove frivolous features
Eric response #3a:  It is optional to implement.  If someone doesn't want to implement it, they don't have to.  
Eric response #3b:  This is not frivolous.  In fact my personal preference would be to have the start time for a configured subscription be at the known/consistent/anchored boot time.  The default right now is that the start time varies based on the point of first transport establishment -> and this will vary by receiver.  Which means independent receivers of a single subscription may have a different first event.   Sadly this default cannot be change to my preference because replay is an optional feature.   
Eric response #3c: As security people want this behavior, they will be forced to attempt Option 2 should Option 1 not exist.  Which has all the CON drawbacks described on the slide.  

(Kent #4) the current text in draft is confusing about replay-start-time.  
Eric response #4:  This is not a benefit.  And the text has been updated.

> BTW, the description statement on replay-start-time seems incorrect, with the
> node now being config false, how can it be used to "trigger" anything?
> 
> <Kent14> right, and that's fine, but it's not okay for configured subscriptions.
> Subscription-policy uses subscription-policy-dynamic, the description needs to
> be refined, like it was for the subscription-started notification.

<Eric 15> Replay-start-time is config false.  Therefore it will not appear populated for a configured subscription in the data nodes.  (Note: What I did do is delete 'replay-start-time' from the refined definition, as that option no longer exists.)

Eric