Subject: [Dots] Telemetry draft: Vendor Specific data reduction

From: "Jon Shallow" <[email-masked]>
Date: Thu, 23 Apr 2020 10:48:51 +0100

Hi All,

When passing telemetry attack information back and forth, there are some
ways that we need to consider on data reduction, thus reducing the
likelihood of having to do Block transfers.

My understanding is that there is a one-to-one relationship between
"attack-id" and "attack-name".

My first suggestion is that the client is able to upload to a server, and
the server can download on request, a vendor's mapping of "attack-id" to
"attack-name" for the specific vendor "id".  Then, whenever there is
telemetry information "id" + "attack-id" need to be provided, but much space
can be saved by not having to also include "attack-name".

Second suggestion is that "attack-id" is an integer instead of a string to
again save on space in the telemetry data.

Regards

Jon

---

From: "Jon Shallow" <[email-masked]>
Date: Tue, 28 Apr 2020 09:29:27 +0100

Hi All,

Any thoughts on this data reduction?

While it is possible for a Vendor to come up with their own augmented YANG
to cover their vendor specifics, it gets problematic when 2 or more Vendor
specifics need to be understood by a client or a server.

Having a "/vendor-mapping" operation path means that vendor mapping of
"attack-id" and "attack-name" can easily be exchanged.

If "attack-id" is an integer instead of a string, then "attack-id" could
become a Vendor specific set of enums (they do not need to start from 1)
based on "id".

Regards

Jon

From: Dots [mailto: [email-masked]] On Behalf Of Jon Shallow
Sent: 23 April 2020 10:49
To: [email-masked]
Subject: [Dots] Telemetry draft: Vendor Specific data reduction

Hi All,

Regards

Jon

---

From: <[email-masked]>
Date: Tue, 28 Apr 2020 11:00:23 +0000

Hi Jon,

Apologies for the delay to follow on this one.

attack-name is more about a description than a name. This field may also be used to map attack details from distinct vendors because there is no a global registry (and we don't want to create one).

Having the ability to retrieve a list prior to an attack is interesting to consider but the (optional) attribute may still be needed to be included for new attack types.

Note that the name attribute is also used by a DOTS client to send telemetry to a DOTS server.

Attack-id is defined as a string because we inspired from existing event notification formats. Some of these formats allow for an even ID to be integer or string.

Cheers,
Med

De : Dots [mailto:[email-masked]] De la part de Jon Shallow
Envoyé : mardi 28 avril 2020 10:29
À : [email-masked]
Objet : Re: [Dots] Telemetry draft: Vendor Specific data reduction

Hi All,

While it is possible for a Vendor to come up with their own augmented YANG to cover their vendor specifics, it gets problematic when 2 or more Vendor specifics need to be understood by a client or a server.

Having a "/vendor-mapping" operation path means that vendor mapping of "attack-id" and "attack-name" can easily be exchanged.

If "attack-id" is an integer instead of a string, then "attack-id" could become a Vendor specific set of enums (they do not need to start from 1) based on "id".

Regards

Jon

Hi All,

When passing telemetry attack information back and forth, there are some ways that we need to consider on data reduction, thus reducing the likelihood of having to do Block transfers.

My understanding is that there is a one-to-one relationship between "attack-id" and "attack-name".

My first suggestion is that the client is able to upload to a server, and the server can download on request, a vendor's mapping of "attack-id" to "attack-name" for the specific vendor "id".  Then, whenever there is telemetry information "id" + "attack-id" need to be provided, but much space can be saved by not having to also include "attack-name".

Second suggestion is that "attack-id" is an integer instead of a string to again save on space in the telemetry data.

Regards

Jon

---

From: "Jon Shallow" <[email-masked]>
Date: Tue, 28 Apr 2020 16:15:45 +0100

Hi Med,

To give an example for “attack-id” and “attack-name” the DDOS Mitigator that
I work with has several components that identify the same attack

Index: 3016

Short-Name: tcpattack_synflood

Descriptive-Name: “TCP Attack - Syn Flood"

And in the code I was using the index for “attack-id” and the
Descriptive-Name for the “attack-name” for the recent telemetry Interop with
Kaname.

With a multi-vector attack, the descriptive name information was a
substantive part of the telemetry information being passed back to the
client.

Similarly, if the DDoS Mitigator was to act as a client to an upstream DOTS
server (which in my case it can), then again there is a lot of information
being relayed that could be reduced with a mapping between” attack-id” and
“attack-name” for a specific vendor “id” being uploaded ahead of time – and
can be refreshed if new attacks are discovered and mitigated.

Regards

Jon

From: Dots [mailto: [email-masked]] On Behalf Of
[email-masked]
Sent: 28 April 2020 12:00
To: Jon Shallow; [email-masked]
Subject: Re: [Dots] Telemetry draft: Vendor Specific data reduction

Hi Jon,

attack-name is more about a description than a name. This field may also be
used to map attack details from distinct vendors because there is no a
global registry (and we don’t want to create one). 

Having the ability to retrieve a list prior to an attack is interesting to
consider but the (optional) attribute may still be needed to be included for
new attack types. 

Note that the name attribute is also used by a DOTS client to send telemetry
to a DOTS server. 

Attack-id is defined as a string because we inspired from existing event
notification formats. Some of these formats allow for an even ID to be
integer or string. 

Cheers,

Med

Hi All,

Having a “/vendor-mapping” operation path means that vendor mapping of
“attack-id” and “attack-name” can easily be exchanged.

If “attack-id” is an integer instead of a string, then “attack-id” could
become a Vendor specific set of enums (they do not need to start from 1)
based on “id”.

Regards

Jon

Hi All,

My understanding is that there is a one-to-one relationship between
“attack-id” and “attack-name”.

My first suggestion is that the client is able to upload to a server, and
the server can download on request, a vendor’s mapping of “attack-id” to
“attack-name” for the specific vendor “id”.  Then, whenever there is
telemetry information ”id” + “attack-id” need to be provided, but much space
can be saved by not having to also include “attack-name”.

Second suggestion is that “attack-id” is an integer instead of a string to
again save on space in the telemetry data.

Regards

Jon

---

From: <[email-masked]>
Date: Tue, 28 Apr 2020 15:56:42 +0000

Re-,

If attack-name is completely removed for the attack-details, this means the remote peer can't make use of the information till the list is refreshed.

Isn't better to maintain the attribute as in the current design but an agent uses this attribute only for new attacks?

Cheers,
Med

De : Jon Shallow [mailto:[email-masked]]
Envoyé : mardi 28 avril 2020 17:16
À : BOUCADAIR Mohamed TGI/OLN; [email-masked]
Objet : RE: [Dots] Telemetry draft: Vendor Specific data reduction

Hi Med,

To give an example for "attack-id" and "attack-name" the DDOS Mitigator that I work with has several components that identify the same attack

Index: 3016
Short-Name: tcpattack_synflood
Descriptive-Name: "TCP Attack - Syn Flood"

And in the code I was using the index for "attack-id" and the Descriptive-Name for the "attack-name" for the recent telemetry Interop with Kaname.

With a multi-vector attack, the descriptive name information was a substantive part of the telemetry information being passed back to the client.

Similarly, if the DDoS Mitigator was to act as a client to an upstream DOTS server (which in my case it can), then again there is a lot of information being relayed that could be reduced with a mapping between" attack-id" and "attack-name" for a specific vendor "id" being uploaded ahead of time - and can be refreshed if new attacks are discovered and mitigated.

Regards

Jon

From: Dots [mailto: [email-masked]] On Behalf Of [email-masked]
Sent: 28 April 2020 12:00
To: Jon Shallow; [email-masked]
Subject: Re: [Dots] Telemetry draft: Vendor Specific data reduction

Hi Jon,

Cheers,
Med

Hi All,

Regards

Jon

Hi All,

Regards

Jon

---

From: "Jon Shallow" <[email-masked]>
Date: Tue, 28 Apr 2020 20:24:05 +0100

Hi Med,

I was not thinking of dropping attack-name.  I was more thinking of
uploading / downloading as appropriate the entire list of
id->attack-id<->attack-name.  The attack-id<->attack-name list will evolve
over time so there will need to be a mechanism of doing updates, or
signalling that there are new attack-id in use.

However, the first time that an attack-id is used, the attack-name can also
be included.  Two immediate issues that come to mind are 

“What happens if the telemetry information packet that contains the
attack-name gets lost in transit”

“If a major attack kicks off with many vectors, all of the attack-ids for
the first time, there will be a lot of traffic”

Regards

Jon

Sent: 28 April 2020 16:57

Re-, 

If attack-name is completely removed for the attack-details, this means the
remote peer can’t make use of the information till the list is refreshed. 

Isn’t better to maintain the attribute as in the current design but an agent
uses this attribute only for new attacks? 

Cheers,

Med

Hi Med,

Index: 3016

Regards

Jon

Hi Jon,

Cheers,

Med

Hi All,

Regards

Jon

Hi All,

Regards

Jon

---

From: <[email-masked]>
Date: Wed, 29 Apr 2020 08:53:18 +0000

Hi Jon,

Please see inline.

Cheers,
Med

De : Jon Shallow [mailto:[email-masked]]
Envoyé : mardi 28 avril 2020 21:24
À : BOUCADAIR Mohamed TGI/OLN; [email-masked]
Objet : RE: [Dots] Telemetry draft: Vendor Specific data reduction

Hi Med,

I was not thinking of dropping attack-name.  I was more thinking of uploading / downloading as appropriate the entire list of id->attack-id<->attack-name.  The attack-id<->attack-name list will evolve over time so there will need to be a mechanism of doing updates, or signalling that there are new attack-id in use.
[Med] I understood that. I was commenting on the implication on the module.

Putting aside defining attack-id as an integer, I thought your proposal would be as follows (with removing attack-name from the attack-detail).

    +--:(vendor-mapping) {dots-telemetry}?
    |  +--rw attack-detail* [attack-id]
    |     +--rw vendor-id?     uint32
    |     +--rw attack-id      string
    |     +--rw attack-name    string
    +--:(telemetry) {dots-telemetry}?
       +--rw pre-or-ongoing-mitigation* [cuid tmid]
          ...
          +--rw attack-detail* [attack-id]
             +--rw vendor-id?         uint32
             +--rw attack-id          string
             +--rw attack-severity?   attack-severity
             +--rw start-time?        uint64
             +--rw end-time?          uint64
             ...

I was suggesting to leave the name under attack-detail but use it only when an attack-id/name mapping is not already shared with the peer. The structure would be as follows:

    +--:(vendor-mapping) {dots-telemetry}?
    |  +--rw attack-detail* [attack-id]
    |     +--rw vendor-id?     uint32
    |     +--rw attack-id      uint32
    |     +--rw attack-name    string
    +--:(telemetry) {dots-telemetry}?
       +--rw pre-or-ongoing-mitigation* [cuid tmid]
          ...
          +--rw attack-detail* [attack-id]
             +--rw vendor-id?         uint32
             +--rw attack-id          uint32
             +--rw attack-name?       string
             +--rw attack-severity?   attack-severity
             +--rw start-time?        uint64
             +--rw end-time?          uint64
             ...

Please note that the examples we have in the draft do not include attack-name to insist that this is an optional attribute, e.g.,

==
           "attack-detail": [
             {
               "attack-id": "an-id",
               "start-time": "1957811234",
               "attack-severity": "emergency"
             }
           ]
==

However, the first time that an attack-id is used, the attack-name can also be included.
[Med] That's better. My suggestion goes a little bit further: the name will be used till the mapping table is updated.

  Two immediate issues that come to mind are
"What happens if the telemetry information packet that contains the attack-name gets lost in transit"
[Med] We don't have this issue with the proposal above.

"If a major attack kicks off with many vectors, all of the attack-ids for the first time, there will be a lot of traffic"

[Med] Removing attack-name does not eliminate this risk. The discussion we had about block2/4 or managing this at the DOTS layer applies here.

Regards

Jon

Re-,

Cheers,
Med

Hi Med,

Index: 3016

Regards

Jon

Hi Jon,

Cheers,
Med

Hi All,

Regards

Jon

Hi All,

Regards

Jon

---

From: "Jon Shallow" <[email-masked]>
Date: Wed, 29 Apr 2020 10:10:55 +0100

Hi Med,

Please see inline Jon>

Regards

Jon

Sent: 29 April 2020 09:53

Hi Jon, 

Please see inline.

Cheers,

Med

Hi Med,

Jon> Sure

Putting aside defining attack-id as an integer, I thought your proposal
would be as follows (with removing attack-name from the attack-detail). 

    |     +--rw attack-id      string

          …

             +--rw vendor-id?         uint32

             +--rw attack-id          string

             +--rw start-time?        uint64

             +--rw end-time?          uint64

             …

I was suggesting to leave the name under attack-detail but use it only when
an attack-id/name mapping is not already shared with the peer. The structure
would be as follows:

    |     +--rw attack-id      uint32

          …

             +--rw vendor-id?         uint32

             +--rw attack-id          uint32

Jon> I see this is now an integer

             +--rw start-time?        uint64

             +--rw end-time?          uint64

             …

Please note that the examples we have in the draft do not include
attack-name to insist that this is an optional attribute, e.g.,

Jon> Agreed – however “an-id” needs to be made more human readable at some
point.

==

           "attack-detail": [

             {

               "attack-id": "an-id",

             }

           ]

==

be included.

[Med] That’s better. My suggestion goes a little bit further: the name will
be used till the mapping table is updated.  

Jon> I am beginning to think that vendor-id should be a key as well as
attack-id, as attack-id could be the same across multiple vendors but mean
different things.

[Med] We don’t have this issue with the proposal above.

Jon> Agreed until attack mapping has been refreshed – which has the
potential to fail when not in peace time.

Jon>  Does the mapping refresh want to be done over the data channel as it
could be quite large?

[Med] Removing attack-name does not eliminate this risk. The discussion we
had about block2/4 or managing this at the DOTS layer applies here.

Jon> Agreed.  However if the attack mapping is in place, there will be a lot
less data which would potentially ease the issue.

~Jon

Regards

Jon

Re-, 

Cheers,

Med

Hi Med,

Index: 3016

Regards

Jon

Hi Jon,

Cheers,

Med

Hi All,

Regards

Jon

Hi All,

Regards

Jon

---

From: <[email-masked]>
Date: Wed, 29 Apr 2020 10:29:26 +0000

Re-,

We need to agree if the client has to share its mappings with the server as well. The use of the data channel would make sense as id/name mapping is similar to the alias handling functionality.

Vendor-id should be a key too.

Cheers,
Med

De : Jon Shallow [mailto:[email-masked]]
Envoyé : mercredi 29 avril 2020 11:11
À : BOUCADAIR Mohamed TGI/OLN; [email-masked]
Objet : RE: [Dots] Telemetry draft: Vendor Specific data reduction

Hi Med,

Please see inline Jon>

Regards

Jon

Hi Jon,

Please see inline.

Cheers,
Med

Hi Med,

Please note that the examples we have in the draft do not include attack-name to insist that this is an optional attribute, e.g.,
Jon> Agreed - however "an-id" needs to be made more human readable at some point.
==
           "attack-detail": [
             {
               "attack-id": "an-id",
               "start-time": "1957811234",
               "attack-severity": "emergency"
             }
           ]
==

However, the first time that an attack-id is used, the attack-name can also be included.
[Med] That's better. My suggestion goes a little bit further: the name will be used till the mapping table is updated.
Jon> I am beginning to think that vendor-id should be a key as well as attack-id, as attack-id could be the same across multiple vendors but mean different things.

  Two immediate issues that come to mind are
"What happens if the telemetry information packet that contains the attack-name gets lost in transit"
[Med] We don't have this issue with the proposal above.
Jon> Agreed until attack mapping has been refreshed - which has the potential to fail when not in peace time.
Jon>  Does the mapping refresh want to be done over the data channel as it could be quite large?

[Med] Removing attack-name does not eliminate this risk. The discussion we had about block2/4 or managing this at the DOTS layer applies here.
Jon> Agreed.  However if the attack mapping is in place, there will be a lot less data which would potentially ease the issue.
~Jon

Regards

Jon

Re-,

Cheers,
Med

Hi Med,

Index: 3016

Regards

Jon

Hi Jon,

Cheers,
Med

Hi All,

Regards

Jon

Hi All,

Regards

Jon

---

From: "Jon Shallow" <[email-masked]>
Date: Wed, 29 Apr 2020 11:45:38 +0100

It is my belief that the client should share the mappings with the server as
well, even though the server may actually have the mappings because the
server is supplied by the same vendor.

There is no harm on the server reporting any differences – there is the
possibility that 2 clients have 2 different mitigator release versions
though what the client does with the differences (other than log them), I am
not sure.

Regards

Jon

Sent: 29 April 2020 11:29

Re-,

We need to agree if the client has to share its mappings with the server as
well. The use of the data channel would make sense as id/name mapping is
similar to the alias handling functionality.

Cheers,

Med

Hi Med,

Please see inline Jon>

Regards

Jon

Hi Jon, 

Please see inline.

Cheers,

Med

Hi Med,

Jon> Sure

    |     +--rw attack-id      string

          …

             +--rw vendor-id?         uint32

             +--rw attack-id          string

             +--rw start-time?        uint64

             +--rw end-time?          uint64

             …

    |     +--rw attack-id      uint32

          …

             +--rw vendor-id?         uint32

             +--rw attack-id          uint32

             +--rw start-time?        uint64

             +--rw end-time?          uint64

             …

==

           "attack-detail": [

             {

               "attack-id": "an-id",

             }

           ]

==

~Jon

Regards

Jon

Re-, 

Cheers,

Med

Hi Med,

Index: 3016

Regards

Jon

Hi Jon,

Cheers,

Med

Hi All,

Regards

Jon

Hi All,

Regards

Jon

---

From: <[email-masked]>
Date: Wed, 29 Apr 2020 11:36:36 +0000

Re-,

The augment to the data channel would be:

  augment /ietf-data:dots-data/ietf-data:dots-client:
    +--rw vendor-mapping {dots-telemetry}?
       +--rw attack-detail* [vendor-id attack-id]
          +--rw vendor-id      uint32
          +--rw attack-id      string
          +--rw attack-name    string
  augment /ietf-data:dots-data/ietf-data:capabilities:
    +--ro vendor-mapping {dots-telemetry}?
       +--ro attack-detail* [vendor-id attack-id]
          +--ro vendor-id      uint32
          +--ro attack-id      string
          +--ro attack-name    string

Please see inline.

Cheers,
Med

De : Jon Shallow [mailto:[email-masked]]
Envoyé : mercredi 29 avril 2020 12:46
À : BOUCADAIR Mohamed TGI/OLN; [email-masked]
Objet : RE: [Dots] Telemetry draft: Vendor Specific data reduction

It is my belief that the client should share the mappings with the server as well, even though the server may actually have the mappings because the server is supplied by the same vendor.
[Med] Not sure to understand the last part of your sentence.

There is no harm on the server reporting any differences - there is the possibility that 2 clients have 2 different mitigator release versions though what the client does with the differences (other than log them), I am not sure.
[Med] The mapping is per client, not per domain. A diff can't be interpreted as a conflict.

Regards

Jon

Re-,

Cheers,
Med

Hi Med,

Please see inline Jon>

Regards

Jon

Hi Jon,

Please see inline.

Cheers,
Med

Hi Med,

Regards

Jon

Re-,

Cheers,
Med

Hi Med,

Index: 3016

Regards

Jon

Hi Jon,

Cheers,
Med

Hi All,

Regards

Jon

Hi All,

Regards

Jon

---

From: "Jon Shallow" <[email-masked]>
Date: Wed, 29 Apr 2020 12:57:39 +0100

Hi Med,

See inline Jon1>

Regards

Jon

Sent: 29 April 2020 12:37

Re-, 

          +--rw vendor-id      uint32

          +--rw attack-id      string

          +--rw attack-name    string

          +--ro vendor-id      uint32

          +--ro attack-id      string

          +--ro attack-name    string

Please see inline. 

Cheers,

Med

Jon1> If the server and client are provided by the same vendor, then they
will both know the vendor-id specific mappings so for the client to send
them (or the server to respond to respond with them) is not necessarily
needed, but is useful for consistency across all vendors.

[Med] The mapping is per client, not per domain. A diff can’t be interpreted
as a conflict. 

Jon1> Good thinking – not an issue.  However, the client and server from the
same vendor may be on different code releases (or attack matching details)
and so there could be a difference (but I would have expected that one of
the mappings is a superset of the other, not with any replacements, but
there may be local modifications ….).  However, if the client tells the
server, the server than updates its local, per client, mapping and only uses
that mapping for any reporting etc.  Similarly,  when the client pulls back
the server version of the mapping, it updates the active mapping table to
use for telemetry being sent by the server, but continues to use the mapping
tables it told the server for telemetry from client to server.

~Jon1

Regards

Jon

Re-,

Cheers,

Med

Hi Med,

Please see inline Jon>

Regards

Jon

Hi Jon, 

Please see inline.

Cheers,

Med

Hi Med,

Jon> Sure

    |     +--rw attack-id      string

          …

             +--rw vendor-id?         uint32

             +--rw attack-id          string

             +--rw start-time?        uint64

             +--rw end-time?          uint64

             …

    |     +--rw attack-id      uint32

          …

             +--rw vendor-id?         uint32

             +--rw attack-id          uint32

             +--rw start-time?        uint64

             +--rw end-time?          uint64

             …

==

           "attack-detail": [

             {

               "attack-id": "an-id",

             }

           ]

==

~Jon

Regards

Jon

Re-, 

Cheers,

Med

Hi Med,

Index: 3016

Regards

Jon

Hi Jon,

Cheers,

Med

Hi All,

Regards

Jon

Hi All,

Regards

Jon