Subject: [Anima] GRASP maximum message size considerations

From: Brian E Carpenter <[email-masked]>
Date: Sat, 21 Aug 2021 17:17:10 +1200

Hi,

When thinking about next steps for draft-ietf-anima-grasp-distribution, I began to think about the fact that GRASP (RFC8990) defines a default maximum message size. For distribution of significant amounts of data, this could be an issue.

The RFC says this:
"GRASP nodes MUST be able to receive unicast messages of at least GRASP_DEF_MAX_SIZE bytes. GRASP nodes MUST NOT send unicast messages longer than GRASP_DEF_MAX_SIZE bytes unless a longer size is explicitly allowed for the objective concerned. For example, GRASP negotiation itself could be used to agree on a longer message size."

GRASP_DEF_MAX_SIZE is defined as 2048.

The RFC also says:

"The maximum size of multicast messages (M_DISCOVERY and M_FLOOD) depends on the link-layer technology or the link-adaptation layer in use."

I'm not 100% sure that is correct, because UDP fragmentation works. In practice, my code simply uses GRASP_DEF_MAX_SIZE for that.

There are two cases to consider:

(1) Flooding (M_FLOOD) messages. These are UDP multicasts, so in effect all nodes must agree on the same maximum size. To send messages above the present limit, the maximum flood message size would have to be increased everywhere in the autonomic network. That is trivial if we allow operator configuration, but since an AN should be self-creating, we want to avoid operator configuration. Therefore, we need GRASP to be able to self-configure this.

(2) Unicast messages (discovery responses, synchronization, negotiation). On investigation, it is very easy to allow such messages over TCP to be of any reasonable length, with no protocol change and without all nodes necessarily agreeing on what a "reasonable length" is. I discovered that my demo implementation of GRASP would already handle messages up to 4096 bytes, and tweaked the code to remove that limitation. I tested it with a GRASP objective exceeding 5000 bytes.

It would perhaps be wise to have a network-wide definition of "reasonable", so again a mechanism for GRASP to self-configure this would be useful.

Therefore, my conclusion is that we need a general mechanism for GRASP to configure itself autonomically across a given AN. Maximum message sizes are just an example.
A second message coming right after this one will tackle that topic.
 
Regards
   Brian Carpenter

---

From: Michael Richardson <[email-masked]>
Date: Sat, 21 Aug 2021 18:43:13 -0400

Brian E Carpenter <[email-masked]> wrote:
    > (1) Flooding (M_FLOOD) messages. These are UDP multicasts, so in effect
    > all nodes must agree on the same maximum size. To send messages above
    > the present limit, the maximum flood message size would have to be
    > increased everywhere in the autonomic network. That is trivial if we
    > allow operator configuration, but since an AN should be self-creating,
    > we want to avoid operator configuration. Therefore, we need GRASP to be
    > able to self-configure this.

For the flooded messages over UDP, it seems unwise to ever assume we can
reliably get more than 1280 through.  In production, this goes over IPsec ESP tunnels.

At some point, we probably should specify use of
    IP-TFS: Aggregation and Fragmentation Mode for ESP and its Use for IP
                         Traffic Flow Security
                      draft-ietf-ipsecme-iptfs-09

This would provide a better fragmentation regime than what we do now, and
there maybe some traffic analysis and predictability advantages here.
(If ACP traffic *always* takes 1% of bandwidth, it becomes more predictable,
but also there may be congestion advantages to having "reserved" that bandwidth)

    > (2) Unicast messages (discovery responses, synchronization,
    > negotiation). On investigation, it is very easy to allow such messages
    > over TCP to be of any reasonable length, with no protocol change and
    > without all nodes necessarily agreeing on what a "reasonable length"
    > is. I discovered that my demo implementation of GRASP would already
    > handle messages up to 4096 bytes, and tweaked the code to remove that
    > limitation. I tested it with a GRASP objective exceeding 5000 bytes.

For the messages over TCP, which in response to a UDP flooded or discovery
message, the discovering node can announce it's ability to receive bigger
messages (and the size).

In the Discovery Response Message, the discovered node could initiate it's
maximum too.

How to fit this into the messages, I don't know yet.

It's not clear me that we need to do this network-wide, but we could do that.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 23 Aug 2021 10:30:16 +1200

One point in line:

> reliably get more than 1280 through.  In production, this goes over IPsec ESP tunnels.

Why would that break IPv6 fragmentation? We can assume a well-defined
MTU within an autonomic network, I think. These are all link-local
addressed packets, so there is no PMTUD problem.

   Brian

---

From: Michael Richardson <[email-masked]>
Date: Sun, 22 Aug 2021 20:26:47 -0400

> addressed packets, so there is no PMTUD problem.

But, it's not over link-local.
It's over a mesh of point to point IPsec ESP tunnels.

Choices are:
  1) fragment before encrypt.  Reassemble after decrypt.
  2) encrypt and then fragment the ESP.
     Depends upon ESP assembly buffer being big enough.

Both tend to work, up to some ill-defined limit which is not always 64K.
(1) is likely easier to fix if it's broken, since re-assembly happens in the
control plane CPU, rather than, possibly, in some IPsec hardware.

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 23 Aug 2021 13:59:28 +1200

On 23-Aug-21 12:26, Michael Richardson wrote:
>     >> > increased everywhere in the autonomic network. That is trivial 
if
>     > we

Sure, but the VRF has to emulate LL multicast to ff02::13, and
there must be a well-defined MTU for that emulation, regardless
of the mesh.
 
>   1) fragment before encrypt.  Reassemble after decrypt.

Please.

>      Depends upon ESP assembly buffer being big enough.

Please not.

   Brian