Subject: [Lsvr] Tsvart early review of draft-ietf-lsvr-l3dl-03

From: Joerg Ott via Datatracker <[email-masked]>
Date: Tue, 05 May 2020 11:58:36 -0700

Reviewer: Joerg Ott
Review result: Ready with Issues

The draft describes a peer/neighbour discovery mechanisms for large-scale L2/L3
topologies in data centres. The aim is provide a protocol by means of which the
involved nodes can learn about other nodes connected to their (broadcast or
point-to-point) L2 links and about their respectively support encapsulation
schemes, identifiers, L2/L3 addresses, etc. This information is then provided
to a higher layer for further processing.

The document is well written and fairly easy to follow, but could benefit from
a bit of extra context and target application domain in the introduction. E.g.,
explaining explicitly who would talk L3DL to whom.

>From a transport perspective, I see three potential issues that deserve
clarification or reconsideration:

1. Section 10 spells out a default HELLO interval of 60 seconds. With a large
broadcast domain, this may create quite a bit of traffic. While this may not be
an issue in well-provisioned data center networks,  a remark about sensible
value ranges and the implications may be worthwhile. Just to provide some
guidelines to implementers (who want to offer choices) and operators (who pick
them).

2. Section 10 also suggest that in response to HELLO messages nodes will issue
OPEN PDUs to newly discovered peers. This appears to bear the clear risk of an
OPEN implosion when many system come up at the same time. Shouldn't guidance be
given to avoid repeated traffic surges and possible losses and thus unnecessary
delays? (I noted that other places foresee exponential backoff when
retransmitting OPEN and other ACKed PDUs).

3. When the protocol applies fragmentation, should there be a note on
preventing bursts?

Other notes:
Section 7 on the checksum needs more detail. It also talks about a "suggested"
algorithm but this should be clearly mandated or way to choose one by means of
configuration for a complete data centre would need to be made explicit. I also
assume that the pseudo code on p.11 would benefit from a leader '0' in
0xffffffff -> 0x0ffffffff, otherwise expansion to 64 bits might fill the high
order bits with '1's, which is clearly not intended.

Section 11, p.17, second to last para ("If a properly authenticated...").  From
the text, it is unclear what is meant by an "OPEN with the Serial Number of the
last data received".

I am curious about the error code, providing 16 bits for additional
explanation. Why not a text field? Also wondering if repeated retries (due to
failure, not lost packets) could yield fast repeated transmissions.

Section 15, should the KEEPALIVE interval have suggested (lower) bounds?
At the top of p.26, it says "One per second is the default", the previous page
at the bottom refers to the inter-KEEPALIVE interval of ten seconds. Not sure
if the two are the same, I suppose so. If they are, the numbers should match.
If they are not, we'll need some extra text to explain the difference.

Nits:
There are two spellings of "Encapsulation", capitalised and lower case. Use one
consistently. p10, first para: comprise -> comprising

---

From: Randy Bush <[email-masked]>
Date: Tue, 05 May 2020 15:45:09 -0700

wow!  thanks!  great review.

unfortunately the wg has gone dormant, so we have let the l3dl drafts
expire.  should it ever wake up, i will happly merge in your excellent
suggestions.

again, thank you!

randy

---

From: Joerg Ott <[email-masked]>
Date: Wed, 6 May 2020 07:30:31 +0200

Oops.  I was told this was an "early" review, not a "late" one :-)
Was a good read in any case...

Cheers,
Jörg

On 06.05.20 00:45, Randy Bush wrote:
> _______________________________________________
> Tsv-art mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tsv-art
>

---

From: "Van De Velde, Gunter (Nokia - BE/Antwerp)" <[email-masked]>
Date: Wed, 6 May 2020 09:13:16 +0000

Hi Randy,

Wake-up call received. 

Can you refresh the draft-ietf-lsvr-l3dl-03 document and I'll launch WGLC on that document.

Can you refresh " draft-ymbk-lsvr-l3dl-signing " and " draft-ymbk-lsvr-l3dl-ulpc" and we'll launch a WG adoption call.

Brgds,
G/

-----Original Message-----
From: Randy Bush <[email-masked]> 
Sent: Wednesday, May 6, 2020 00:45
To: Joerg Ott via Datatracker <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: Tsvart early review of draft-ietf-lsvr-l3dl-03

unfortunately the wg has gone dormant, so we have let the l3dl drafts expire.  should it ever wake up, i will happly merge in your excellent suggestions.

again, thank you!

randy

> The draft describes a peer/neighbour discovery mechanisms for 
> large-scale L2/L3 topologies in data centres. The aim is provide a 
> protocol by means of which the involved nodes can learn about other 
> nodes connected to their (broadcast or
> point-to-point) L2 links and about their respectively support 
> encapsulation schemes, identifiers, L2/L3 addresses, etc. This 
> information is then provided to a higher layer for further processing.
> 
> The document is well written and fairly easy to follow, but could 
> benefit from a bit of extra context and target application domain in 
> the introduction. E.g., explaining explicitly who would talk L3DL to whom.
> 
> 1. Section 10 spells out a default HELLO interval of 60 seconds. With 
> a large broadcast domain, this may create quite a bit of traffic. 
> While this may not be an issue in well-provisioned data center 
> networks,  a remark about sensible value ranges and the implications 
> may be worthwhile. Just to provide some guidelines to implementers 
> (who want to offer choices) and operators (who pick them).
> 
> 2. Section 10 also suggest that in response to HELLO messages nodes 
> will issue OPEN PDUs to newly discovered peers. This appears to bear 
> the clear risk of an OPEN implosion when many system come up at the 
> same time. Shouldn't guidance be given to avoid repeated traffic 
> surges and possible losses and thus unnecessary delays? (I noted that 
> other places foresee exponential backoff when retransmitting OPEN and other ACKed PDUs).
> 
> algorithm but this should be clearly mandated or way to choose one by 
> means of configuration for a complete data centre would need to be 
> made explicit. I also assume that the pseudo code on p.11 would 
> benefit from a leader '0' in 0xffffffff -> 0x0ffffffff, otherwise 
> expansion to 64 bits might fill the high order bits with '1's, which is clearly not intended.
> 
> Section 11, p.17, second to last para ("If a properly 
> authenticated...").  From the text, it is unclear what is meant by an 
> "OPEN with the Serial Number of the last data received".
> 
> explanation. Why not a text field? Also wondering if repeated retries 
> (due to failure, not lost packets) could yield fast repeated transmissions.
> 
> At the top of p.26, it says "One per second is the default", the 
> previous page at the bottom refers to the inter-KEEPALIVE interval of 
> ten seconds. Not sure if the two are the same, I suppose so. If they are, the numbers should match.
> There are two spellings of "Encapsulation", capitalised and lower 
> case. Use one consistently. p10, first para: comprise -> comprising
> 
> 
>

---

From: Randy Bush <[email-masked]>
Date: Wed, 06 May 2020 04:34:57 -0700

> Oops.  I was told this was an "early" review, not a "late" one :-)

it was an early review.  so early that too many people seemed to be
asleep.  but i see that the excellence of your review may have wakened
them.  thank you.

randy

---

From: Randy Bush <[email-masked]>
Date: Wed, 06 May 2020 04:36:44 -0700

hi gunter,

> Wake-up call received.

good morning!  :)

> Can you refresh " draft-ymbk-lsvr-l3dl-signing " and "
> draft-ymbk-lsvr-l3dl-ulpc" and we'll launch a WG adoption call.

will do so later today.  on euro/noam research calls at this hour.

thanks!

randy

---

From: Randy Bush <[email-masked]>
Date: Wed, 06 May 2020 15:40:52 -0700

jörg:

again, thanks for taking the time for a strong review.  it is much
appreciated.

> could benefit from a bit of extra context and target application
> domain in the introduction. E.g., explaining explicitly who would talk
> L3DL to whom.

how about the last para of intro getting a few more words along the line
of

   In this document, the use case for L3DL is for point to point links
   in a datacenter Clos in order to exchange the data needed for BGP-SPF
   [I-D.ietf-lsvr-bgp-spf] bootstrap and continuity.  Once layer two
   connectivity has been leveraged to get layer three addressability and
   forwarding capabilities, normal layer three forwarding and routing
   can take over.

   L3DL might be found to be more widely applicable to a range of
   routing and similar protocols which need layer three discovery and
   characterisation.

> a large broadcast domain, this may create quite a bit of
> traffic. While this may not be an issue in well-provisioned data
> center networks, a remark about sensible value ranges and the
> implications may be worthwhile. Just to provide some guidelines to
> implementers (who want to offer choices) and operators (who pick
> them).

   If the configured destination address is one that is propagated by
   switches, the HELLO SHOULD be repeated at a configured interval, with
   a default of 60 seconds.  This allows discovery by new devices which
   come up on the layer-2 mesh.  In this multi-link scenario, the
   operator should be aware of the trade-off between timer tuning and
   network noise and adjust the inter-HELLO timer accordingly.

> other places foresee exponential backoff when retransmitting OPEN and

added

   To ameliorate possible load spikes during bootstrap or event
   recovery, there SHOULD be a jittered delay between receipt of a HELLO
   and issue of the OPEN.  The default delay range SHOULD BE zero to
   five seconds, and MUST be configurable.

> 3. When the protocol applies fragmentation, should there be a note on

likely part of this is our fault, as we did not mean 'fragmentation' in
the classic "oops!  we found a hop with a small mtu."

does augmenting sec 6 as follow help?

   This is not classic 'fragmentation', but rather decomposition at the
   origin to allow PDU payloads larger than the frame allows.  There are
   no intermediate devices capable of further fragmentation or
   reassembly.

   L3DL is carrying relatively small amounts of data on relatively high
   bandwidth links, and at a time when the link is not active with other
   data as it does not yet have layer three connectivity.  So congestion
   is not considered a sufficiently significant risk to warrent
   additional complexity.

> Section 7 on the checksum needs more detail.

could you be more specific re in what area?

> It also talks about a "suggested" algorithm but this should be clearly
> mandated or way to choose one by means of configuration for a complete
> data centre would need to be made explicit.

   The following code describes a suggested algorithm.  This
   specification avoids mandatory to implement, algorithm agility, etc.
   What matters is that the same algorithm is used consistently in any
   deployment.

> I also assume that the pseudo code on p.11 would benefit from a leader
> '0' in 0xffffffff -> 0x0ffffffff, otherwise expansion to 64 bits might
> fill the high order bits with '1's, which is clearly not intended.

ok, but what kinky compiler are you using?  :)

     result = (result >> 32) + (result & 0x0FFFFFFFF);
     result = (result >> 32) + (result & 0x0FFFFFFFF);

> "OPEN with the Serial Number of the last data received".

   If a properly authenticated OPEN arrives with a new Nonce from an
   LLEI with which the receiving logical link endpoint believes it
   already has an L3DL session (OPENs have already been exchanged), and
   the Serial Number in the OPEN PDU is non-zero, the receiver SHOULD
   establish a new session by sending an OPEN with the Serial Number
   being the same as that of the last sent and ACKed PDU.  Each party
   MUST resume sending encapsulations etc. subsequent to the other
   party's Sequence Number.  And each MUST retain all previously
   discovered encapsulation and other data.

> explanation.  Why not a text field?

parsing by the receiver of error codes then becomes an larger unbounded
chase.  but this bit of syntactic sugar may be more pain than it is
worth.  free form or under-defined fields are too fertile a ground for
interoperational incompatibility.  opinions solicited.

> Also wondering if repeated retries (due to failure, not lost packets)
> could yield fast repeated transmissions.

could you be specific about what you think could be improved in

   12.1.  Retransmission

   If a PDU sender expects an ACK, e.g. for an OPEN, an Encapsulation, a
   VENDOR PDU, etc., and does not receive the ACK for a configurable
   time (default one second), and the interface is live at layer 2, the
   sender resends the PDU using exponential back-off, see [RFC1122].
   This cycle MAY be repeated a configurable number of times (default
   three) before it is considered a failure.  The session MAY BE
   considered closed in case of this ACK failure.

> Section 15, should the KEEPALIVE interval have suggested (lower)
> bounds?  At the top of p.26, it says "One per second is the default",
> the previous page at the bottom refers to the inter-KEEPALIVE interval
> of ten seconds. Not sure if the two are the same, I suppose so. If
> they are, the numbers should match.  If they are not, we'll need some
> extra text to explain the difference.

whoops.  how about ten seconds for both?

> case. Use one consistently.

mind if we stall on this one?  we were attempting to differentiate
between the proper noun of a PDU type and normal use of the world.  but
looking at the text, it's a bit messy.  full employment for rfc editors!
:)

> p10, first para: comprise -> comprising

s/compare/comparing/  in

      [RFC1982] on DNS Serial Number Arithmetic for too much detail on
      comparing and incrementing a wrapping sequence number.

pre-published text and xml are at https://git.rg.net/randy/draft-l3dl

thanks again!

randy

---

From: Rob Austein <[email-masked]>
Date: Wed, 06 May 2020 19:41:03 -0400

>      result = (result >> 32) + (result & 0x0FFFFFFFF);

Last I checked, the pedantic C syntax would be 0xFFFFFFFFU.
But pseudo-code means whatever we define it to mean :)

---

From: Joerg Ott <[email-masked]>
Date: Thu, 7 May 2020 09:34:35 +0200

> But pseudo-code means whatever we define it to mean :)

Just leave it as is.

If my dim memory on this serves me somewhat well, the issue arose with
different compilers at some time in the past.  Just checked and it seems
fine for XCode now.  I came across this multiple times, which is
probably why it stuck.

Jörg

---

From: Randy Bush <[email-masked]>
Date: Thu, 07 May 2020 05:06:15 -0700

gunter et alia

> Can you refresh the draft-ietf-lsvr-l3dl-03 document and I'll launch

i have published -04 after Jörg Ott's early transport review.

i am not completely comfortable that it is of wglc quality, though i
never am.  i really would deeply appreciate if a few folk would take
the time to give it a read.

thanks.

randy

---

From: Joerg Ott <[email-masked]>
Date: Mon, 11 May 2020 22:40:27 +0200

Randy,

thanks much, inline.

>     routing and similar protocols which need layer three discovery and

Sounds (well: reads) good.

>     network noise and adjust the inter-HELLO timer accordingly.

Ack.

>     five seconds, and MUST be configurable.

Ok.

> the classic "oops!  we found a hop with a small mtu."

I didn't take it to mean classic fragmentation but rather ALF-style
operation.  Still, this could generate bursts depending on how much
information there is to 'fragment'.

>     is not considered a sufficiently significant risk to warrent

Fine by me.  I assume that the limited amount of information you carry
in the first place would not yield more than 2 or 3 packets, so that the
above sounds good.

> could you be more specific re in what area?

Let me go back:

The code is fine (and certainly a good idea).  I was kind of expecting
the code to be described in text, but given the code is there, the
concise description is fine.  Except that I got confused by
"repeat until zero".  What should be zero?  Looking at the code, surely
not the result.

>     What matters is that the same algorithm is used consistently in any

Ok.

>       result = (result >> 32) + (result & 0x0FFFFFFFF);

We had this and I am happy to undo this comment.  Hasn't occurred
lately but surely earlier.

Probably not having the leading zero would ensure that people don't
assume that it's just 7 x F instead of 8.

>     discovered encapsulation and other data.

Clear now.

> interoperational incompatibility.  opinions solicited.

fine by me either way, was just curious.

> could you be specific about what you think could be improved in

I was trying to decipher my handwriting on the tablet, and it appears to
say "wait before retry to prevent fast loops", scribbled on the bottom
of p.22.  I had to go back tot he text and I believe this refers back to
p.18, first para:

"The Key is specific to the operational environment.  A failure to
    authenticate is a failure to start the L3DL session, an ERROR PDU
    MUST BE sent (Error Code 3), and HELLOs MUST be restarted."

If you have a systematic auth failure, this would cause HELLOs to be
restarted.  Without any jittering (now you introduced some above),
you could, in principle, get a loop:

HELLO -> OPEN -> (Auth) ERROR -> HELLO -> OPEN -> (Auth) ERROR -> ...

So misconfig could be bad unless there is some delay built in.

> whoops.  how about ten seconds for both?

I have no specific opinion as long as it is consistent.  10s for both is
fine.

> looking at the text, it's a bit messy.  full employment for rfc editors!

yay! :-)

> pre-published text and xml are at https://git.rg.net/randy/draft-l3dl

Thanks much for the super-quick responses!

Cheers,
Jörg

---

From: Randy Bush <[email-masked]>
Date: Sat, 23 May 2020 11:32:42 -0700

sorry to take a while to get back to this.  your review is really
appreciated.

i think we converged on most things.  the following seemed open.

> information there is to 'fragment'.

yes, it is app level framing.  perhaps we should call it that explicitly
or even segmentataion or some term less well known.

do you perhaps have a specific suggestion?

:)

    Sum up 32-bit unsigned ints in a 64-bit long, then take the
    high-order section, shift it right filling on the left with zeros,
    rotate, add it in, repeat until the high order 32 bits are all zero.

> So misconfig could be bad unless there is some delay built in.

so i added

    Although delay and jitter in responding with an OPEN were specified
    above, beware of load created by long strings of authentication
    failures and retries.

but i am unsure of what action to recommend.

>> pre-published text and xml are at https://git.rg.net/randy/draft-l3dl

still holds true

randy

---

From: Joerg Ott <[email-masked]>
Date: Mon, 25 May 2020 21:27:11 +0200

No worries, I am happy about any break I get...

> do you perhaps have a specific suggestion?

Not really.  This all appears artificial if you need two or three
packets for app layer fragmentation.  Maybe one could write something
substantially improved along the lines of:
To prevent packet bursts, a sender SHOULD pace the transmission of
application layer fragmented data units as follows: A sender MAY
transmit up to K packets containing fragments in a burst and SHOULD
pace bursts ... (but how?)

>      rotate, add it in, repeat until the high order 32 bits are all zero.

Sold :-)

> but i am unsure of what action to recommend.

Count to N, raise an alert, pause.  Or something like this?

>>> pre-published text and xml are at https://git.rg.net/randy/draft-l3dl

Thanks -- best,
Jörg

---

From: Randy Bush <[email-masked]>
Date: Mon, 25 May 2020 22:35:20 -0700

> pace bursts ... (but how?)

ok.  i have stared at this three times today and have no bright ideas.
i do not want to start pacing by measuring rtt or other known deep
holes.  so i will try to think some more.

> Count to N, raise an alert, pause.  Or something like this?

    A confugurable failure count limit (default 8) SHOULD result in giving
    up on the connection attempt.

randy

---

From: Joerg Ott <[email-masked]>
Date: Tue, 26 May 2020 07:51:20 +0200

> holes.  so i will try to think some more.

One option could be to just have a note of warning.  Any ideas how big
your application layer messages will usually get...?

---

From: Randy Bush <[email-masked]>
Date: Mon, 25 May 2020 23:04:31 -0700

> your application layer messages will usually get...?

as you pointed out up-thread, a few frames.  a dozen would be highly
unlikely.  but, it's the internet; so you never know.

randy