Subject: Re: [Pals] Some comments on draft-ietf-l2vpn-mpls-tp-mac-wd-00.txt

From: "Shah, Himanshu" <[email-masked]>
Date: Tue, 10 Feb 2015 18:10:51 -0500

Hi Stewart –
Many apologies for the late response. Usual excuse apply – day job!!! ☺
Firstly, thanks for detailed comments. Please see responses in line.

As a side note – we have implemented this solution and deployed.
It is working great, so far..

Thanks,
himanshu

From: Stewart Bryant [mailto:[email-masked]]
Sent: Monday, January 26, 2015 10:48 AM
To: [email-masked]; [email-masked]; [email-masked]
Subject: Some comments on draft-ietf-l2vpn-mpls-tp-mac-wd-00.txt

Hi, I have been looking at this draft and I have a few concerns which

I think need to be addressed.

I think that you need to take a careful look at the error

cases and the MTU issues that arise as a result of the use

of the ACK mechanism you designed to replace TCP.

This system is rather different from the system that you

cloned it from. Firstly the original was just sending status

whereas you are potentially sending a long string of MAC addresses

and secondly the original had a heartbeat that allowed

it to eventually send the status, which you do not seem to have.

In the LDP version the status always gets through and there

is no MTU problem.

I am fearful that if you implemented this it would mostly work

but when some error case occurred you would get some very strange

behavior that would be very difficult to debug, and would look

like a s/w error when it was a protocol error.

- Stewart

[himanshu>]
Today MAC withdrawal scheme does not exist for static PWs.
This draft fills the black holes created due to lack of this feature.
If this proposed solution did not exist, the service will work – albeit with some limping.
So – one can say that this proposal ‘enhances’ the static PW functioning, in most cases greatly,
But in some cases, may be not optimally.

Two points you raise –

-          MTU exceed when sending list of MAC addresses: We have proposed two options –

o   If the list exceed MTU, it is fragmented and sent in multiple packets with incremental sequence number.

o   For simpler implementation, one could use empty MAC list to flush ‘all’.
Both of these scheme works

-          Reliable delivery (as compared to TCP) : 3 retries at one second interval is adequate enough, in our view,

as best effort from the sender. Remember, this is in-band OAM signal. If you are not able to deliver MAC WD,

you are not delivering service data as well. Health of PW is outside-the-scope of this draft but VCCV BFD is

popular tool to make sure that PW is operational.

Besides, there is no calamity if, for some reason, MAC withdraw is not received. MAC flush will not occur,

Receiver will either re-learn MAC address from different PW or black hole will occur until MAC address times out.

Nothing gained, nothing lost. Please note that this is QUITE DIFFERENT from reliability requirement of

PW STATUS SIGNALING. Non-delivery could result in permanent/long-duration-of  service OUTAGE.

===

When the number of MAC addresses to be removed is

   large, empty MAC List TLV may be used.

- I think that should be "the empty MAC"

[himanshu>] Don’t follow. Draft is being explicit in saying that TLV must exist with either zero

value entry or a list of MAC addresses. This is how the base draft defines as well.

Unless you are commenting on English verbiage.

===

   A PW can be signaled via LDP or can be statically provisioned.  In

   the case of static PW, LDP based MAC withdrawal mechanism cannot be

- the LDP.....

[himanshu>] will correct

===

3.  MAC Withdraw OAM Message

   A given MAC withdrawal notification is

   sent as a PW OAM message, and the sender keeps re-transmitting the

   message until it receives an ACK for that message.

- OK, but you need a backoff mechanism which I do not see described

[himanshu>] 3 retransmission at 1 second (default) interval (described in sender behavior para).

Are you suggesting there should be exponential backoff in absence of ACK receive? That could be a

good suggestion, in fact that is how we have implemented in order for it to scale.

=====

   Once a receiver

   successfully remove MAC address(es) in response to a MAC address

   withdraw OAM message, it should not unnecessarily remove MAC

   address(es) upon getting refresh message(s).  To facilitate this, the

   proposed mechanism uses sequence number, and defines a new TLV to

   carry the sequence number.

- I think that using s/n is fine, but you do not need a TLV you

can make it part of the MAC Withdraw OAM Message since it is

always present. The way that you have the message designed below

you always send 64 bits to send 32 bits.

   The format of the MAC address withdraw OAM message is shown in

   Figure 1.  The PW OAM message header is exactly the same as what is

   defined in [RFC6478].  Since the MAC withdrawal PW OAM message is not

   refreshed forever.  A MAC address withdraw OAM message MUST contain a

   "Sequence Number TLV" otherwise the entire message is dropped.

- Suggest you just make it part of the non-TLV message.

[himanshu>] while this is a good suggestion, I don’t see any issue on keeping it as a separate

TLV.

   It

   MAY contain MAC Flush Parameter TLVs defined in [MAC-OPT]  when

   static PWs are deployed in H-VPLS and PBB-VPLS scenarios.  The first

   2 bits of the sequence number TLV are reserved and MUST be set to 0

   on transmit and ignored on receipt.

- Why?

[himanshu>] MAC-OPT is now RFC and is indeed good optimization for MAC WD scheme.

       0                   1                   2                   3

       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      |0 0 0 1|Version|   Reserved    |0xZZ MAC Withdraw OAM Message  |

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      |            Reserved           |  TLV Length   |A|R| Flags     |

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      |Res| Sequence Number TLV (TBD) |  Sequence Number TLV Length   |

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      |                        Sequence Number                        |

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

      |                                                               |

      |                         MAC List TLV                          |

      ~                MAC Flush Parameter TLV (optional)             ~

      |                                                               |

      |                                                               |

      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

            Figure 1: MAC Address Withdraw PW OAM Packet Format

   In this section, MAC List TLV and MAC Flush Parameter TLV are

   collectively referred to as "MAC TLV(s)".  The processing rules of

   MAC List TLV are governed by [RFC4762], and the corresponding rules

   of MAC Flush Parameter TLV are governed by [MAC-OPT].

   "TLV Length" is the total length of all TLVs in the message, and

   "Sequence Number TLV Length" is the length of the sequence number

   field.

   A single bit (called A-bit) is set to indicate if a MAC withdraw

   message is for ACK.  Also, ACK does not include MAC TLV(s).

   Only half of the sequence number space is used.  Modular arithmetic

   is used to detect wrapping of sequence number.  When sequence number

   wraps, all MAC addresses are flushed and the sequence number is

   reset.

- That sounds rather brutal. I am not sure how often this will happen

but if you got rid of the TLV and used 64 bits the answer would be never.

Given the rarity you might just take the PW down.

[himanshu>] 31-bits (~2 billion) sequence numbers are not frequent rollover. We do not see

sequence number roll-over to be an issue. Remember this is event-triggered unlike status

signaling which is  refresh-based.

   A single bit (called R-bit) is set to indicate if the sender is

   requesting reset of the sequence numbers.  The sender sets this bit

   when the Pseudowire is restarted and has no local record of send and

   expected receive sequence number.

 4.  Operation

   This section describes how the initial MAC withdraw OAM messages are

   sent and retransmitted, as well as how the messages are processed and

   retransmitted messages are identified.

4.1.  Operation of Sender

   Each PW is associated with a counter to keep track of the sequence

   number of the transmitted MAC withdrawal messages.  Whenever a node

   sends a new set of MAC TLVs, it increments the transmitted sequence

   number counter, and include the new sequence number in the message.

   The transmit sequence number is initialized to 1 at the onset.

   The sender expects an ACK from the receiver within a time interval

   which we call "Retransmit Time" which can be either a default (1

   second) or configured value.

- I think that you should include this in the message so both parties

know the value.

[himanshu>] Not sure if including the retransmit time in the message has any benefit.

For instance, what would receiver do if it did not receive another MAC withdraw in

retransmit time? The ACK sent by receiver may be in flight.

   If the ACK does not arrive within the

   Retransmit Time, the sender retransmits the message with the same

   sequence number as the original message.  The retransmission is

   ceased anytime when ACK is received or after three retries.

- So what happens if three is not enough? I think that you need

to continue and that is what your earlier text said. However I think

you also need to back off the rate since it may be getting lost

due to some congestion.

[himanshu>] See above for discussions. Endless retransmits has no value – MAC WD

is already decayed and MAC motion and/or MAC age time-out  may already have occurred to

re-learn, late arrival would do more harm than good.

   This

   avoids unended retransmissions in the absence of acknowledgements.

   In addition, if during the period of retransmission, if a need to

   send a new MAC withdraw message with updated sequence number arises

   then retransmission of the older unacknowledged withdraw message is

   suspended and retransmit time for the new sequence number is

   initiated.  In essence, sender engages in retrasmission logic only

   for the latest send withdraw message for a given PW.

- So what happens to the MAC addresses in the old (lost message)?

Are they concatenated with the new addresses or simply ignored?

What happens when the MTU is exhausted - Something that will not happen

with a TCP channel but will happen in an OAM channel?

[himanshu>] See above for this discussions – loss of subset of MAC addresses in not a calamity.

MTU exhaust is handled by dispatching MAC addresses lists broken in MTU sizes each

with seq# increment.

   In the event that a Pseudowire was deleted and re-added or the router

   is restarted with configuration, the local node may lose information

   about the send sequence number of previous incarnation.  This becomes

   problematic for the remote peer as it will continue to ignore the

   received MAC withdraw messages with lower sequence numbers.  In such

   cases, it is desirable to reset the sequence numbers at both ends of

   the Pseudowire.  The 'R' reset bit is set in the first MAC withdraw

   to notify the remote peer to reset the send and receive sequence

   numbers.  The 'R' bit must be cleared in subsequent MAC withdraw

   messages after the acknowledgement is received

- A case that you have not considered is late arrival of an OAM

message - which can happen with FRR. Won't this reset the sequence

numbers at the receiver?

[himanshu>] No. Seq# are not reset. If a receiver receives out-of-seq MAC WD, older one

is not used for flush but acked with latest receive seq#. Seq# always move upwards.

This scheme is no DIFFERENT than seq# scheme we use in control word – it simply works..

Do you always need to send the the R bit whenever there is a wrapped?

[himanshu>] No. You never have to send R-bit UNLESS your PW restarted (on your box) and lost track of

local copy of send and receive sequence#

 4.2.  Operation of Receiver

   Each PW is associated with a register to keep track of the sequence

   number of the MAC withdrawal message received last.  Whenever a MAC

   withdrawal message is received, and if the sequence number on the

   message is greater than the value in the register, the MAC

Sivabalan, et al.        Expires January 2, 2015                [Page 5]

Internet-Draft    MAC Withdrawal over Static Pseudowire        July 2014

   address(es) contained in the MAC TLV(s) is/are removed, and the

   register is updated with the received sequence number.  The receiver

   sends an ACK whose sequence number is the same as that in the

   received message.

   If the sequence number in the received message is smaller than or

   equal to the value in the register, the MAC TLV(s) is/are not

   processed.

- So if it were a superseded message those MACs are not withdrawn?

[himanshu>] Yes – it is not withdrawn.

   However, an ACK with the received sequence number MUST be

   sent as a response.  The receiver processes the ACK message as an

   acknowledgement for all the MAC withdraw messages sent up to the

   sequence number present in the ACK message and terminates

   retransmission.

   As mentioned above, since only half of the sequence number space is

   used, the receiver MUST use modular arithmetic to detect wrapping of

   the sequence number.

- I do not understand how you distinguish a late from a wrap?

[himanshu>] Please check how sequence number logic works in control word.

I believe Andy Malis wrote RFC for it.

   A MAC withdraw message with 'R' bit set MUST be processed by

   resetting the send and receive sequence number first.  The rest of

   MAC withdraw message processing is performed as described above.  The

   acknowledgement is sent with 'R' bit cleared.

5.  IANA Considerations

   The proposed mechanism requests IANA to a assign new channel type

   (recommended value 0x0028) from the registry named "Pseudowire

   Associated Channel Types".  The description of the new channel type

   is "Pseudowire MAC Withdraw OAM Channel".

   IANA needs to create a new registry for Pseudowire Associated Channel

   TLVs, and create an entry for "Sequence Number TLV".  The recommended

   value is 0x0001.

- NO!!!!! we decided a long time ago that ACH TLVs would live in a registry

associated with the channel type, not in a global registry, although in

this case I do not think you need a registry at all.

[himanshu>] Is this good news? We can use value 1 without asking from IANA

- Where does the MAC TLV get assigned?

[himanshu>] from the base PW RFC

- Again do you need a TLV for the MAC addresses - why not a list length?

[himanshu>] See above – MAC LIST TLV is used as is from the base PW RFC.

+++++++++