Subject: [rbridge] merging draft-tissa-trill-oam and draft-ietf-trill-rbridge-oam

From: Santosh Rajagopalan <[email-masked]>
Date: Tue, 6 Mar 2012 11:48:41 -0500

Hi,
We currently have two competing approaches to TRILL OAM, draft-ietf-trill-rbridge-oam and draft-tissa-trill-oam. Both have their merits, which calls for a revised draft with the best ideas from both.

draft-tissa-trill-oam has a dependancy on IP/ICMP/ARP etc.  The major reason for this dependency seems to be for multipath path discovery, and I have some suggestions as to how to fix this in draft-ietf-trill-rbridge-oam without using IP. draft-tissa-trill-oam also has several features which should be rightly queried from a MIB.

I think the OAM draft should concentrate on facilitating actions that would be difficult or impossible to perform in its absence. So querying counters has no place here, because you could easily figure them out by doing an SNMP query or by logging into the switch. I also think it's worthwhile to look at the features traditionally supported in ethernet OAM and to see if they have a place here.

Using draft-ietf-trill-rbridge-oam as a basis:

1) Ping/loopback: draft-ietf-trill-rbridge-oam handles this well enough (it does need a multicast addition). The purpose of ping is to verify connectivity between two rbridges, not to discover the various ways of connecting the two.  One suggestion I have here is to be able to include a cookie in the echo request message which gets reflected back by the receiver. This cookie could be used to include the sender's timestamp in the ping message. The response could then be used to measure round-trip path latency and jitter. If the sender and receiver's clocks are synchronized using some external mechanism then this can also be used to measure one-way delay and jitter. This is a feature borrowed from ethernet OAM.

2) Traceroute: IP traceroute doesn't rely on icmp for the outgoing packet (you could use ICMP, but that's a different matter). In fact, UNIX implementations usually just send UDP packets with port numbers in the 33K range, with appropriate TTL values. Note that traceroute in the IP world gives you "a" path to a destination, not "all possible paths". However, being able to enumerate all possible paths, as well as to pinpoint the particular path a flow would take are good to have features. 
Here're the modifications needed in draft-ietf-trill-rbridge-oam to achieve this:
Multicast unpruned tree discovery: you can discover all existing unpruned multicast trees by setting the erbridge value of an rbridge channel message to the respective roots, M=1 and the dmac to "all-egress-rbridges". Just like regular ping, you can send this as an echo request message with incrementing ttl values, and the ttl expiry messages you get will tell you what the unpruned tree looks like. Changing the egress_rbridge will paint you a picture of the different trees.
Multicast pruned tree discovery: just do a), but replace the "all-egress-rbridges" multicast address in the inner dmac with the mac address of the multicast group. As you vary the egress_rbridge field, this will enumerate all possible pruned trees for that group. Note that draft-ietf-trill-rbridge-oam uses draft-ietf-trill-rbridge-channel, which requires the inner dmac to be all-egress-rbridges. However, the ethertype field can be used by itself to classify an OAM packet, and I am advocating that the inner dmac requirement in draft-ietf-trill-rbridge-channel be relaxed to allow for alternative inner dmacs where needed.
Multicast pruned tree discovery for a particular flow: I recommend building a packet with an empty payload which looks like the actual flow (with the header fields from the flow being tested instead of the rbridge channel header). The IP address, mac address, L4 headers etc all match with that of the flow you're trying to debug. However, to ensure that any leaked packets get dropped, set the IP checksum to be invalid (the IP checksum is never used for the ecmp hash). Now send out successive packets with the ttl field in incrementing values. Note that you're sending a "pseudo-real" packet into the network and relying on received ttl expiry messages to figure out the path.
Unicast path discovery: the erbridge ID is that of the switch you're trying to ping. Let's send this echo request message with a fake inner DMAC. Construct the dmac so that the U/L bit is 1 and the I/G bit is 0 (basically, its a local scope unicast mac). Set the last 32 bits to a random value. This is a fake mac address, but that's ok because the network routes only on the erbridge anyway. The M bit in the TRILL header is 0. Send this as an echo request message with incrementing ttl values. This gives you "a" path from the source to the destination rbridge. Increment the dmac used earlier, and repeat. All rbridges hash at least on the inner dmac, so this will allow you to enumerate all possible paths from source to destination.
Unicast path discovery for a particular flow: Similar to what we did for multicast, construct a real packet of length 0. The IP address, mac address, L4 headers etc all match with that of the flow you're trying to debug. However, to ensure that any leaked packets get dropped, set the IP checksum to be invalid. Now set the ttl field in incrementing values. Note that you're sending a "pseudo-real" packet into the network and relying on received ttl expiry messages to figure out the path.
Note that doing this eliminates the reliance on IP/ICMP, while preserving the objectives of draft-tissa-trill-oam. The fake packets generated are technically not OAM packets, just suggestions on how to achieve a certain objective.

3) Fault notification: although neither draft talks about this, this is a part of the ethernet OAM feature set. I think draft-ietf-trill-rbridge-oam needs to mention that the expectation is that BFD will work over the rbridge channel to achieve this.

4) draft-tissa-trill-oam  has a big section on TTM. The equivalent in Ethernet OAM is performance management by checking counters and send/receive delays, etc. We can cover send/receive delay/jitter measurements using ping (as described earlier). I frankly don't see the benefit to including remote discovery of counters in this OAM proposal. The administrator can always just login to the rbridge of interest and get a read-out of the counters directly, or use snmp remotely. We shouldnt be replicating functionality which is better done elsewhere, and I think this entire section on TTM in draft-tissa-trill-oam should be taken out.

5) I don't see any benefits to DRB/AF discovery OAM messages. This is probably done most easily by logging into the rbridge in question and doing a "show l2 isis", or extending the SNMP MIB. 

6) draft-tissa-trill-oam has a section on MAC discovery (basically, being able to query all rbridges to see what their rbridge<->mac binding is). This is again something that should be best queried using SNMP - this is a classic MIB walk case.

7) Address binding verification should be out of scope for a TRILL OAM draft, and besides, should just be queried at the rbridge with the CLI.

8) End-Station Attachment Point Discovery: should be out of scope of this draft, and is too vendor specific (it uses VN-tags).

best,
Sunny Rajagopalan

_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Thu, 8 Mar 2012 08:54:15 -0800

Firstly there are few critical high-level points that needed to be stated

1.      OAM cannot be just detecting Liveliness and Path discovery. It is about Operational aspects of networks which include but not limited to Detecting Liveliness, Path Discovery, Fault finding, Fault Isolation etc. OAM is never constrained to ping and trace route.  Look at where OAM is heading to in MPLS and MPLS-TP.

2.      TRILL cannot be compared 1-1 with Ethernet or IP. TRILL encapsulate the user payload with the TRILL header and utilize that to forward using TRILL header and some parts of user payload (e,g, ECMP and mcast forwarding and pruning). TRILL utilize multiple paths and multi destination tress. In that regard it is similar to a tunneling protocol. Most logical comparison of TRILL OAM is RFC 4379 MPLS OAM If one takes a look at TRILL evolution, multipathing is the fundamental point of leveraging IP technology within bridging. If one cannot perform OAM functions to verify all the paths, it is a moot point to develop OAM functions.

3.      What is fundamentally required in TRILL OAM is to have OAM framework that allows to keep OAM frames as close as possible to data packet. ICMP is not utilized as a Ping or Traceroute. Motivation here is to provide such framework in backward compatible manner without introducing additional restrictions. As an example, channel draft, which define the framework of OAM channel you are proposing require to use RBridge critical alert bit. RBridges that are not supporting that cannot claim it is Channel capable. Additionally, Channel header follows right after the inner MAC header. This limit including any L3 frames. It is a common practice to select ECMP based on IP flow information and multicast prune based on IP address. (Please see rfc6326 bis for Group IP Address sub-TLV.  draft-ietf-trill-rbridge-channel, which you are proposing to use do not provide that flexibility. In your response below you have proposed several alternatives but in our mind they do not achieve what is required in OAM. Please see specific responses embedded.

4.      Current suggestion from WG members are to utilize RFC 4379 framework, as that allow to utilize lots of existing knowledge, implementation and user experience from MPLS OAM.

5.      The embedded message channel is utilized so receiver can identify OAM frames from data frames and take appropriate actions.  Without limiting what payloads can be include in the OAM packet. 

6.      You have proposed to use incremental TTL value method for multicast with invalid IP checksum. Firstly given the tree depth is different in different branches, this will invariably leak packets out. And will lead to incrementing IP invalid packet count on end stations. In hosting application this may generate unwanted support calls, and something that should not be done. In draft-tissa-trill-oam-03 section 6.3 propose to use this exact method only for unciast. Additionally because we can identify OAM frames from real data with TTL expiry due to embedded message channel, we notify that as the last hop bridge and user would not generate N+1 TTL packet.

7.      Below you have proposed to utilize multicast for path trace (trace route).  Whole purpose of OAM is data plane integrity verification. Trace route in particular, used for fault isolation. Multicast use a different forwarding logic than that of unicast. Use of Multicast to trace route unicat is not going to achieve what is needed and negate the use of OAM. Additionally there are several issues such multicast tree may not cover all links in the campus and so on. Please see specific response embedded in the relevant section

8.      You are proposing to merge the two drafts and I do not see a commonality based on your responses below.

Please see more specific answers below

From: [email-masked] <mailto:[email-masked]>  [mailto:[email-masked]] On Behalf Of Santosh Rajagopalan
Sent: Tuesday, March 06, 2012 8:49 AM
To: [email-masked] <mailto:[email-masked]> 
Cc: Santosh Rajagopalan
Subject: [rbridge] merging draft-tissa-trill-oam anddraft-ietf-trill-rbridge-oam

Hi,

[Answer] Seem the paragraph is contradicting itself. Firstly you say ping is to detect the liveliness. Then you say it can be detected to measure jitter, delay etc. If there are 10 ECMP paths and if you can only measure latency of one it does not give true picture of the information. 

[Answer] This is not a good to have feature, but a MUST, because it is fundamental to fault isolation. Take a look at MPLS OAM and its problem statement. It is no different here. If you model the way you say, running ICMP ping between RBridge suffice the need. Each Rbridge has IP stack anyway. Also, in Linux we do have –I options to trigger ICMP.

[Answer] All of the below is based on using multicast trees. Are you suggesting to use multicast to do trace route. Whole purpose of OAM is data plane integrity verification. Trace route in particular, used for fault isolation. Multicast use a different forwarding logic than that of unicast. Use of Multicast to trace route unicat is not going to achieve what is needed. Secondly, multicast trees will not cover all links. Case in point is having a single tree.

	a.	Multicast unpruned tree discovery: you can discover all existing unpruned multicast trees by setting the erbridge value of an rbridge channel message to the respective roots, M=1 and the dmac to "all-egress-rbridges". Just like regular ping, you can send this as an echo request message with incrementing ttl values, and the ttl expiry messages you get will tell you what the unpruned tree looks like. Changing the egress_rbridge will paint you a picture of the different trees.

[Answer] First reaction to this was am I reading this correctly. After reading this one more time it appeared I am reading it correctly. There are three main issues: 1. There can be 1000 of RBRidges, and getting a response from all of them going to overwhelm. Probably choke the CPU of the requester. 2. When you do incremental TTL value, it will leak out at the last RBridge. 3. Tree does not include all paths. Assume you have 1 tree, (Which is a perfectly a valid configuration), how are you going to cover links not in the tree.

	b.	Multicast pruned tree discovery: just do a), but replace the "all-egress-rbridges" multicast address in the inner dmac with the mac address of the multicast group. As you vary the egress_rbridge field, this will enumerate all possible pruned trees for that group. Note that draft-ietf-trill-rbridge-oam uses draft-ietf-trill-rbridge-channel, which requires the inner dmac to be all-egress-rbridges. However, the ethertype field can be used by itself to classify an OAM packet, and I am advocating that the inner dmac requirement in draft-ietf-trill-rbridge-channel be relaxed to allow for alternative inner dmacs where needed.

[Answer] Fundamental problem:. Prune tree works based on VLAN and MAC address or VLAN and IP address. Are you suggesting to try all VLAN all address here ?

	c.	Multicast pruned tree discovery for a particular flow: I recommend building a packet with an empty payload which looks like the actual flow (with the header fields from the flow being tested instead of the rbridge channel header). The IP address, mac address, L4 headers etc all match with that of the flow you're trying to debug. However, to ensure that any leaked packets get dropped, set the IP checksum to be invalid (the IP checksum is never used for the ecmp hash). Now send out successive packets with the ttl field in incrementing values. Note that you're sending a "pseudo-real" packet into the network and relying on received ttl expiry messages to figure out the path.

[Answer] Aha you are now acknowledging IP is used for ECMP hash. Now I am wondering how Channel header can facilitate that. The concept of utilizing invalid checksum is already covered in draft-tissa-trill-oam-03 Section 6.3. Secondly using TTL in multicast is a dangerous thing, as tree depth can be different and packets can leak out. You may not want to trigger invalid IP checksum counter on your hosting servers. As that may generate additional support calls. Checksum invalid SHOULD only be used to protect against unlikely case packets leaking out. Not the default. 

Question: Are you suggesting to include IP Multicast packets in the native form here or encap within channel header ? 

	d.	Unicast path discovery: the erbridge ID is that of the switch you're trying to ping. Let's send this echo request message with a fake inner DMAC. Construct the dmac so that the U/L bit is 1 and the I/G bit is 0 (basically, its a local scope unicast mac). Set the last 32 bits to a random value. This is a fake mac address, but that's ok because the network routes only on the erbridge anyway. The M bit in the TRILL header is 0. Send this as an echo request message with incrementing ttl values. This gives you "a" path from the source to the destination rbridge. Increment the dmac used earlier, and repeat. All rbridges hash at least on the inner dmac, so this will allow you to enumerate all possible paths from source to destination.

[Answer] You are making big assumption here. Firstly hash is not defined in any standard and it is implementation depended. Some devices may use SMAC:DMAC VLAN combination, Some may use IP flow information. May be low ends use just DMAC. In anycase selecting Random values are not going to gurentee a covering all paths. Secondly one may not even know how many ECMP options available on an RBridge 3 hops away. This is the reason more elaborate methods are required RFC 4379 explain How this can be done in MPLS. Based on the feedback received during the last WG session in Taipei, we have also enhanced to included this. Please see section 9.9. This needed to be done in more systematic manner rather than randomly generating DMAC. DMAC means Destination MAC. SMAC means Source MAC

	e.	Unicast path discovery for a particular flow: Similar to what we did for multicast, construct a real packet of length 0. The IP address, mac address, L4 headers etc all match with that of the flow you're trying to debug. However, to ensure that any leaked packets get dropped, set the IP checksum to be invalid. Now set the ttl field in incrementing values. Note that you're sending a "pseudo-real" packet into the network and relying on received ttl expiry messages to figure out the path.

[Answer] This same concept is in section 6.3 of draft-tissa-trill-oam.  

[Answer] Your understanding is incorrect. TTM  is utilized not for counters but to ensure that specific flows are reaching specifc devices. This is included as part of feedback received from the WG in Tapei. This is an adaptation of Data Driven CFM .

[Answer] How do you know which device to login to use show commands/

[Answer] How do you know which device is having the MAC so you can do the walk ?

[Answer] Firstly how do you know which device to login to use CLI. Secondly, anything that improve usability and experience of users can be always included.

[Answer] Terminology can be changed to include what is defined in VXLAN, NVGRE etc. The case in point is when there are other overlay technologies how to identify different address association. Certain VM may use the same MAC address to associate with different VXLAN segments.

best,
Sunny Rajagopalan

---

From: Santosh Rajagopalan <[email-masked]>
Date: Mon, 12 Mar 2012 15:06:34 -0700

1) Let's say that you are trying to find all possible paths between two 
rbridges (or liveliness) for a)unicast, b) multicast (unpruned) and c) 
multicast (pruned). My proposal  below will let you discover all three, 
without needing IP.  Since we're using the rbridge channel ethertype, 
rbridges can trap these and prevent them from getting to hosts at the 
edge. Note that we rely on unicast forwarding of rbridge channel packets 
for unicast discovery and multicast forwarding of rbridge channel packets 
for multicast discovery. This happens naturally in any trill compliant 
switch.

2) Let's say that you already have the information from 1) above, but you 
are trying to figure out which of the many paths you know of will be used 
for a particular flow. Now, draft-tissa-trill-oam assumes that the inner 
ethernet header is never used for hashing for an IP flow. It assumes that 
as long as you keep the L3 and L4 fields the same as that of the original 
packet, hardware will hash to the same values as that from the original 
flow, even though the L2 header is different ("6.2. Identification of IP 
Flows" in  draft-tissa-trill-oam ). I do not think this is a valid 
assumption to make. The only way to make sure your probe packet takes the 
same path as a regular packet is to make it identical to the original 
packet, all the way including the L2 headers. The IP checksum is never 
used for hashing, so you can invalidate it to make sure such packets get 
dropped at the host. This is the approach I was advocating for doing path 
discovery for a particular flow. 

3) The only hash behavior we can be sure of is that if you give any asic a 
non-IP packet, it will do load balancing using the only available fields, 
which is ethernet smac + dmac. I'm exploiting this property to do path 
discovery (the rbridge ethertype makes asics treat these packets as 
non-IP). Therefore, if you vary the inner dmac, you can exercise all the 
available paths. Paths you discover using this approach are applicable to 
both IP and non-IP traffic, since trill routes are common for both. The 
only hash assumption I've made is that L2 unknown ethertype (rbridge 
channel)  packets get load balanced using the inner dmac/smac. I am making 
no assumption about the fields used by the hardware for hashing IP 
packets.

4) People adopting TRILL are more likely to come from an ethernet 
background than from an MPLS background. For our target audience, it's 
important for OAM to look like what they're already familiar with, which 
is ethernet OAM. TRILL competes with SPB for the same space, and note that 
they make a virtue of being ethernet OAM compatible. Do many people on 
this list feel MPLS OAM is the way to go for us? 

5) I'm assuming that when operators lay out a network, they have access to 
a picture that shows the interconnects (or at least have some management 
tool speaking snmp that does). So when they need to discover the DRB on a 
multi-access link, then can send an SNMP query to the rbridges connected 
on that link to get the DRB information. This also goes for finding the 
mac addresses on a given rbridge, finding the address bindings at a 
particular rbridge etc. All of these "property query" type messages are 
exactly what SNMP was made for. Some things like finding the vn-tag, vxlan 
ID, nvgre ID etc are within the domain of SNMP but out of scope for TRILL. 
What is trill-specific about these? 

I haven't responded to any statements below which are based on a 
misunderstanding of my original email, please re-review it and I'll be 
happy to answer questions on it.

best regards,
Sunny Rajagopalan

From:   "Tissa Senevirathne (tsenevir)" <[email-masked]>
To:     <[email-masked]>
Date:   03/08/2012 09:28 AM
Subject:        Re: [rbridge] merging draft-tissa-trill-oam 
anddraft-ietf-trill-rbridge-oam
Sent by:        [email-masked]

1.      OAM cannot be just detecting Liveliness and Path discovery. It is 
about Operational aspects of networks which include but not limited to 
Detecting Liveliness, Path Discovery, Fault finding, Fault Isolation etc. 
OAM is never constrained to ping and trace route.  Look at where OAM is 
heading to in MPLS and MPLS-TP.
2.      TRILL cannot be compared 1-1 with Ethernet or IP. TRILL 
encapsulate the user payload with the TRILL header and utilize that to 
forward using TRILL header and some parts of user payload (e,g, ECMP and 
mcast forwarding and pruning). TRILL utilize multiple paths and multi 
destination tress. In that regard it is similar to a tunneling protocol. 
Most logical comparison of TRILL OAM is RFC 4379 MPLS OAM If one takes a 
look at TRILL evolution, multipathing is the fundamental point of 
leveraging IP technology within bridging. If one cannot perform OAM 
functions to verify all the paths, it is a moot point to develop OAM 
functions.
3.      What is fundamentally required in TRILL OAM is to have OAM 
framework that allows to keep OAM frames as close as possible to data 
packet. ICMP is not utilized as a Ping or Traceroute. Motivation here is 
to provide such framework in backward compatible manner without 
introducing additional restrictions. As an example, channel draft, which 
define the framework of OAM channel you are proposing require to use 
RBridge critical alert bit. RBridges that are not supporting that cannot 
claim it is Channel capable. Additionally, Channel header follows right 
after the inner MAC header. This limit including any L3 frames. It is a 
common practice to select ECMP based on IP flow information and multicast 
prune based on IP address. (Please see rfc6326 bis for Group IP Address 
sub-TLV.  draft-ietf-trill-rbridge-channel, which you are proposing to use 
do not provide that flexibility. In your response below you have proposed 
several alternatives but in our mind they do not achieve what is required 
in OAM. Please see specific responses embedded.
4.      Current suggestion from WG members are to utilize RFC 4379 
framework, as that allow to utilize lots of existing knowledge, 
implementation and user experience from MPLS OAM.
5.      The embedded message channel is utilized so receiver can identify 
OAM frames from data frames and take appropriate actions.  Without 
limiting what payloads can be include in the OAM packet. 
6.      You have proposed to use incremental TTL value method for 
multicast with invalid IP checksum. Firstly given the tree depth is 
different in different branches, this will invariably leak packets out. 
And will lead to incrementing IP invalid packet count on end stations. In 
hosting application this may generate unwanted support calls, and 
something that should not be done. In draft-tissa-trill-oam-03 section 6.3 
propose to use this exact method only for unciast. Additionally because we 
can identify OAM frames from real data with TTL expiry due to embedded 
message channel, we notify that as the last hop bridge and user would not 
generate N+1 TTL packet.
7.      Below you have proposed to utilize multicast for path trace (trace 
route).  Whole purpose of OAM is data plane integrity verification. Trace 
route in particular, used for fault isolation. Multicast use a different 
forwarding logic than that of unicast. Use of Multicast to trace route 
unicat is not going to achieve what is needed and negate the use of OAM. 
Additionally there are several issues such multicast tree may not cover 
all links in the campus and so on. Please see specific response embedded 
in the relevant section
8.      You are proposing to merge the two drafts and I do not see a 
commonality based on your responses below.

From: [email-masked] [mailto:[email-masked]] On 
Behalf Of Santosh Rajagopalan
Sent: Tuesday, March 06, 2012 8:49 AM
To: [email-masked]
Cc: Santosh Rajagopalan
Subject: [rbridge] merging draft-tissa-trill-oam 
anddraft-ietf-trill-rbridge-oam

Hi,
We currently have two competing approaches to TRILL OAM, 
draft-ietf-trill-rbridge-oam and draft-tissa-trill-oam. Both have their 
merits, which calls for a revised draft with the best ideas from both.

draft-tissa-trill-oam has a dependancy on IP/ICMP/ARP etc.  The major 
reason for this dependency seems to be for multipath path discovery, and I 
have some suggestions as to how to fix this in 
draft-ietf-trill-rbridge-oam without using IP. draft-tissa-trill-oam also 
has several features which should be rightly queried from a MIB.

I think the OAM draft should concentrate on facilitating actions that 
would be difficult or impossible to perform in its absence. So querying 
counters has no place here, because you could easily figure them out by 
doing an SNMP query or by logging into the switch. I also think it's 
worthwhile to look at the features traditionally supported in ethernet OAM 
and to see if they have a place here.

1) Ping/loopback: draft-ietf-trill-rbridge-oam handles this well enough 
(it does need a multicast addition). The purpose of ping is to verify 
connectivity between two rbridges, not to discover the various ways of 
connecting the two.  One suggestion I have here is to be able to include a 
cookie in the echo request message which gets reflected back by the 
receiver. This cookie could be used to include the sender's timestamp in 
the ping message. The response could then be used to measure round-trip 
path latency and jitter. If the sender and receiver's clocks are 
synchronized using some external mechanism then this can also be used to 
measure one-way delay and jitter. This is a feature borrowed from ethernet 
OAM.
[Answer] Seem the paragraph is contradicting itself. Firstly you say ping 
is to detect the liveliness. Then you say it can be detected to measure 
jitter, delay etc. If there are 10 ECMP paths and if you can only measure 
latency of one it does not give true picture of the information. 

2) Traceroute: IP traceroute doesn't rely on icmp for the outgoing packet 
(you could use ICMP, but that's a different matter). In fact, UNIX 
implementations usually just send UDP packets with port numbers in the 33K 
range, with appropriate TTL values. Note that traceroute in the IP world 
gives you "a" path to a destination, not "all possible paths". However, 
being able to enumerate all possible paths, as well as to pinpoint the 
particular path a flow would take are good to have features. 
[Answer] This is not a good to have feature, but a MUST, because it is 
fundamental to fault isolation. Take a look at MPLS OAM and its problem 
statement. It is no different here. If you model the way you say, running 
ICMP ping between RBridge suffice the need. Each Rbridge has IP stack 
anyway. Also, in Linux we do have –I options to trigger ICMP.

Here're the modifications needed in draft-ietf-trill-rbridge-oam to 
achieve this:
[Answer] All of the below is based on using multicast trees. Are you 
suggesting to use multicast to do trace route. Whole purpose of OAM is 
data plane integrity verification. Trace route in particular, used for 
fault isolation. Multicast use a different forwarding logic than that of 
unicast. Use of Multicast to trace route unicat is not going to achieve 
what is needed. Secondly, multicast trees will not cover all links. Case 
in point is having a single tree.
a.      Multicast unpruned tree discovery: you can discover all existing 
unpruned multicast trees by setting the erbridge value of an rbridge 
channel message to the respective roots, M=1 and the dmac to 
"all-egress-rbridges". Just like regular ping, you can send this as an 
echo request message with incrementing ttl values, and the ttl expiry 
messages you get will tell you what the unpruned tree looks like. Changing 
the egress_rbridge will paint you a picture of the different trees.
[Answer] First reaction to this was am I reading this correctly. After 
reading this one more time it appeared I am reading it correctly. There 
are three main issues: 1. There can be 1000 of RBRidges, and getting a 
response from all of them going to overwhelm. Probably choke the CPU of 
the requester. 2. When you do incremental TTL value, it will leak out at 
the last RBridge. 3. Tree does not include all paths. Assume you have 1 
tree, (Which is a perfectly a valid configuration), how are you going to 
cover links not in the tree.
b.      Multicast pruned tree discovery: just do a), but replace the 
"all-egress-rbridges" multicast address in the inner dmac with the mac 
address of the multicast group. As you vary the egress_rbridge field, this 
will enumerate all possible pruned trees for that group. Note that 
draft-ietf-trill-rbridge-oam uses draft-ietf-trill-rbridge-channel, which 
requires the inner dmac to be all-egress-rbridges. However, the ethertype 
field can be used by itself to classify an OAM packet, and I am advocating 
that the inner dmac requirement in draft-ietf-trill-rbridge-channel be 
relaxed to allow for alternative inner dmacs where needed.
[Answer] Fundamental problem:. Prune tree works based on VLAN and MAC 
address or VLAN and IP address. Are you suggesting to try all VLAN all 
address here ?
c.      Multicast pruned tree discovery for a particular flow: I recommend 
building a packet with an empty payload which looks like the actual flow 
(with the header fields from the flow being tested instead of the rbridge 
channel header). The IP address, mac address, L4 headers etc all match 
with that of the flow you're trying to debug. However, to ensure that any 
leaked packets get dropped, set the IP checksum to be invalid (the IP 
checksum is never used for the ecmp hash). Now send out successive packets 
with the ttl field in incrementing values. Note that you're sending a 
"pseudo-real" packet into the network and relying on received ttl expiry 
messages to figure out the path.
[Answer] Aha you are now acknowledging IP is used for ECMP hash. Now I am 
wondering how Channel header can facilitate that. The concept of utilizing 
invalid checksum is already covered in draft-tissa-trill-oam-03 Section 
6.3. Secondly using TTL in multicast is a dangerous thing, as tree depth 
can be different and packets can leak out. You may not want to trigger 
invalid IP checksum counter on your hosting servers. As that may generate 
additional support calls. Checksum invalid SHOULD only be used to protect 
against unlikely case packets leaking out. Not the default. 
Question: Are you suggesting to include IP Multicast packets in the native 
form here or encap within channel header ? 
d.      Unicast path discovery: the erbridge ID is that of the switch 
you're trying to ping. Let's send this echo request message with a fake 
inner DMAC. Construct the dmac so that the U/L bit is 1 and the I/G bit is 
0 (basically, its a local scope unicast mac). Set the last 32 bits to a 
random value. This is a fake mac address, but that's ok because the 
network routes only on the erbridge anyway. The M bit in the TRILL header 
is 0. Send this as an echo request message with incrementing ttl values. 
This gives you "a" path from the source to the destination rbridge. 
Increment the dmac used earlier, and repeat. All rbridges hash at least on 
the inner dmac, so this will allow you to enumerate all possible paths 
from source to destination.
[Answer] You are making big assumption here. Firstly hash is not defined 
in any standard and it is implementation depended. Some devices may use 
SMAC:DMAC VLAN combination, Some may use IP flow information. May be low 
ends use just DMAC. In anycase selecting Random values are not going to 
gurentee a covering all paths. Secondly one may not even know how many 
ECMP options available on an RBridge 3 hops away. This is the reason more 
elaborate methods are required RFC 4379 explain How this can be done in 
MPLS. Based on the feedback received during the last WG session in Taipei, 
we have also enhanced to included this. Please see section 9.9. This 
needed to be done in more systematic manner rather than randomly 
generating DMAC. DMAC means Destination MAC. SMAC means Source MAC
e.      Unicast path discovery for a particular flow: Similar to what we 
did for multicast, construct a real packet of length 0. The IP address, 
mac address, L4 headers etc all match with that of the flow you're trying 
to debug. However, to ensure that any leaked packets get dropped, set the 
IP checksum to be invalid. Now set the ttl field in incrementing values. 
Note that you're sending a "pseudo-real" packet into the network and 
relying on received ttl expiry messages to figure out the path.
[Answer] This same concept is in section 6.3 of draft-tissa-trill-oam. 
Note that doing this eliminates the reliance on IP/ICMP, while preserving 
the objectives of draft-tissa-trill-oam. The fake packets generated are 
technically not OAM packets, just suggestions on how to achieve a certain 
objective.

3) Fault notification: although neither draft talks about this, this is a 
part of the ethernet OAM feature set. I think draft-ietf-trill-rbridge-oam 
needs to mention that the expectation is that BFD will work over the 
rbridge channel to achieve this.

4) draft-tissa-trill-oam  has a big section on TTM. The equivalent in 
Ethernet OAM is performance management by checking counters and 
send/receive delays, etc. We can cover send/receive delay/jitter 
measurements using ping (as described earlier). I frankly don't see the 
benefit to including remote discovery of counters in this OAM proposal. 
The administrator can always just login to the rbridge of interest and get 
a read-out of the counters directly, or use snmp remotely. We shouldnt be 
replicating functionality which is better done elsewhere, and I think this 
entire section on TTM in draft-tissa-trill-oam should be taken out.

[Answer] Your understanding is incorrect. TTM  is utilized not for 
counters but to ensure that specific flows are reaching specifc devices. 
This is included as part of feedback received from the WG in Tapei. This 
is an adaptation of Data Driven CFM .

5) I don't see any benefits to DRB/AF discovery OAM messages. This is 
probably done most easily by logging into the rbridge in question and 
doing a "show l2 isis", or extending the SNMP MIB. 

6) draft-tissa-trill-oam has a section on MAC discovery (basically, being 
able to query all rbridges to see what their rbridge<->mac binding is). 
This is again something that should be best queried using SNMP - this is a 
classic MIB walk case.
[Answer] How do you know which device is having the MAC so you can do the 
walk ?

7) Address binding verification should be out of scope for a TRILL OAM 
draft, and besides, should just be queried at the rbridge with the CLI.
[Answer] Firstly how do you know which device to login to use CLI. 
Secondly, anything that improve usability and experience of users can be 
always included.

8) End-Station Attachment Point Discovery: should be out of scope of this 
draft, and is too vendor specific (it uses VN-tags).

[Answer] Terminology can be changed to include what is defined in VXLAN, 
NVGRE etc. The case in point is when there are other overlay technologies 
how to identify different address association. Certain VM may use the same 
MAC address to associate with different VXLAN segments.

best,
Sunny Rajagopalan

---

From: Sam Aldrin <[email-masked]>
Date: Tue, 13 Mar 2012 11:35:55 -0700

See inline for my response to your questions.

-sam

Sent from my iPad

On Mar 12, 2012, at 3:06 PM, Santosh Rajagopalan <[email-masked]> wrote:

> 1) Let's say that you are trying to find all possible paths between two rbridges (or liveliness) for a)unicast, b) multicast (unpruned) and c) multicast (pruned). My proposal  below will let you discover all three, without needing IP.  Since we're using the rbridge channel ethertype, rbridges can trap these and prevent them from getting to hosts at the edge. Note that we rely on unicast forwarding of rbridge channel packets for unicast discovery and multicast forwarding of rbridge channel packets for multicast discovery. This happens naturally in any trill compliant switch. 
%sam - Discovering all of the paths is one part and discovering a path a specific flow takes is another problem. What we are trying to say is channel draft does not allow the full flexibility, for Unicast and multicast combinations. As was answered in the original email by Tissa, that multicast pruning happens with IP VLAN combinations with channel drafts such things are not possible. IP invalid checksum method has danger of triggering invalid packet counters  and lead to administration nightmare in hosting providers. We encourage you to check this with providers. The same was attempted while doing OAM for PW's etc, and received a strong no-no.
> 
> 2) Let's say that you already have the information from 1) above, but you are trying to figure out which of the many paths you know of will be used for a particular flow. Now, draft-tissa-trill-oam assumes that the inner ethernet header is never used for hashing for an IP flow. It assumes that as long as you keep the L3 and L4 fields the same as that of the original packet, hardware will hash to the same values as that from the original flow, even though the L2 header is different ("6.2. Identification of IP Flows" in  draft-tissa-trill-oam ). I do not think this is a valid assumption to make. The only way to make sure your probe packet takes the same path as a regular packet is to make it identical to the original packet, all the way including the L2 headers. The IP checksum is never used for hashing, so you can invalidate it to make sure such packets get dropped at the host. This is the approach I was advocating for doing path discovery for a particular flow. 
%sam- You have not read section 6..3 of draft-tissa-trill-oam, (Tissa pointed this out in his earlier response as well). Section 6.3 is all about using unmodified header. Please read section 6.3. Also this method cannot be used for Multicast as answered above will trigger invalid packet count.
> 
> 3) The only hash behavior we can be sure of is that if you give any asic a non-IP packet, it will do load balancing using the only available fields, which is ethernet smac + dmac. I'm exploiting this property to do path discovery (the rbridge ethertype makes asics treat these packets as non-IP). Therefore, if you vary the inner dmac, you can exercise all the available paths. Paths you discover using this approach are applicable to both IP and non-IP traffic, since trill routes are common for both. The only hash assumption I've made is that L2 unknown ethertype (rbridge channel)  packets get load balanced using the inner dmac/smac. I am making no assumption about the fields used by the hardware for hashing IP packets. 
%sam- Are you suggesting ingress node to randomely change the DMAC, Point is how you going to randomly change a DMAC and ensure it is taking a specific path. ? 
> 
> 4) People adopting TRILL are more likely to come from an ethernet background than from an MPLS background. For our target audience, it's important for OAM to look like what they're already familiar with, which is ethernet OAM. TRILL competes with SPB for the same space, and note that they make a virtue of being ethernet OAM compatible. Do many people on this list feel MPLS OAM is the way to go for us? 
%sam - that is your assumption, not necessary that others share the same view. OAM, irrespective of different layers, share common characteristics. Reusing what we learnt in other areas is very useful. Having said that, folks who use OAM do not see how it is implemented, rather it is for the implementor who cares for it. If you go by your logic, mpls OAM is not right because it uses IP model and mpls is not IP.
Trill as such has evolved by bringing ip and bridging technologies together, with best of both in each category. Just because one likes Ethernet OAM, doesn't mean, anything to do with Ethernet , should always do Ethernet OAM. The same argument was used for mpls TP OAM as well. We all know the result.
The proposal here is to use the proven methodology and is working. If it is solving problem, why reinvent the wheel all over again. Better make use of it. If the solution cannot work, then we can discuss about those points, rather than predefining the solution to fit based on what one feels.
As far as Qbp goes, the concept of flows and it's identification is different. You cannot just compare as is for TRILL. If you want, we can discuss in that list, to get complete perspective.
> 
> 5) I'm assuming that when operators lay out a network, they have access to a picture that shows the interconnects (or at least have some management tool speaking snmp that does). So when they need to discover the DRB on a multi-access link, then can send an SNMP query to the rbridges connected on that link to get the DRB information. This also goes for finding the mac addresses on a given rbridge, finding the address bindings at a particular rbridge etc. All of these "property query" type messages are exactly what SNMP was made for. Some things like finding the vn-tag, vxlan ID, nvgre ID etc are within the domain of SNMP but out of scope for TRILL. What is trill-specific about these? 
%sam - if everything is available through Snmp, even performing OAM functions is redundant. Basing a model on Snmp is expensive and intensive. If we go by the argument of yours, Ethernet OAM, why do you need to export performance stats in band, when you can query the same using snmp?
My answer is simple. If one feels the need to export data or query data, we provide a mechanism without relying solely on SNMP. While we find it useful, everyone may not, that doesn't mean, it is not useful. If a vendor chooses not to implement, it is fine.
> 
> I haven't responded to any statements below which are based on a misunderstanding of my original email, please re-review it and I'll be happy to answer questions on it. 
I do not see what was not understood. You can clarify if something was not addressed right.

Sam
> Subject:        Re: [rbridge] merging draft-tissa-trill-oam        anddraft-ietf-trill-rbridge-oam 
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Santosh Rajagopalan

---

From: Santosh Rajagopalan <[email-masked]>
Date: Thu, 15 Mar 2012 15:48:47 -0600

I think that we're starting to talk past each other. May I suggest taking 
each of the ping/traceroute situations on a case-by-case basis to see how 
each proposal handles it? 

Basically, can one of the authors summarize the approach in 
draft-tissa-trill-oam for ping/traceroute to discover:

a. multidestination unpruned trees.

b. multidestination pruned trees (per multicast group).

c. tracing the path taken by a real multicast flow.

d. unicast path discovery for each multipath

e. unicast path discovery for real flow.

I understand this will be a restatement of information that's present in 
draft-tissa-trill-oam, and I apologize for that. I'm looking for a more 
concise presentation of the approach. I am having trouble arranging the 
information in draft-tissa-trill-oam into the categories I've suggested 
above. You can use as a template the email I sent out originally starting 
this thread.

As for the discussion below about whether properties should be queried via 
SNMP or ICMP OAM, I would simply suggest that we make a choice one way or 
the other. After all, if you add the ability to query mac addresses, DRBs 
etc to OAM (and these are already present in draft-ietf-trill-rbridge-mib
), then why stop there? Why not incorporate everything in the SNMP MIB 
into the ICMP query framework? After all, if SNMP is truly "expensive and 
intensive", why stop with DRB queries? I would then suggest doing away 
with the trill mib and using icmp as the query mechanism for all 
properties. Really, I'm ok either way, but I think we should pick one so 
that people don't end up implementing the same thing twice.

--
Sunny Rajagopalan

From:   Sam Aldrin <[email-masked]>
To:     Santosh Rajagopalan/Santa Clara/IBM@IBMUS
Cc:     "[email-masked]" <[email-masked]>
Date:   03/13/2012 11:59 AM
Subject:        Re: [rbridge] merging draft-tissa-trill-oam 
anddraft-ietf-trill-rbridge-oam
Sent by:        [email-masked]

-sam

Sent from my iPad

On Mar 12, 2012, at 3:06 PM, Santosh Rajagopalan <
[email-masked]> wrote:

1) Let's say that you are trying to find all possible paths between two 
rbridges (or liveliness) for a)unicast, b) multicast (unpruned) and c) 
multicast (pruned). My proposal  below will let you discover all three, 
without needing IP.  Since we're using the rbridge channel ethertype, 
rbridges can trap these and prevent them from getting to hosts at the 
edge. Note that we rely on unicast forwarding of rbridge channel packets 
for unicast discovery and multicast forwarding of rbridge channel packets 
for multicast discovery. This happens naturally in any trill compliant 
switch. 
%sam - Discovering all of the paths is one part and discovering a path a 
specific flow takes is another problem. What we are trying to say is 
channel draft does not allow the full flexibility, for Unicast and 
multicast combinations. As was answered in the original email by Tissa, 
that multicast pruning happens with IP VLAN combinations with channel 
drafts such things are not possible. IP invalid checksum method has danger 
of triggering invalid packet counters  and lead to administration 
nightmare in hosting providers. We encourage you to check this with 
providers. The same was attempted while doing OAM for PW's etc, and 
received a strong no-no.

2) Let's say that you already have the information from 1) above, but you 
are trying to figure out which of the many paths you know of will be used 
for a particular flow. Now, draft-tissa-trill-oam assumes that the inner 
ethernet header is never used for hashing for an IP flow. It assumes that 
as long as you keep the L3 and L4 fields the same as that of the original 
packet, hardware will hash to the same values as that from the original 
flow, even though the L2 header is different ("6.2. Identification of IP 
Flows" in  draft-tissa-trill-oam ). I do not think this is a valid 
assumption to make. The only way to make sure your probe packet takes the 
same path as a regular packet is to make it identical to the original 
packet, all the way including the L2 headers. The IP checksum is never 
used for hashing, so you can invalidate it to make sure such packets get 
dropped at the host. This is the approach I was advocating for doing path 
discovery for a particular flow. 
%sam- You have not read section 6..3 of draft-tissa-trill-oam, (Tissa 
pointed this out in his earlier response as well). Section 6.3 is all 
about using unmodified header. Please read section 6.3. Also this method 
cannot be used for Multicast as answered above will trigger invalid packet 
count.

%sam- Are you suggesting ingress node to randomely change the DMAC, Point 
is how you going to randomly change a DMAC and ensure it is taking a 
specific path. ? 

4) People adopting TRILL are more likely to come from an ethernet 
background than from an MPLS background. For our target audience, it's 
important for OAM to look like what they're already familiar with, which 
is ethernet OAM. TRILL competes with SPB for the same space, and note that 
they make a virtue of being ethernet OAM compatible. Do many people on 
this list feel MPLS OAM is the way to go for us? 
%sam - that is your assumption, not necessary that others share the same 
view. OAM, irrespective of different layers, share common characteristics. 
Reusing what we learnt in other areas is very useful. Having said that, 
folks who use OAM do not see how it is implemented, rather it is for the 
implementor who cares for it. If you go by your logic, mpls OAM is not 
right because it uses IP model and mpls is not IP.
Trill as such has evolved by bringing ip and bridging technologies 
together, with best of both in each category. Just because one likes 
Ethernet OAM, doesn't mean, anything to do with Ethernet , should always 
do Ethernet OAM. The same argument was used for mpls TP OAM as well. We 
all know the result.
The proposal here is to use the proven methodology and is working. If it 
is solving problem, why reinvent the wheel all over again. Better make use 
of it. If the solution cannot work, then we can discuss about those 
points, rather than predefining the solution to fit based on what one 
feels.
As far as Qbp goes, the concept of flows and it's identification is 
different. You cannot just compare as is for TRILL. If you want, we can 
discuss in that list, to get complete perspective.

5) I'm assuming that when operators lay out a network, they have access to 
a picture that shows the interconnects (or at least have some management 
tool speaking snmp that does). So when they need to discover the DRB on a 
multi-access link, then can send an SNMP query to the rbridges connected 
on that link to get the DRB information. This also goes for finding the 
mac addresses on a given rbridge, finding the address bindings at a 
particular rbridge etc. All of these "property query" type messages are 
exactly what SNMP was made for. Some things like finding the vn-tag, vxlan 
ID, nvgre ID etc are within the domain of SNMP but out of scope for TRILL. 
What is trill-specific about these? 
%sam - if everything is available through Snmp, even performing OAM 
functions is redundant. Basing a model on Snmp is expensive and intensive. 
If we go by the argument of yours, Ethernet OAM, why do you need to export 
performance stats in band, when you can query the same using snmp?
My answer is simple. If one feels the need to export data or query data, 
we provide a mechanism without relying solely on SNMP. While we find it 
useful, everyone may not, that doesn't mean, it is not useful. If a vendor 
chooses not to implement, it is fine.

I haven't responded to any statements below which are based on a 
misunderstanding of my original email, please re-review it and I'll be 
happy to answer questions on it. 
I do not see what was not understood. You can clarify if something was not 
addressed right.

Sam

best regards, 
Sunny Rajagopalan 

  

  
Hi,

c.        Multicast pruned tree discovery for a particular flow: I 
recommend building a packet with an empty payload which looks like the 
actual flow (with the header fields from the flow being tested instead of 
the rbridge channel header). The IP address, mac address, L4 headers etc 
all match with that of the flow you're trying to debug. However, to ensure 
that any leaked packets get dropped, set the IP checksum to be invalid 
(the IP checksum is never used for the ecmp hash). Now send out successive 
packets with the ttl field in incrementing values. Note that you're 
sending a "pseudo-real" packet into the network and relying on received 
ttl expiry messages to figure out the path. 

best,
Sunny Rajagopalan

---

From: Sam Aldrin <[email-masked]>
Date: Thu, 15 Mar 2012 15:08:48 -0700

Inline for my reply.

Sent from my iPhone

On Mar 15, 2012, at 2:48 PM, Santosh Rajagopalan <[email-masked]> wrote:

> I think that we're starting to talk past each other. May I suggest taking each of the ping/traceroute situations on a case-by-case basis to see how each proposal handles it? 
Draft is meant for that. If you have specific case where you feel it doesn't solve it, ask it away. Will be glad to answer them. Last thing we should do is to make it a design document. 
> 
> Basically, can one of the authors summarize the approach in draft-tissa-trill-oam for ping/traceroute to discover: 
> 
> I understand this will be a restatement of information that's present in draft-tissa-trill-oam, and I apologize for that. I'm looking for a more concise presentation of the approach. I am having trouble arranging the information in draft-tissa-trill-oam into the categories I've suggested above. You can use as a template the email I sent out originally starting this thread. 
We plan to to present at ietf, again. Hopefully we can clarify them, there. As the status is still draft,we plan to add more content. We will definitely like to add data, if not present, to care for majority of the scenarios if not all.
> 
> As for the discussion below about whether properties should be queried via SNMP or ICMP OAM, I would simply suggest that we make a choice one way or the other. After all, if you add the ability to query mac addresses, DRBs etc to OAM (and these are already present in draft-ietf-trill-rbridge-mib), then why stop there? Why not incorporate everything in the SNMP MIB into the ICMP query framework?
Why just stop with mib?
> After all, if SNMP is truly "expensive and intensive", why stop with DRB queries? I would then suggest doing away with the trill mib and using icmp as the query mechanism for all properties.
You should ask mib authors that question. If you go by your logic, if snmp is present, we should not attempt any with other technology, including net conf.
> Really, I'm ok either way, but I think we should pick one so that people don't end up implementing the same thing twice. 
The goal is not to duplicate the effort, but to get something operationally useful to fulfill oam functions. We feel differently to what you think. It is not a magic wand trying to make others vanish, rather give additional tools to manage networks.

Sam
> http://mailman.postel.org/mailman/listinfo/rbridge_______________________________________________
> rbridge mailing list
> [email-masked]

---

From: "Rohit Watve (rwatve)" <[email-masked]>
Date: Fri, 16 Mar 2012 18:16:26 -0700

Hi Santosh,

I had a question about following:

2. c) Multicast pruned tree discovery for a particular flow: I recommend building a packet with an empty payload which looks like the actual flow (with the header fields from the flow being tested instead of the rbridge channel header). The IP address, mac address, L4 headers etc all match with that of the flow you're trying to debug. However, to ensure that any leaked packets get dropped, set the IP checksum to be invalid (the IP checksum is never used for the ecmp hash). Now send out successive packets with the ttl field in incrementing values. Note that you're sending a "pseudo-real" packet into the network and relying on received ttl expiry messages to figure out the path.

How do you prevent OAM packets with Invalid IP checksum from leaking out to customers?

Thanks
Rohit

 

Hi,

best,
Sunny Rajagopalan

---

From: Santosh Rajagopalan <[email-masked]>
Date: Mon, 19 Mar 2012 10:48:29 -0700

Hi, Rohit
You could have a packet or two leak to the host, and the host OS will drop 
it because of the invalid checksum (this number of drops would be at the 
level of noise for most systems). A more difficult alternative would be to 
have switches drop packets with an invalid IP checksum from leaving 
host-facing ports.

1) You employ this method only after all your other means of debugging 
have failed to root cause the issue. This means that you've already used 
the means outlined below to trace the path that this packet should  be 
taking (2.a and 2.b below, which use the rbridge channel header and result 
in no leaks). 

2) After you've used 2.a and 2.b, you need to check what the hardware on 
the path is really doing with a packet that looks like the flow you're 
debugging. For this, you cannot change any of the familiar header fields 
to help you capture the packet at the egress, because that could change 
the hash result on the way (This isn't theoretical - I know of asics that 
hash on IP, trill header, tcp/udp, inner ethernet header, outer ethernet 
header). Since you need a probe packet that looks just like a real flow, 
your best bet is to clone the real flow packet, but with an invalid IP 
checksum and then send it out with incrementing ttl values.

--
Sunny Rajagopalan

From:   "Rohit Watve (rwatve)" <[email-masked]>
To:     Santosh Rajagopalan/Santa Clara/IBM@IBMUS, <[email-masked]>
Date:   03/16/2012 06:38 PM
Subject:        Re: [rbridge] merging draft-tissa-trill-oam 
anddraft-ietf-trill-rbridge-oam
Sent by:        [email-masked]

Hi Santosh,
 
2. c) Multicast pruned tree discovery for a particular flow: I recommend 
 
How do you prevent OAM packets with Invalid IP checksum from leaking out 
to customers?
 
Thanks
Rohit

 
Hi,

best,
Sunny Rajagopalan_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge

---

From: "Tissa Senevirathne (tsenevir)" <[email-masked]>
Date: Mon, 19 Mar 2012 14:01:56 -0700

Santosh

One  cannot assume when troubleshooting an issue it will be only be
limited to few packets. (we all will be very happy if it was the case
but it never the norm but rather the exception) Secondly in hosting
environments, where network provider has no control on end systems this
kind of things generate unwanted support calls.  We should not be
designing OAM protocol based on this assumption.

Also 2.a and 2.b will not allow you to verify IP payload, something
working for 2.a and 2.b does not gurantee that IP is working.

From: [email-masked] [mailto:[email-masked]] On
Behalf Of Santosh Rajagopalan
Sent: Monday, March 19, 2012 10:48 AM
To: Rohit Watve (rwatve)
Cc: [email-masked]; [email-masked]
Subject: Re: [rbridge] mergingdraft-tissa-trill-oam
anddraft-ietf-trill-rbridge-oam

Hi, Rohit 
You could have a packet or two leak to the host, and the host OS will
drop it because of the invalid checksum (this number of drops would be
at the level of noise for most systems). A more difficult alternative
would be to have switches drop packets with an invalid IP checksum from
leaving host-facing ports. 

1) You employ this method only after all your other means of debugging
have failed to root cause the issue. This means that you've already used
the means outlined below to trace the path that this packet should  be
taking (2.a and 2.b below, which use the rbridge channel header and
result in no leaks). 

2) After you've used 2.a and 2.b, you need to check what the hardware on
the path is really doing with a packet that looks like the flow you're
debugging. For this, you cannot change any of the familiar header fields
to help you capture the packet at the egress, because that could change
the hash result on the way (This isn't theoretical - I know of asics
that hash on IP, trill header, tcp/udp, inner ethernet header, outer
ethernet header). Since you need a probe packet that looks just like a
real flow, your best bet is to clone the real flow packet, but with an
invalid IP checksum and then send it out with incrementing ttl values. 

-- 
Sunny Rajagopalan 

To:        Santosh Rajagopalan/Santa Clara/IBM@IBMUS,
<[email-masked]> 

________________________________

Hi Santosh, 
  
I had a question about following: 
2. c) Multicast pruned tree discovery for a particular flow: I recommend
building a packet with an empty payload which looks like the actual flow
(with the header fields from the flow being tested instead of the
rbridge channel header). The IP address, mac address, L4 headers etc all
match with that of the flow you're trying to debug. However, to ensure
that any leaked packets get dropped, set the IP checksum to be invalid
(the IP checksum is never used for the ecmp hash). Now send out
successive packets with the ttl field in incrementing values. Note that
you're sending a "pseudo-real" packet into the network and relying on
received ttl expiry messages to figure out the path. 

Thanks
Rohit 
  
From: [email-masked] [mailto:[email-masked]
<mailto:[email-masked]> ] On Behalf Of Santosh Rajagopalan
Sent: Tuesday, March 06, 2012 8:49 AM
To: [email-masked]
Cc: Santosh Rajagopalan
Subject: [rbridge] merging draft-tissa-trill-oam
anddraft-ietf-trill-rbridge-oam 

Hi,

draft-tissa-trill-oam has a dependancy on IP/ICMP/ARP etc.  The major
reason for this dependency seems to be for multipath path discovery, and
I have some suggestions as to how to fix this in
draft-ietf-trill-rbridge-oam without using IP. draft-tissa-trill-oam
also has several features which should be rightly queried from a MIB.

I think the OAM draft should concentrate on facilitating actions that
would be difficult or impossible to perform in its absence. So querying
counters has no place here, because you could easily figure them out by
doing an SNMP query or by logging into the switch. I also think it's
worthwhile to look at the features traditionally supported in ethernet
OAM and to see if they have a place here.

1) Ping/loopback: draft-ietf-trill-rbridge-oam handles this well enough
(it does need a multicast addition). The purpose of ping is to verify
connectivity between two rbridges, not to discover the various ways of
connecting the two.  One suggestion I have here is to be able to include
a cookie in the echo request message which gets reflected back by the
receiver. This cookie could be used to include the sender's timestamp in
the ping message. The response could then be used to measure round-trip
path latency and jitter. If the sender and receiver's clocks are
synchronized using some external mechanism then this can also be used to
measure one-way delay and jitter. This is a feature borrowed from
ethernet OAM.

2) Traceroute: IP traceroute doesn't rely on icmp for the outgoing
packet (you could use ICMP, but that's a different matter). In fact,
UNIX implementations usually just send UDP packets with port numbers in
the 33K range, with appropriate TTL values. Note that traceroute in the
IP world gives you "a" path to a destination, not "all possible paths".
However, being able to enumerate all possible paths, as well as to
pinpoint the particular path a flow would take are good to have
features. 
Here're the modifications needed in draft-ietf-trill-rbridge-oam to
achieve this: 
a.        Multicast unpruned tree discovery: you can discover all
existing unpruned multicast trees by setting the erbridge value of an
rbridge channel message to the respective roots, M=1 and the dmac to
"all-egress-rbridges". Just like regular ping, you can send this as an
echo request message with incrementing ttl values, and the ttl expiry
messages you get will tell you what the unpruned tree looks like.
Changing the egress_rbridge will paint you a picture of the different
trees. 
b.        Multicast pruned tree discovery: just do a), but replace the
"all-egress-rbridges" multicast address in the inner dmac with the mac
address of the multicast group. As you vary the egress_rbridge field,
this will enumerate all possible pruned trees for that group. Note that
draft-ietf-trill-rbridge-oam uses draft-ietf-trill-rbridge-channel,
which requires the inner dmac to be all-egress-rbridges. However, the
ethertype field can be used by itself to classify an OAM packet, and I
am advocating that the inner dmac requirement in
draft-ietf-trill-rbridge-channel be relaxed to allow for alternative
inner dmacs where needed. 
c.        Multicast pruned tree discovery for a particular flow: I
recommend building a packet with an empty payload which looks like the
actual flow (with the header fields from the flow being tested instead
of the rbridge channel header). The IP address, mac address, L4 headers
etc all match with that of the flow you're trying to debug. However, to
ensure that any leaked packets get dropped, set the IP checksum to be
invalid (the IP checksum is never used for the ecmp hash). Now send out
successive packets with the ttl field in incrementing values. Note that
you're sending a "pseudo-real" packet into the network and relying on
received ttl expiry messages to figure out the path. 
d.        Unicast path discovery: the erbridge ID is that of the switch
you're trying to ping. Let's send this echo request message with a fake
inner DMAC. Construct the dmac so that the U/L bit is 1 and the I/G bit
is 0 (basically, its a local scope unicast mac). Set the last 32 bits to
a random value. This is a fake mac address, but that's ok because the
network routes only on the erbridge anyway. The M bit in the TRILL
header is 0. Send this as an echo request message with incrementing ttl
values. This gives you "a" path from the source to the destination
rbridge. Increment the dmac used earlier, and repeat. All rbridges hash
at least on the inner dmac, so this will allow you to enumerate all
possible paths from source to destination. 
e.        Unicast path discovery for a particular flow: Similar to what
we did for multicast, construct a real packet of length 0. The IP
address, mac address, L4 headers etc all match with that of the flow
you're trying to debug. However, to ensure that any leaked packets get
dropped, set the IP checksum to be invalid. Now set the ttl field in
incrementing values. Note that you're sending a "pseudo-real" packet
into the network and relying on received ttl expiry messages to figure
out the path. 
Note that doing this eliminates the reliance on IP/ICMP, while
preserving the objectives of draft-tissa-trill-oam. The fake packets
generated are technically not OAM packets, just suggestions on how to
achieve a certain objective.

3) Fault notification: although neither draft talks about this, this is
a part of the ethernet OAM feature set. I think
draft-ietf-trill-rbridge-oam needs to mention that the expectation is
that BFD will work over the rbridge channel to achieve this.

4) draft-tissa-trill-oam  has a big section on TTM. The equivalent in
Ethernet OAM is performance management by checking counters and
send/receive delays, etc. We can cover send/receive delay/jitter
measurements using ping (as described earlier). I frankly don't see the
benefit to including remote discovery of counters in this OAM proposal.
The administrator can always just login to the rbridge of interest and
get a read-out of the counters directly, or use snmp remotely. We
shouldnt be replicating functionality which is better done elsewhere,
and I think this entire section on TTM in draft-tissa-trill-oam should
be taken out.

6) draft-tissa-trill-oam has a section on MAC discovery (basically,
being able to query all rbridges to see what their rbridge<->mac binding
is). This is again something that should be best queried using SNMP -
this is a classic MIB walk case.

8) End-Station Attachment Point Discovery: should be out of scope of
this draft, and is too vendor specific (it uses VN-tags).

best,
Sunny Rajagopalan_______________________________________________
rbridge mailing list
[email-masked]
http://mailman.postel.org/mailman/listinfo/rbridge
<http://mailman.postel.org/mailman/listinfo/rbridge>

---

From: Sam Aldrin <[email-masked]>
Date: Mon, 19 Mar 2012 14:42:32 -0700

Hi Santosh,

Be it one packet or two, leaking OAM packets to the hosts is big no no in any kind of networks.
Secondly, order of doing OAM, like doing trace after ping etc, is more of a procedural thing than specification. One cannot just assume every user does the same, given the tools at hand.

Cheers
Sam

Sent from my iPad

On Mar 19, 2012, at 10:48 AM, Santosh Rajagopalan <[email-masked]> wrote:

> Hi, Rohit 
> You could have a packet or two leak to the host, and the host OS will drop it because of the invalid checksum (this number of drops would be at the level of noise for most systems). A more difficult alternative would be to have switches drop packets with an invalid IP checksum from leaving host-facing ports. 
> 
> 1) You employ this method only after all your other means of debugging have failed to root cause the issue. This means that you've already used the means outlined below to trace the path that this packet should  be taking (2.a and 2.b below, which use the rbridge channel header and result in no leaks). 
> 
> 2) After you've used 2.a and 2.b, you need to check what the hardware on the path is really doing with a packet that looks like the flow you're debugging. For this, you cannot change any of the familiar header fields to help you capture the packet at the egress, because that could change the hash result on the way (This isn't theoretical - I know of asics that hash on IP, trill header, tcp/udp, inner ethernet header, outer ethernet header). Since you need a probe packet that looks just like a real flow, your best bet is to clone the real flow packet, but with an invalid IP checksum and then send it out with incrementing ttl values. 
> 
> -- 
> Sunny Rajagopalan 
> 
> 
> 
>