Subject: [manet] Request to consider for DLEP

From: "Snyder, Brian" <[email-masked]>
Date: Thu, 24 Jul 2014 14:02:37 +0000

Hi all - had a couple conversations with folks yesterday and did some reading and one thing I'd like to request be added into DLEP is the ability to set the equivalent of a default destination.  Analogous to a default route at L3 layer.   IE: In the hub and spoke architecture (mostly all satellite networks - or any non-mesh design)  there are networking guarantees that we can take for granted.    Essentially, we know the spoke will only ever have one direction to pass traffic, so there is no need to manage or teach the router about connected destinations (mac-addresses).   This could be accomplished in either 2 ways... one a specific mac address 0xFFFFFFFF could be used to say, you can send me broadcasts - which if interpreted the way I'd like - would tell a router - go ahead and start your multicast IGP discovery (or if configured your unicast arping) to find any/all L3 routing peers.  The advantage in my opinion to allowing for this:

*        Simplify router operations in this mode. Because the radio already knows he is a spoke - there would otherwise be no need to track hub mac addresses because he knows if a destination is not on his local lan - it must go over the air.

*        Tracking hub side mac addresses in a modem can become a very large scale problem.  There will be thousands to a hundred thousand of learnt mac addresses coming from the hub side (all CE routers behind modems) could get learnt.

*        Without special config the modem can't know which MAC is special to be the hub and which isn't, so he'd have to manage it all.

*        With special config there are complications in OAM. If something changes there is either a lot of configuration to modify and transfer. This would be an OAM nightmare of potential errors.

*        It would require "special" work and knowledge for the modems to track, that break the layers - they are trying to simply be a transparent bridge.

I don't think this is requesting any special burden on the routers - all I'm asking is they behave exactly as they would as if a physical cable was just plugged in - but we'll have DLEP virtually controlling the up/down of the interface.  When set to up it would start to learn/converge like it would if dlep was uninvolved.

My main motivator, in addition to simplicity, is that those of us in the sat industry want to simplify or automate all things related to the remote modem, lan, and equipment behind it.  This is because years of experience tells us something that can possibly go wrong at one point in time - will - (I'm sure we've all learned that lesson) but the difference is if something goes wrong on a sat lan - its not normal to have IT people nearby - and to 'fix' it could take a plane flight or a lot of local coordinating. Multiply that by thousands of modems - and if we can avoid doing something special or needing to configure something - we sure would prefer it.

NOTE: I understand on the hub side we would need specific destination to demux the dlep connections because the hub side will peer will all modem lans.  This is okay to me because complexity at the hub is acceptable - that's managed by a trained NOC - so if things go wrong, people can deal with it locally.... also much more powerful equipment and in general the complication is needed there because demuxing must occur. At the modem side, there is no demuxing in non-mesh networks.

I hope I've clearly explained what I'm thinking.  ;)

Thanks for the consideration,
Brian

Brian Snyder Snr. Applied Researcher  * p: +1 703.648.8228 c: +1 703.220.6782
iDirect, 13861 Sunrise Valley Drive, Suite 300, Herndon, VA 20171 * www.idirect.net * Advancing a Connected World

---

From: Rick Taylor <[email-masked]>
Date: Thu, 24 Jul 2014 15:46:32 +0000

As one of the people involved in the extensive discussions with Brian over this use case, and it was I who suggested the FF:FF:FF:FF:FF:FF solution, I thought I ought to justify myself.

Yes, it's a cludge.  In an ideal world, the spoke modem in a hub and spoke topology would announce it's link-state using the MAC address of the hub as the destination identifier in the relevant signals.  But, as Brian points out, there are examples of existing technologies where a-priori configuration of the hub MAC, or retrofitting the passing of the hub MAC in the link-layer signal is infeasible.  In these cases, there needs to be a way for the spoke modem to announce the link-state of the one and only link it has in a generic way.  Given the lack of knowledge of the hub MAC, any attached router will need to ARP/ND for it anyway, using the FF:FF:FF:FF:FF:FF address, so there is some consistency there.

Even if the all-FFs MAC is being used as a destination, there is nothing preventing the same modem announcing other MAC destinations, with alternate link state at a another time, so it doesn't break the existing DLEP model, in the same way that layer-3 has the default route concept (but obviously with MAC's there is no concept of longest prefix, etc...)

I think there needs to be some verbiage somewhere that states something along the lines of "If a modem will only ever have one link, e.g. a spoke in a hub and spoke topology, and the MAC address of the destination router cannot be determined by the lower layer signalling, then the FF:FF:FF:FF:FF:FF MAC address MAY be used, but this is NOT RECOMMENDED" (i.e. Do it properly unless you *really* can't).

@Brian: I think you might be arguing yourself into writing an informational draft entitles "DLEP use-cases for Satcom".

Also, as I think about it, this model might work for the 3G access point use-case.  There is a link to a single destination, but, as with Satcom, the MAC address of the destination router may not be available.  Although, on further consideration (I'm definitely thinking on my feet here) I think the xG/LTE use-case is better addressed by my layer-3 extensions, as access-points don't operate as layer-2 bridges, and there may not even be a DLEP router at the destination.

Anyway, all food for thought and discussion...

Rick

---

From: Henning Rogge <[email-masked]>
Date: Thu, 24 Jul 2014 17:57:03 +0200

On Thu, Jul 24, 2014 at 5:46 PM, Rick Taylor
<[email-masked]> wrote:
> As one of the people involved in the extensive discussions with Brian over
> this use case, and it was I who suggested the FF:FF:FF:FF:FF:FF solution, I
> thought I ought to justify myself.

Hmm...

> Yes, it's a cludge.  In an ideal world, the spoke modem in a hub and spoke
> topology would announce it's link-state using the MAC address of the hub as
> the destination identifier in the relevant signals.  But, as Brian points
> out, there are examples of existing technologies where a-priori
> configuration of the hub MAC, or retrofitting the passing of the hub MAC in
> the link-layer signal is infeasible.

So there is no regular multicast traffic from the hub (which would be
reasonable cheap because there is only a single hub)?

Can the spokes hear the traffic from the Hub to other spokes? This
would deliver the MAC address of the Hub too...

> In these cases, there needs to be a
> way for the spoke modem to announce the link-state of the one and only link
> it has in a generic way.  Given the lack of knowledge of the hub MAC, any
> attached router will need to ARP/ND for it anyway, using the
> FF:FF:FF:FF:FF:FF address, so there is some consistency there.

So the idea is start only with a broadcast target and then later
announce the unicast target as soon as we know it? We will get it as
soon as we receive the first data packet anyways.

I am not even sure we need to modify DLEP in any ways to support this
hub&spoke topology. Even at the hub, DLEP should be fine to announce
the coming and going of spoke modems as detected by the linklayer...
this might be a scenario where Stans 'stateful' solution is really
better than my older 'stateless' approach. Because we don't need to
signal anything for a spoke that stays in the network... we just
signal once "its there".

Henning Rogge

---

From: "Stan Ratliff (sratliff)" <[email-masked]>
Date: Thu, 24 Jul 2014 16:44:40 +0000

Rick/Brian,

I'm having a tough time understanding this. It appears to be a way to kludge DLEP support for an environment where it's not needed (that being the spoke router).

Maybe I need a call, or some additional information to understand what the goal is here.

Regards,
Stan

On Jul 24, 2014, at 11:46 AM, Rick Taylor <[email-masked]<mailto:[email-masked]>> wrote:

Rick

_______________________________________________
manet mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/manet

---

From: "Snyder, Brian" <[email-masked]>
Date: Thu, 24 Jul 2014 16:49:38 +0000

The other idea that was proposed was to define a new TLV at session init/discovery to tell a router - the same idea... This is a link up message -> now go converge.  Rick thought using the FFFFFFFF thing would 'merge' the ideas a little better.... My concern with that is that it could affect other decisions being made based on that, like multicast wildcard or something.  Ideally, imho,  we could have a special tlv to state : 
1)  Send all multicast this way
2)  Send all unicast this way
3) Send all broadcast this way
4) 1,2,3 can be combined

All of the above being optional -- of course one could set MAC / IP destinations if it's desired to any specific host. And these options should be combinable... for instance
if you set all multicast this way, it would use default link metrics (or whatever is set) and if you want to specify a more specific rate control on a particular destination you could do that too.

However, in SATRAN, no one really relies on the router to do any qos because it doesn't know anything that's going on  over the shared (and oversubscribed) radio link, so its much more common to let the modem handle that... (there is nothing wrong with trying to do it via router qos - but it will not work as well).

>Can the spokes hear the traffic from the Hub to other spokes? This would deliver the MAC address of the Hub too...
The outroute is broadcast, so if for instance an ARP goes downstream over the air, then every modem will see it (within the appropriate SVN definition (VLAN, qinq) namespaces) .  Now if a arp comes on the inroute then said SVN's configuration will define if other modems will see it (IE: Will the hub hairpin it?).  I can speak for my implementation here but we have the ability to per SVN define if it should behave in VPWS or VPLS mode.

> Because we don't need to signal anything for a spoke that stays in the network... we just signal once "its there".

Perhaps I misunderstood then --- I thought the way its defined if a destination wasn't set - it wouldn't transmit any data to it.... What I'm going for is more I guess bfd like (I feel the bfd proxy paper I presented is very well described as a kind of a DLEP lite -> interface up/down with no link metrics) but once set down, it just kills the routing adjency (clearing the FIB) and triggers L3 peer rediscovery to begin again.  I liked that a lot because its 0 config and makes the L2 equipment completely transparent.  If I don't have to tell the downstream router anything about what destinations he can reach when I enter the network... (yet still let L3 routing begin convergence in either direction).... then perhaps there are no changes needed.  That wasn't my impression from various discussions had.... am I wrong?

Thanks all,
Brian

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]] 
Sent: Thursday, July 24, 2014 11:57 AM
To: Rick Taylor
Cc: Snyder, Brian; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

On Thu, Jul 24, 2014 at 5:46 PM, Rick Taylor <[email-masked]> wrote:
> As one of the people involved in the extensive discussions with Brian 
> over this use case, and it was I who suggested the FF:FF:FF:FF:FF:FF 
> solution, I thought I ought to justify myself.

Hmm...

> Yes, it's a cludge.  In an ideal world, the spoke modem in a hub and 
> spoke topology would announce it's link-state using the MAC address of 
> the hub as the destination identifier in the relevant signals.  But, 
> as Brian points out, there are examples of existing technologies where 
> a-priori configuration of the hub MAC, or retrofitting the passing of 
> the hub MAC in the link-layer signal is infeasible.

So there is no regular multicast traffic from the hub (which would be reasonable cheap because there is only a single hub)?

Can the spokes hear the traffic from the Hub to other spokes? This would deliver the MAC address of the Hub too...

> way for the spoke modem to announce the link-state of the one and only 
> link it has in a generic way.  Given the lack of knowledge of the hub 
> MAC, any attached router will need to ARP/ND for it anyway, using the 

So the idea is start only with a broadcast target and then later announce the unicast target as soon as we know it? We will get it as soon as we receive the first data packet anyways.

I am not even sure we need to modify DLEP in any ways to support this hub&spoke topology. Even at the hub, DLEP should be fine to announce the coming and going of spoke modems as detected by the linklayer...
this might be a scenario where Stans 'stateful' solution is really better than my older 'stateless' approach. Because we don't need to signal anything for a spoke that stays in the network... we just signal once "its there".

Henning Rogge

---

From: "Stan Ratliff (sratliff)" <[email-masked]>
Date: Thu, 24 Jul 2014 16:59:09 +0000

On Jul 24, 2014, at 12:49 PM, "Snyder, Brian" <[email-masked]> wrote:

> The other idea that was proposed was to define a new TLV at session init/discovery to tell a router - the same idea... This is a link up message -> now go converge.  

How is this different from running the spoke router without DLEP, as it is today?  The interface comes up, and convergence starts… 

I guess another piece of the puzzle I'm missing is - "When (or under what circumstances) would this 'all destinations' modem ever issue a DLEP 'Destination Down' message?" 

Regards,
Stan

> Rick thought using the FFFFFFFF thing would 'merge' the ideas a little better.... My concern with that is that it could affect other decisions being made based on that, like multicast wildcard or something.  Ideally, imho,  we could have a special tlv to state : 
> [email-masked]

---

From: "Snyder, Brian" <[email-masked]>
Date: Thu, 24 Jul 2014 17:13:51 +0000

The reason why DLEP is still desired is because it will allow one to configure the L3 timers to be much much higher.  I can rely on DLEP to tell me radio link up/down (even though Ethernet side is always connected).  Hence I can tune the L3 stuff to be a lot calmer and save a ton hello/ack packets AFTER convergence, since if the modem leaves network I can get a DLEP down message to drive the router to withdraw the established L3 peer and converge L3 in an event driven manner.  In satellite we must protect the RF bandwidth because its very over subscribed and very expensive.  The main driver for me looking into these technologies is too protect the bandwidth -> hence I am relying on dlep to pass radio interface events to drive convergence.

Brian

-----Original Message-----
From: Stan Ratliff (sratliff) [mailto:[email-masked]] 
Sent: Thursday, July 24, 2014 12:59 PM
To: Snyder, Brian
Cc: Henning Rogge; Rick Taylor; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

> The other idea that was proposed was to define a new TLV at session init/discovery to tell a router - the same idea... This is a link up message -> now go converge.  

How is this different from running the spoke router without DLEP, as it is today?  The interface comes up, and convergence starts... 

Regards,
Stan

> All of the above being optional -- of course one could set MAC / IP 
> destinations if it's desired to any specific host. And these options should be combinable... for instance if you set all multicast this way, it would use default link metrics (or whatever is set) and if you want to specify a more specific rate control on a particular destination you could do that too.
> 
>> spoke topology would announce it's link-state using the MAC address 
>> of the hub as the destination identifier in the relevant signals.  
>> But, as Brian points out, there are examples of existing technologies 
>> where a-priori configuration of the hub MAC, or retrofitting the 
>> passing of the hub MAC in the link-layer signal is infeasible.
> 
>> way for the spoke modem to announce the link-state of the one and 
>> only link it has in a generic way.  Given the lack of knowledge of 
>> the hub MAC, any attached router will need to ARP/ND for it anyway, 
>> using the FF:FF:FF:FF:FF:FF address, so there is some consistency there.
>

---

From: "Stan Ratliff (sratliff)" <[email-masked]>
Date: Thu, 24 Jul 2014 17:16:34 +0000

Oh, one more observation inline : 

>> However, in SATRAN, no one really relies on the router to do any qos because it doesn't know anything that's going on  over the shared (and oversubscribed) radio link, so its much more common to l…

This is EXACTLY one of the things that DLEP is intended to address. Currently, the router DOESN'T know what is going on over the radio link. Hence, we created a protocol for the radio to TELL the router what's going on, so all of the QoS tools can be utilized. 

Regards,
Stan

---

From: "Stan Ratliff (sratliff)" <[email-masked]>
Date: Thu, 24 Jul 2014 17:34:13 +0000

> The reason why DLEP is still desired is because it will allow one to configure the L3 timers to be much much higher.  I can rely on DLEP to tell me radio link up/down (even though Ethernet side is…

But I think the same problem exists on "Destination Down" as with "Destination Up". Specifically, you said in an earlier email that identifying the hub router, by MAC address, is difficult to do for providing the "Destination Up". If that's the case, isn't it just as tough to identify the hub router when it goes down? I don't understand the use case for when the hub router and/or modem drops - how do you know to issue Destination Down to the spokes?

Another thing to note - DLEP presentation of "Destination Up" does NOT necessarily guarantee a routing adjacency to said destination. It means "something connected via the RF". If the router's implementation is to issue a HELLO, and that HELLO is not answered by anyone other than the hub router, then it all works as you want it to (I think).

Regards,
Stan

---

From: Henning Rogge <[email-masked]>
Date: Thu, 24 Jul 2014 19:39:13 +0200

> Another thing to note - DLEP presentation of "Destination Up" does NOT necessarily guarantee a routing adjacency to said destination. It means "something connected via the RF". If the router's impl…

I wonder if it would be feasible in this "hub and spoke" topology for
the hub to send a regular HELLO packet over the satellite link. This
would allow the spokes to detect the hubs presence and its MAC
address.

Unless the satcom terminal itself "knows" if its connected or not.

Henning Rogge

---

From: "Snyder, Brian" <[email-masked]>
Date: Thu, 24 Jul 2014 17:41:20 +0000

Yes the SATCOM terminal knows it's state in the network.  That's what I am using to determine if DLEP should send up/down... that's the exact link state I am reporting on.  Sorry for not explaining that before - I thought it was clear.  Funny how when the worlds collide what we all take for granted is different ;)

Thanks,brian

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]] 
Sent: Thursday, July 24, 2014 1:39 PM
To: Stan Ratliff (sratliff)
Cc: Snyder, Brian; Rick Taylor; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

> Another thing to note - DLEP presentation of "Destination Up" does NOT necessarily guarantee a routing adjacency to said destination. It means "something connected via the RF". If the router's impl…

I wonder if it would be feasible in this "hub and spoke" topology for the hub to send a regular HELLO packet over the satellite link. This would allow the spokes to detect the hubs presence and its MAC address.

Henning Rogge

---

From: Rick Taylor <[email-masked]>
Date: Thu, 24 Jul 2014 20:30:51 +0000

Brian, All,

As is often the case with me, I have changed my position on this one!

After a long and messy fajita lunch with John Dowdell, he has pressed me to actually drill down to what is the actual problem we believe Brian is trying to solve, and I believe I now have a clearer view.

To recap: DLEP provides link state notification, what links are up or down, what's the link like, and what's on the other end.  The way DLEP identifies a link is by the MAC address of the router interface over which the remote DLEP session is running.  This serves 2 purposes: 1) To uniquely identify a link. 2) To allow a routing protocol to fast-start building adjacencies without having to rely on hello timers, by advertising a layer-2 address to send packets to.

As I proposed in my extension draft, it is possible to split these 2 purposes, if and only if there is enough information supplied through DLEP to address IP packets to the remote router via a link.  This is the most important part of identifiers, if there is no obvious way for a local router to match a remote router to a link, then it's all broken.  In core DLEP (as already mentioned) the MAC address does this, in my draft, the remote routers IP address(es) MUST be supplied.

And so to Brian's use-case.  In his layer-2 bridging hub-spoke model, the hub isn't the issue, it's just stock DLEP.  But, in the spoke modems, Brian wants to avoid having to know the MAC address of the hub router, as there is only ever 1 link that is either up or down.  So, the proposal was to identify the link with the broadcast MAC (FF:FF:FF:FF:FF:FF) and forcing routers to recognize this MAC as meaning "Go probe for the routers MAC".

In retrospect, I have 3 problems with my aforementioned suggestion.
1) It adds a special case.
2) FF:FF:FF:FF:FF:FF is the broadcast MAC, and therefore should (like multicast MAC addresses) indicate information about broadcast, not treated as 'Unknown'.
3) By not supplying a MAC address, a lot of the fast-start advantages for routing protocols are lost, in this case, you still need link local multicast hellos, all you know is when to start and stop saying hello.

Also, Brian has latterly mentioned that in some configurations, spoke modems can actually pass layer-2 traffic to other spoke modems (via the hub at the RF layer).  These are therefore alternate links/destinations, and there must be some way of mapping the address of the attached router to the link over which traffic is flowing, so FF:FF:FF:FF:FF:FF is definitely no good, you must have MACs.  Using 00:00:00:00:00:00 instead only fixes half the problem, you are still breaking fast-start.

So, sorry Brian, I believe I was wrong, reusing the broadcast address as a 'wildcard' link/destination ID isn't going to work consistently in the common DLEP use-cases, and in yours it's probably a bad cludge.

I have had this problem myself before, with military radios that did not (yet) support DLEP, and only identified 'neighbours' by the radios RF-side MAC.  My solution was to add to my DLEP radio implementation for that kit (which proxied their management interface) a link-local multicast hello step.  This post-RF neighbour up, but pre-DLEP destination up, broadcasts some "Who is the router attached to radio MAC X?" and the router who is replies with a unicast "It's me!", and the receiver can then unpack the senders MAC, and then send on the DLEP destination up with the learned MAC.

This is undeniably a pain.  But, the pain is not DLEP's fault, it's just that if the information DLEP needs does not flow over the air within the radio's magic, it has to be done as a separate IP step within the radio DLEP process (a workaround), or the radio magic needs to be changed (the best way).  In Brian's case, and in the case of all the other radio manufacturers who are reading this, you need to decide which of these two ways you want to do it, but I now firmly believe that DLEP can't do what it can do by adding a 'wildcard' address.

I hope that helps?

Rick

________________________________
From: manet [[email-masked]] on behalf of Rick Taylor [[email-masked]]
Sent: 24 July 2014 16:46
To: Snyder, Brian; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

Rick

---

From: "Snyder, Brian" <[email-masked]>
Date: Thu, 24 Jul 2014 22:51:24 +0000

Comments embedded…..

From: Rick Taylor [mailto:[email-masked]] 
Sent: Thursday, July 24, 2014 4:31 PM
To: [email-masked]; Snyder, Brian
Subject: RE: Request to consider for DLEP

Brian, All, 

[BSS] – And I guess I am looking for something different.  I’m looking for link state notifications (with metrics) to  - and this is a different #2 – allow a routing protocol to be tuned for minimum KEEP ALIVE timers to minimize bandwidth usage to conserve very expensive and limited bandwidth.  In a satellite network, there is no need for fast-starts, L1 convergence can (and in a suitably sized network will take seconds).  However, because modems tend to stay in network quite stably (trees and buildings don’t get the way often in the ocean☺), the building of adjacencies does not happen all that often. It’s way more important to me to conserve bandwidth and be transparent.

And so to Brian's use-case.  In his layer-2 bridging hub-spoke model, the hub isn't the issue, it's just stock DLEP.  
[BSS] – Agreed.

But, in the spoke modems, Brian wants to avoid having to know the MAC address of the hub router, as there is only ever 1 link that is either up or down.  So, the proposal was to identify the link with the broadcast MAC (FF:FF:FF:FF:FF:FF) and forcing routers to recognize this MAC as meaning "Go probe for the routers MAC".  

In retrospect, I have 3 problems with my aforementioned suggestion.
1) It adds a special case.\
[BSS] – Satellite is always a special case ;)  Most folks don’t ‘defend’ bandwidth like we do.  Every decision we make is with the notion that it costs thousands and thousands of dollars per MB per month in bandwidth costs alone and as a result of that networks are usually very oversubscribed.

2) FF:FF:FF:FF:FF:FF is the broadcast MAC, and therefore should (like multicast MAC addresses) indicate information about broadcast, not treated as 'Unknown'.
[BSS] – That’s why it’s a special case ;).  I have no real desire to use FFFFFFFF – I don’t really care about that, I propose a special TLV for this purpose because if you recall I didn’t want to re-use something that already has a  purpose – that’s definitely more complex either with config or state machine knowledge, and therefore a harder sell.

3) By not supplying a MAC address, a lot of the fast-start advantages for routing protocols are lost, in this case, you still need link local multicast hellos, all you know is when to start and stop saying hello.
[BSS] – Actually, you can always send your multicast probes… if I’m out of network they just won’t go anywhere. What I want is for a DOWN to UP transition to reset any connected L3 peers and associated routes (as if you pulled the NIC cable). The rest should and could happen without any special knowledge – the router should act like normal.

Also, Brian has latterly mentioned that in some configurations, spoke modems can actually pass layer-2 traffic to other spoke modems (via the hub at the RF layer).  These are therefore alternate links/destinations, and there must be some way of mapping the address of the attached router to the link over which traffic is flowing, so FF:FF:FF:FF:FF:FF is definitely no good, you must have MACs.  Using 00:00:00:00:00:00 instead only fixes half the problem, you are still breaking fast-start.
[BSS] – Yes I am knowingly breaking fast-start.  It’s possible someone could configure us to ‘know’ the mac if they wanted this feature – but I doubt it would be very popular.  Most folks wouldn’t want the OAM burden to save 1 sec on something that will take say 5 anyway.  And there is another difference you’re missing.  That difference is because I’m not MESH I *DON’T* need to know any macs on the other side of the network. Currently modems do not know that information.  Regardless of VPWS or VPLS mode.  Remember the modems have a ‘NOT-LOCAL MAC then FWD OTA’ logic.  This works well for us, and in fact is mandatory.  Reason being that if one is configured to allow hairpinning, the modems would potentially have to learn MAC addresses in the thousands to hundreds of thousands. That simply can’t and won’t scale on modems. We consciously decided NOT to store the hub side mac information – because we don’t need it.  It’s not a mesh system anyway so if  a customer device wants to talk to another customer device off a modem lan – he still MUST go through the hub anyway.  The hub will have a full FIB of the network and hairpin it if it is necessary (and allowed) to do so.  If I have to maintain this L2 state for the router then I have too much complication in a modem that it is not designed nor capable of handling.  Even if it could, it stops me from being transparent which is a very important design goal for us.  More importantly though, putting this OAM burden on the modem will have scale ramifications that I’m not sure we could live with. (It could be investigated though).  See a SAT modem in a star network will only ever have 2 interfaces – one lan, one rf.  I understand a router may have many, but essentially on the interface we are internetworking on, I guess I am trying to get the router to think like the modem a little bit because it will greatly simplify the system.  I fully understand my use case might be the only one that thinks this way --- it’s clear DLEP wasn’t designed for this use case from the beginning – which is why I ended up using BFD to begin with.

[BSS] I don’t understand why it’s a cludge. All I am asking for is the router to behave exactly like it would if dlep wasn’t being used at all. But with DLEP it’s possible to calm down the “STILL-THERE” probes once routing adjacencies are discovered.  I’m not asking a router to do something it can’t intrinsically do.  I know it can behave this way because that’s essentially exactly what BFD does (just without metrics).  

[BSS] I understand the request is only applicable to modes in which we know there will not be multiple DLEP sessions – but given that’s a valid use case (all non-mesh networks would fit this I believe) I think it would work, if there was any interest besides mine ;)  

Again, I’m not coming at this because I’m afraid to make some changes to the modem. I could probably code this up in a prototype easy enough either way.  I’m sticking to my guns on this one because I have concerns about:

• it adds additional OAM and config management complexity to thousands of nodes.  I think most mesh people consider 40 modems a big network – but I have to think about things in terms of a hundred thousand.  Having to configure something then I don't really have to configure nor otherwise know is just one more thing that could go wrong... we don't like things to go wrong on a modem because they are usually very inconveniently located ;)
• however, auto figuring everything out and trying to share all possible MACS is a scale issue, I’m doubtful I can solve.  So it would therefore limit this feature to small networks. (Exactly how small – I’d have to do some math).
• Most importantly --- It would break our design goal of being a transparent bridge.  If we had to ‘know’ and configure a special MAC or set of MAC we are not transparent.  If we aren't transparent, then a case could be made to just go ahead and use us in L3 mode and none of this is needed. We can just withdraw routes when modem leaves network.

So, yes, I am asking for this use case to get special consideration…. I wouldn’t consider it the Satellite use case though, I would consider it the “Simplified operations in a non-mesh radio network” …. therefore it could apply to more people.

If there is no interest in further consideration of this use case, fair enough – I’m used to satellite being a lone wolf ;) -- I still have the bfd proxy I presented which mostly solves my problem, I can always just look to mature that within the bfd group.

It was great meeting you all this week and experiencing my first IETF.  Thanks to all for that!

Thanks,
Brian

I hope that helps?

Rick

________________________________________

Rick

---

From: Henning Rogge <[email-masked]>
Date: Fri, 25 Jul 2014 08:35:38 +0200

> [BSS] – And I guess I am looking for something different.  I’m looking for link state notifications (with metrics) to  - and this is a different #2 – allow a routing protocol to be tuned for minimu…

I must admit I still fail to see why a special case is even necessary.

We are talking about a network with hundred thousands clients
connected via satellite to a Hub station. Yes, so we need to make sure
that the clients don't talk more than necessary.

But you said your client satellite modems are able to receive
broadcast traffic... and maybe even unicasts delivered to a different
client. This means pushing information from the Hub to all clients is
reasonable cheap.

This means transferring the Mac address to a client is dirt cheap,
especially if Layer-2 at the Hub is able to tell that a new client
connected to the satellite.

What is the typical uptime of a client? And what percentage of traffic
goes from the Hub to the clients? A rough guess would be great to
better understand your problem.

Henning Rogge

---

From: Rick Taylor <[email-masked]>
Date: Fri, 25 Jul 2014 13:36:21 +0000

I'm trying to stay relatively quiet on this topic now, because I'm hoping some of the DLEP guru's will chip in with their advice (thanks Henning).

But, DLEP gives you link state/sensing, so you can turn down/off the IGP heartbeats as the DLEP modem will tell you the state of the link it has learnt from the underlying RF.  

Cheers,
Rick  
________________________________________
From: Henning Rogge [[email-masked]]
Sent: 25 July 2014 07:35
To: Snyder, Brian
Cc: Rick Taylor; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

Henning Rogge

---

From: "Snyder, Brian" <[email-masked]>
Date: Fri, 25 Jul 2014 13:37:09 +0000

Embedded....

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]] 
Sent: Friday, July 25, 2014 2:36 AM
To: Snyder, Brian
Cc: Rick Taylor; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

[BSS] - Perhaps not necessary - but I'm very concerned with trying to optimize for scale and transparency.

We are talking about a network with hundred thousands clients connected via satellite to a Hub station. Yes, so we need to make sure that the clients don't talk more than necessary.

But you said your client satellite modems are able to receive broadcast traffic... and maybe even unicasts delivered to a different client. This means pushing information from the Hub to all clients is reasonable cheap.
[BSS] - They won't receive unicast for someone else.  HDLC addressing demuxes that.  Pushing broadcast is cheap.  IE: ARPs can flow freely and cheaply on the downstream.

This means transferring the Mac address to a client is dirt cheap, especially if Layer-2 at the Hub is able to tell that a new client connected to the satellite.
[BSS] - In this case transferring a mac address is cheap - the arp's do it through us -- though the modems currently don't track and record any of the learnt address OTA... we just let the customer devices do what they should do by all normal conventions. (ARP, timers, etc).  Modems have no notion of weather or not another modem is in or out of network.  No star system would. They just track themselves. So if MAC addresses are supposed to be stored and managed by the modem for DLEP management automatically, we a) aren't transparent anymore b) Have a scale problem managing all those macs c) If there are more mac addresses (not routers we would care about via dlep) how is a transparent switch supposed to "KNOW" which mac address is the special one that should be 'setup' as the dlep target.  I think that would require some special config that then has to be managed (also breaking transparency) and creating OAM and change management issues. Remember the LAN is not 'our' equipment, so all the sudden one would have to get and manage mac addresses for all these sites.  We are going L2 in-order to help us and our clients not worry about each other's equipment and networking.

What is the typical uptime of a client? And what percentage of traffic goes from the Hub to the clients? A rough guess would be great to better understand your problem.
[BSS] - Typical uptime would be measured in from hours (Maybe a plane moving across satellite footprints) to hopefully years - a stationary modem.  Though practically that’s maybe more normally months due to weather, upgrades, etc.

Henning Rogge

---

From: Rick Taylor <[email-masked]>
Date: Fri, 25 Jul 2014 14:05:06 +0000

Top-posted...

Sorry Brian,

I think my understanding of your problem might still be wrong, but I'll make some suggestions anyway...

You don't need to preconfigure the spoke router MAC that gets sent to the hub because you can learn it from the endpoints of the spoke DLEP session.  Then when an RF link is established, you send it up to the hub, and it can report it through it's local DLEP session, and vice-versa.

You don't need to report all destinations in all sessions, only relevant destinations.  So in the spoke modem, you only need to report the hub MAC in the common case, unless you have turned on hair-pinning for some subset of spokes, in which case you only need to report those addressable MACs as DLEP destinations.

As I see it, your problem is along the lines of "OMG, I have 10,000's spokes and I can't have a master list of spoke router MACs, and I don't want to report 10,000's DLEP sessions everywhere when spokes can really only talk to the hub".  And my answer is, "You don't have to report everything, all the time, and you can learn the MACs at runtime."

Cheers,
Rick
 
________________________________________
From: Snyder, Brian [[email-masked]]
Sent: 25 July 2014 14:37
To: Henning Rogge
Cc: Rick Taylor; [email-masked]
Subject: RE: [manet] Request to consider for DLEP

Embedded....

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]]
Sent: Friday, July 25, 2014 2:36 AM
To: Snyder, Brian
Cc: Rick Taylor; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

Henning Rogge

---

From: "Snyder, Brian" <[email-masked]>
Date: Fri, 25 Jul 2014 14:18:06 +0000

>As I see it, your problem is along the lines of "OMG, I have 10,000's spokes and I can't have a master list of spoke router MACs, and I don't want to report 10,000's DLEP sessions everywhere when spokes can really only talk to the hub".  And my answer is, >"You don't have to report everything, all the time, and you can learn the MACs at runtime."

Yes, that is a concern of mine... but not the only one.  I understand I only have to report what I want too but I'm also concerned with the notion I can auto-learn it.  How would I know, in a VPLS network which mac is the upstream router? I could be inundated with MACs from all over the network. Maybe some I'd care about and some I wouldn't.  But I certainly couldn't track them individually. I have no idea of another modems state.  I only know MY state in relation to the hub - IE: am I in network or not.  Also recognize that if I have a hundred thousand MAC addresses and I had to track them all -- well a common design with our networks is for people to use multiple SVNs (vlans/qinqs) tags PER modem.  Many designs have 12 each. Our namespaces are kept separately.... so a hundred thousand modems with 12 vlans would be 1.2 MILLION macs that could potentially be seen.

Granted, I don’t think that scale is going to be roll out #1 of DLEP use case - but we do have customers operating at that scale.  If 'DLEP' feature gets released - they will want to use it.

No modem currently tracks any MACs from the hub side for this reason -- and we don't need too (yet:)).

brian

-----Original Message-----
From: Rick Taylor [mailto:[email-masked]] 
Sent: Friday, July 25, 2014 10:05 AM
To: Snyder, Brian; Henning Rogge
Cc: [email-masked]
Subject: RE: [manet] Request to consider for DLEP

Top-posted...

Sorry Brian,

Cheers,
Rick

Embedded....

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]]
Sent: Friday, July 25, 2014 2:36 AM
To: Snyder, Brian
Cc: Rick Taylor; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

Henning Rogge

---

From: Rick Taylor <[email-masked]>
Date: Fri, 25 Jul 2014 14:28:42 +0000

Inline comments....

>Yes, that is a concern of mine... but not the only one.  I understand I only have to report what I want too but I'm also concerned with the notion I can auto-learn it.  How would I know, in a VPLS n…

Ah-ha!! The DLEP destination MAC is defined very specifically as the MAC address of the interface that the router is using for the TCP session to the local modem.  The router may have a billion other MAC addresses, but only that one is important.  In the spoke case, the same applies, and if the spokes don't need to know about other spokes, don't tell them.

> Granted, I don’t think that scale is going to be roll out #1 of DLEP use case - but we do have customers operating at that scale.  If 'DLEP' feature gets released - they will want to use it.

Absolutely, which is why I and others are spending some cycles to get this sorted.

> No modem currently tracks any MACs from the hub side for this reason -- and we don't need too (yet:)).

Hopefully my answer above will have helped?

Rick

-----Original Message-----
From: Rick Taylor [mailto:[email-masked]]
Sent: Friday, July 25, 2014 10:05 AM
To: Snyder, Brian; Henning Rogge
Cc: [email-masked]
Subject: RE: [manet] Request to consider for DLEP

Top-posted...

Sorry Brian,

Cheers,
Rick

Embedded....

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]]
Sent: Friday, July 25, 2014 2:36 AM
To: Snyder, Brian
Cc: Rick Taylor; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

Henning Rogge

---

From: "Snyder, Brian" <[email-masked]>
Date: Fri, 25 Jul 2014 16:24:20 +0000

Rick,

>Ah-ha!! The DLEP destination MAC is defined very specifically as the MAC address of the interface that the router is using for the TCP session to the local modem.  The router may have a billion other MAC addresses, but only that one is important.  In the >spoke case, the same applies, and if the spokes don't need to know about other spokes, don't tell them.

Hold on.  I thought it was the MAC address of the far side router (otherwise it would still need to arp).  Your comment --- the LOCAL modem would greatly change the issue in my mind --- but how does knowing a local MAC help L3 converge any faster - that would mean it still needs to arp or mcast hello to find far side guys so I don’t understand then how the 'fast-start' stuff works you were discussing as the point of this.....

In this diagram below:

                                                                                  |-----  MDM1 ---- RTR1 --- NET1
                                                                                  |
UpRtr -----   SatHUB   ------  OUTERSPACE ----------   MDM2 ---- RTR2 --- NET2
                                                                                  |
                                                                                  |-----  MDM3 ---- RTR3 --- NET3

Let's say RTR1, RTR2, and RTR3 all want to make a [OSPF|EIGRP|BGP] neighbor to UpRTR.  UPRTR wants 3 neighbors.  There are 2 vlans on each.

What does MDMX have to tell RTRX via DLEP?   How many MAC Addresses (Multiple VLANS) and which ones?

Now what if while the RF system is not MESH, I am allowing hairpinning (VPLS mode).... So NET1 can talk to NET2 via SatHUB.  Then what do they all have learn?

Maybe a concrete example will help this sink in.

Thanks,
Brian

-----Original Message-----
From: Rick Taylor [mailto:[email-masked]] 
Sent: Friday, July 25, 2014 10:29 AM
To: Snyder, Brian; Henning Rogge
Cc: [email-masked]
Subject: RE: [manet] Request to consider for DLEP

Inline comments....

Rick

-----Original Message-----
From: Rick Taylor [mailto:[email-masked]]
Sent: Friday, July 25, 2014 10:05 AM
To: Snyder, Brian; Henning Rogge
Cc: [email-masked]
Subject: RE: [manet] Request to consider for DLEP

Top-posted...

Sorry Brian,

Cheers,
Rick

Embedded....

-----Original Message-----
From: Henning Rogge [mailto:[email-masked]]
Sent: Friday, July 25, 2014 2:36 AM
To: Snyder, Brian
Cc: Rick Taylor; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

Henning Rogge

---

From: "Stan Ratliff (sratliff)" <[email-masked]>
Date: Fri, 25 Jul 2014 16:30:04 +0000

> Let's say RTR1, RTR2, and RTR3 all want to make a [OSPF|EIGRP|BGP] neighbor to UpRTR.  UPRTR wants 3 neighbors.  There are 2 vlans on each.

*By definition*, UpRtr in this case has 6 neighbors, not 3. Each VLAN is a neighbor.

> What does MDMX have to tell RTRX via DLEP?   How many MAC Addresses (Multiple VLANS) and which ones?

That's the rub. MDM1, for example, can tell RTR1 *only* about UpRtr. (Specifically, the VLANS on UpRtr). So, while UpRtr has 6 neighbors (but we've already said the hub isn't the problem), RTR1 only sees 2. 

Stan

---

From: "Snyder, Brian" <[email-masked]>
Date: Fri, 25 Jul 2014 16:37:06 +0000

embedded....

-----Original Message-----
From: Stan Ratliff (sratliff) [mailto:[email-masked]] 
Sent: Friday, July 25, 2014 12:30 PM
To: Snyder, Brian
Cc: Rick Taylor; Henning Rogge; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

[BSS] - That’s what I expected.

[BSS] - Right. that’s what I was expecting the answer to be.  But then it isn’t the local MAC address RTR1 needs, its UPRTR MAC Address - he is telling him about the REMOTE mac, not local... which means the MDMs would need to be configured to know what that is (per VLAN - even though itll most likely always be the same).

Now if we want NET1 to talk to NET2 - then I'm assuming the RTR2 mac has to be told to RTR1 as well via DLEP too. Correct?  So then, instead of letting the RTRN find RTRN' naturally the way a bunch of routers on an IGP would,  MDMN would have to tell RTRN about the entire set of RTRs (that aren't himself).  

That's what I'm hearing from this conversation... you have to tell them EVERY specific destination.  Now all the sudden, in this VPLS architecture and using DLEP, the MDMs have to maintain a MAC to MDM map that could be HUGE.  I don't think this is the part that scales.

If I'm in VPWS mode (NET1 can't see NET2) then I agree - it's not that hard of a problem to solve - though there still needs to be additional OAM knowledge on the MDM which breaks my transparency design - but I could maybe live with that if it was a bounded set of say 12 MACs (# of VLANs expected to getting routing peers on per modem - worst case).

Are my extrapolations here correct?

Thanks,brian

Stan

---

From: Rick Taylor <[email-masked]>
Date: Fri, 25 Jul 2014 16:59:30 +0000

Brian,

I'm getting baffled as you ASCII diagram lost all formatting.  Can you redraw it as a png or something and send it back out?

Cheers,

Rick
Sent: 25 July 2014 17:37

embedded....

-----Original Message-----
From: Stan Ratliff (sratliff) [mailto:[email-masked]]
Sent: Friday, July 25, 2014 12:30 PM
To: Snyder, Brian
Cc: Rick Taylor; Henning Rogge; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

Thanks,brian

Stan

---

From: Shawn Jury <[email-masked]>
Date: Fri, 25 Jul 2014 13:49:21 -0400

Brian,

    I'm curious about how your BFD proxy would work in this case.  Since you are not letting the BFD packets go between the two sites (MDM1 and MDM2 for example), how does MDM1 know when to tell RTR1 that MDM2 is gone and BFD should tear down the IGP session between the devices, especially since you would still need a BFD session for each VLAN?  In my mind, I'm seeing this mechanic as the same one that is needed for DLEP to work.

Shawn

---

From: "Snyder, Brian" <[email-masked]>
Date: Mon, 28 Jul 2014 15:23:46 +0000

Rick,

Attached is picture in a binary format.

Thanks,brian

-----Original Message-----
From: Rick Taylor [mailto:[email-masked]] 
Sent: Friday, July 25, 2014 12:59 PM
To: Snyder, Brian; Stan Ratliff (sratliff)
Cc: Henning Rogge; [email-masked]
Subject: RE: [manet] Request to consider for DLEP

Brian,

Cheers,

Rick

embedded....

-----Original Message-----
From: Stan Ratliff (sratliff) [mailto:[email-masked]]
Sent: Friday, July 25, 2014 12:30 PM
To: Snyder, Brian
Cc: Rick Taylor; Henning Rogge; [email-masked]
Subject: Re: [manet] Request to consider for DLEP

Thanks,brian

Stan

---

From: Teco Boot <[email-masked]>
Date: Tue, 29 Jul 2014 09:44:23 +0200

Brian,

What satcom topology do you have in mind? Single hop between  MDM1, MDM2 and MDM3? Relay at layer-2 on SatHUB? Relay at layer-3 on UpRTR?

For all these scenario's, DLEP is applicable. DMAC addresses are one IP hop destination MAC addresses. It is up to the modems to provide link metrics for path between the IP neighbors. This could be hard for multi-hop RF paths.

Teco