Subject: RE: multi-homing for provider-assigned IPv6 addresses

From: Chris Bowers <[email-masked]>
Date: Mon, 4 Apr 2016 14:51:24 +0000

Fred,

I don't follow the argument made in the previous email against the proposal to use a simplified form of dest/src routing.  The argument seems to assume that the destination prefix is completely ignored, which is not the case.  With this proposal, a packet is first routed based on longest match for destination prefix.  The source prefix is only considered when the destination route lookup reaches ::/0.  Therefore, communication within the customer network works as expected since there will be destination prefix matches corresponding to the customer network.

In order to understand all of the details, I created an example topology below that I believe covers most of the relevant PA multi-homing scenarios.  The customer site connects using four different ISPs.  ISP1 just offers Internet connectivity.  ISP2 just offers a service corresponding to destination address B2.  ISP3 offers both Internet connectivity and a service corresponding to B3.  ISP4 offers internet connectivity and a service corresponding to B4.  The customer site has a provider-assigned subnet from each of the ISPs (A1, A2, A3, and A4).  The customer site has two LANs.  LAN-Y (with hosts H21 and H22) is directly connected to R1, R2, and R3.  LAN-X (with hosts H31 and H32) has to go through a routed hop (R7 or R8) to reach R1, R2, or R3.  The only way for any of the hosts to reach R4 (and ISP4) is through R8.  The hosts on LAN-X have addresses from a subnet of each of the provider assigned block ( subnets A1x, A2x, A3x, and A4x), while the hosts on LAN-Y have addresses from a subnet of each of the provider assigned block ( subnets A1y, A2y, A3y, and A4y).  B0 represents a destination address somewhere on the Internet unrelated to any of the ISPs.

                                             -----------------
   LAN-X        LAN-Y          ,-----.      /                 \
          +--+         +--+  ,'       `.   :                   :
       +--+R7+-----+---+R1+-+   ISP 1   ---+--                 :
       |  +--+     |   +--+  `.       ,'   :                   :
   H31-+       H21-+           `-----'     :   The Internet    :
       |           |           ,-----.     :                   :
   H32-+  +--+ H22-+   +--+  ,'       `.   :                   :
       +--+R8+-----+---+R2+-+   ISP 2   )  :                   :
          ++-+     |   +- +  `.       ,'   :                   :--B0
           |       |           `--+--'     :                   :
           |       |              |        :                   :
           |       |              B2       :                   :
           |       |                       :                   :
           |       |           ,-----.     :                   :
           |       |   +--+  ,'       `.   :                   :
           |       +---+R3+-+   ISP 3   ---+---                :
           |           +--+  `.       ,'   :                   :
           |                   `--+--'     :                   :
           |                      |        :                   :
           |                      B3       :                   :
           |                               :                   :
           |                   ,-----.     :                   :
           |           +--+  ,'       `.   :                   :
           +-----------+R4|-+   ISP 4   ---+---                :
                       +--+  `.       ,'   :                   :
                               `--+--'     :                   :
                                  |         \                 /
                                  B4         -----------------

We consider how H31 and H21 reach various destinations.  The destinations to consider are B0, B2, B3, B4, H32 and H22.

First, we look at the information that the routers distribute among themselves using an IGP.

R1 originates a route for (D=::/0, S=A1).

R2 originates a route for (D=B2).

R3 originates routes for (D=::/0, S=A3) and (D=B3).

R4 originates routes for (D=::/0, S=A4) and (D=B4).

R7 and R8 originates routes for (D=A1x), (D=A2x), (D=A3x), and (D=A4x), and (D=A1y), (D=A2y), (D=A3y), and (D=A4y).

Next, we look at the information that the routers distribute to hosts on the LAN using the Neighbor Discovery protocol.  From Prefix information Options (PIOs) in Router Advertisements, hosts on LAN-X learn that A1x, A2x, A3x, and A4x are on-link, while hosts on LAN-Y learn that A1y, A2y, A3y, and A4 are on-link.

A router indicates that it acts as a default router by advertising a non-zero Router Lifetime in a Router Advertisement. Or that it doesn't act as a default router by advertising a Router Lifetime value of zero.  A reasonable choice in this network would be to have R1 and R3 advertise that they are default routers on LAN-Y and have R7 and R8 advertise that they are default routers on LAN-X.

It appears to me that with just the information described above (that is, destination routes and simplified dest/source routes distributed among the routers, and the Router Advertisements on the LANs indicated default routers),  packets from H31 and H21 can reach all six destinations, assuming the host uses an acceptable source address for the given destination.  However, the next-hop chosen by the host may not be optimal.

For example, for a packet to get from H21 to H32, it may take the path H21->R1->R7->H32, instead of the more direct path H21->R7->H32.  However, this situation can be improved using existing mechanisms by having R7 advertise Route information Options (RIOs) for prefixes A1x, A2x, A3x, and A4x into LAN-Y so that H21 can choose R7 as its next-hop to directly reach H32.

draft-ietf-6man-multi-homed-host-06 points out an example of a host making a sub-optimal choice of next-hop related to multi-homing.  In the topology above, when H21 sends a packet with src in A1y and dest = B0, it could choose next-hop R3.  This would require R3 to forward the packet to R1 based on the fact that R1 originated an IGP route for (D=::/0, S=A1).  draft-ietf-6man-multi-homed-host-06 proposes to have hosts use the PIOs in Router Advertisements to make better choices of next-hops.  Assuming that only R1 advertises a PIO for A1y, then H21 can use that information to send a packet with src in A1y to R1 as its default next-hop.  This seems like a reasonable optimization.

I would like to make two observations about the analysis above.  First, the solution proposed in draft-ietf-6man-multi-homed-host-06 is consistent with the proposed simplified form of dest/source routing.  draft-ietf-6man-multi-homed-host-06 only allows one to choose a default router based on source prefix.  It doesn't provide a way to pair an arbitrary RIO with a source prefix.

Second, in the solution described above, which would seem to cover most multi-homing scenarios, the routes exchanged between routers using the IGP do not require source addresses to be paired with arbitrary destination prefixes.  The simplified form of dest/source routing (only pairing a source prefix with the default destination route) is sufficient for the routing done by the routers.

At this point, I have not seen a PA multi-homing scenario that I believe would benefit from using arbitrary dest/source routing.  Instead, I think that PA multi-homing scenarios can be efficiently addressed by simplified dest/source routing.  To be clear, this is not an argument against the mechanism in draft-ietf-6man-multi-homed-host-06. Instead, it is an argument against the mechanism in draft-baker-ipv6-isis-dst-src-routing-04, which specifies a general mechanism for pairing arbitrary destination prefixes with arbitrary source prefixes. I think that a simplified mechanism that only pairs source prefixes with the default destination route in the IGP would work just as well for PA multi-homing scenarios.

If there is an error in the above analysis or there is a useful PA multi-homing scenario not covered by the above example, then I look forward to discussing it.

Thanks,
Chris

From: Fred Baker (fred) [mailto:[email-masked]]
Sent: Monday, March 28, 2016 2:47 PM
To: Chris Bowers <[email-masked]>
Cc: [email-masked]
Subject: Re: multi-homing for provider-assigned IPv6 addresses

On Mar 27, 2016, at 9:03 AM, Chris Bowers <[email-masked]<mailto:[email-masked]>> wrote:

Fred,

I would like to better understand the particular use case offered as a counter-example.  One version of this counter-example is described in Section 2.3 of draft-baker-rtgwg-src-dst-routing-use-cases-01 (Specialized Egress Routing).  I have copied figure 3 from that draft below.

                                                ,-------.
                                             ,-'         `-.
             ,---.            ,-----.      ,'               `.
            /     \    +-+  ,'       `.   /                   \
           /       +---+R+-+   ISP 1   --+---                  \
          /         \  +-+  `.       ,' ;                       :
         ; Customer  :        `-----'   |       The Internet    |
         | Network   |        ,-----.   :                       ;
         :           ; +-+  ,'       `.  \                     /
          \         +--+R+-+   ISP 2   )  \                   /
           \       /   +-+  `.       ,'    `.               ,'
            \     /           `--+--'        '-.         ,-'
             `---'               |              `-------'
                           Some specialized
                              Service

       Figure 3: Egress Routing with a specialized upstream network

If a customer network homed to ISP1 and ISP2 (with PA prefixes S1 and S2) needs to reach a destination prefix (D2) associated with a service offered by ISP2, then the customer network can route all traffic with destination address in D2 to ISP2 using existing destination-based routing.

Yes, if the packet first goes through a router that is not depicted in the diagram (I didn't describe the interior of the customer network, which could be as simple as a LAN and as complex as you care to imagine). If the only routing is done by the host itself, the packet will go to the router the host considers to be its default router, which is pretty likely to be the other router in this example. That is the subject of draft-ietf-6man-multi-homed-host. Note, BTW, that the picture is essentially NTT's BFLETS configuration, with the exception that (as I understand it) the two ISPs are served by the same managed router, deployed by NTT and configured to route appropriately.

Therefore, it seems to me that pairing a source prefix with an arbitrary destination prefix is not required to support the Specialized Egress Routing use case in draft-baker-rtgwg-src-dst-routing-use-cases-01. In any case, it would be useful to come to agreement about what is required to address the Specialized Egress Routing use case described in draft-baker-rtgwg-src-dst-routing-use-cases-01.

I'm happy to discuss that. I think it misses the point, however. If the use case described isn't a good one, let's think of a better use case, perhaps one that doesn't involve a PA prefix from an ISP. Perhaps the biggest issue with your proposal, which was to take all traffic using a given source prefix to an appropriate egress router, is that it precludes the use of the address to communicate within the customer network. All traffic would dogleg through the egress router, and I'm not sure I see how packets get from that router to a point inside without looping. There has to be a way to prefer to destination-address route some traffic and source-address route some other traffic. The simplest way I can think of is to use both addresses in every route lookup, interpreting a traditional destination route as a source/destination route from ::/0 to the stated destination, or (if there are multiple route tables by source prefix) duplicate the destination route table in each.

If instead the counter-example use case described below is different from Specialized Egress Routing use case in draft-baker-rtgwg-src-dst-routing-use-cases-01, it would be useful to describe that use case in more detail so that we can evaluate it more thoroughly.

I'll take a look at that. Of course, I can't update the draft until next week.

Thanks,
Chris

-----Original Message-----
From: Fred Baker (fred) [mailto:[email-masked]]
Sent: Thursday, March 24, 2016 11:37 PM
To: Chris Bowers <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: multi-homing for provider-assigned IPv6 addresses

> On Mar 24, 2016, at 9:32 PM, Fred Baker (fred) <[email-masked]<mailto:[email-masked]>> wrote:
>
>
>> On Mar 24, 2016, at 4:34 PM, Chris Bowers <[email-masked]<mailto:[email-masked]>> wrote:
>>
>> It seems to me that most use cases for ipv6 multi-homing with provider-assigned addresses only need to route based on source address when the destination prefix is the default route.  So why not require that source prefixes can only be paired up with the default destination prefix ::/0?
>
> When what one has in mind is an egress route, that probably makes sense. However, it precludes an entire class of use cases mentioned in the use case draft. Why do that?

Let me give you an obvious variant. Imagine, if you will, that I have a PA prefix from each of N ISPs, and therefore a default route to each of N ISPs. Imagine also that I have a particular prefix that I would like to route through via a given one of those N ISPs, in the special case that I happen to have been smart enough to use that source prefix. So now I have N+1 source/destination routes - unless you tie it to default routes.

A premature optimization usually breaks things...

---

From: David Lamparter <[email-masked]>
Date: Thu, 7 Apr 2016 16:55:06 +0200

TL;DR:  note at end about "abort as unreachable" as alternative way to
get rid of complicated backtracking.  (Jump to second/last ----- line.)

Hi all,

here is another shot at my somewhat confusing (sorry) in-room comment on
the policy aspect of limiting to D=::/0.

The scenario I'm thinking of is this:

User (let's say, small-to-medium-ish business, for illustration let's
say the internal network has 2-3 OSPF routers or something.)
\- ISP A, assigned PA prefix, internet/default connectivity
\- ISP B, assigned PA prefix, internet/default connectivity

So far so good, we have two D=::/0 routes with different S.

Now, let's say the CEO of the company heard that clouds are great for
putting your calendar and mails into.  She decides ISP A's "Cirrus
Castellanus" product is great because it comes with some great security
and confidentiality bulletpoints.

For the Cirrus Castellanus service, ISP A installs a separate device in
the user's network, which routes a specific prefix for the cloud.  Could
be some encrypted VPN foobar, physical separate line, doesn't really
matter.  Since it's ISP A, they're aware of their PA assignment to the
customer and push a route for that into the cloud product.  The cloud
product could implement BCP 38, but it could also just not have a route
towards the internet - it doesn't need to and it's "secure".

Now, the installation instructions for the product says "this is the
prefix your cloud service has, put a route for it in your domain".

So, now the User's netadmin is essentially screwed since that's a
D!=default S!=default route.  More correct, it's in fact 2 routes:
- D=cloud, S=ISP-A-PA => via installed cloud termination device
- D=cloud, S=::/0 => unreachable
Which goes to represent that the cloud service is only reachable from
ISP-A's PA prefix assigned to the User.

----------
This, I hope, illustrates how this is a significant policy decision to
make.  It's IETF consensus (I hope) that NAT is a bad thing.  It also
seems to be IETF consensus (as discussed in Fred's slides) that PA is
preferable over PI, and we want to keep tables small.  We've received
feedback -the v6ops request- that PA has some issues to be solved.  If
we now say "we're doing D=::/0 only", we're deciding to solve only part
of the problem posed to us.  (see note below about divide&conquer / "do
D!=::/0 separately")

What's the message to operators here?
"get PI space for that setup"?
"you'll need a cooperative ISP for that setup"?
"don't do this setup, it's stupid"?

In any case, any solution that resticts to D=::/0 carries an implication
of:
   "If you're using PA, the only thing you can rely on being able to
    route is the internet as a whole.
If we're making this policy assumption (which is really a decision), we
need to be aware we're making it.

On Divide&Conquer:
splitting this up into a two-part approach resulting in the 3 levels:
- classic router, (D) lookup
- level-1 SADR, (::/0, S) support
- level-2 SADR, (D, S) support
is certainly possible to write down, but complicates the compatibility
mechanisms in the routing protocols since level-1 and level-2 routers
are not compatible to be in the same link-state/SPF topology.

----------
If we still want the backtracking behaviour (going to less specific D
when not finding a S match under a particular D) gone, I'd prefer
looking at specifying "terminate as unreachable" instead of completely
removing D!=::/0 support.  I haven't thought through use-case impact of
this yet, but in any case the backtracking behaviour could be restored
by advertising additional routes to the same effect.

(The same approach of installing additional routes can be used to
eliminate the performance hit of backtracking by consuming more memory /
hardware route resources through installing additional synthetic routes.
I'll add that to the next version of the draft.)

On a closing note, I'll repeat my mic request for feedback:  could
people provide feedback which part of backtracking they're concerned
about?  Specification complexity?  Implementation complexity?
Performance penalities?  Something else?

-David

---

From: [email-masked]
Date: Sat, 9 Apr 2016 16:01:53 +0200

Chris,

> It seems to me that most use cases for ipv6 multi-homing with provider-assigned addresses only need to route based on source address when the destination prefix is the default route.  So why not require that source prefixes can only be paired up with the default destination prefix ::/0?  That is, we could require that a router can only route based on the source address after the longest match route lookup based on destination address has reached ::/0.  That way, a router can still route packets local to the site based on destination address.  Only once it is clear that the packet is going to the Internet can the router use the source address to pick the correct CE router to get there.
> 
> I see that there was some discussion about this in 2013 on a thread in homenet and rtgwg in the context of draft-baker-rtgwg-src-dst-routing-use-cases, and a number of different opinions were expressed mainly from the point of view of what is needed for homenet.  At this point, RTGWG also needs to consider the impact of proposed solutions on routing scalability.  On the surface, this simplification would seem to improve the scaling properties of the solution in draft-ietf-rtgwg-dst-src-routing, while covering most use cases.

That will not work I'm afraid.
ISPs do not only advertise default routes to CPEs (see RFC6204) but also more specifics (RFC4191).
The more specifics are typically used for walled garden etc.
These will result in (S,D) routes in the network.

> Regardless of how this discussion turns out, I think the draft itself will benefit from more active discussion of the tradeoffs involved.

agreed.

we did a src/dst implementation in VPP (open source software data plane).
the results should be up in the proceedings soon, but until then the slides are here:
https://docs.google.com/presentation/d/1pXJO5vphWFmHAQsrKAgzXYxd-EvMFuxiNYDPXmiMCMk/edit#slide=id.p

in short, back-tracking is quite painful, and if one implemented this as trees of trees, worst case is very bad (128*128). we implemented is a bi-hash with 256 bit key.

a better approach might be to do S lookup first, find the D table, and do the D lookup. that requires duplication of all D,* routes into every D table, but should result in the same behaviour as the D first lookup. worst case then would be 128+128.

there are many alternatives that has been proposed over the years.
e.g.: draft-despres-sam, LISP, ILNP, RFC2260

that is build an overlay from every host/first-hop router to the set of edges.

the problem in the homenet, and I expect in small businesses as well, is that the edge routers are not aware of each other and provided by different entities (the ISPs).

when _you_ say scalability, what do you mean? Multi-prefix multi-homing is designed to alleviate the problem of global route scaling. Within the networks that will do D,S routing, I expect routing tables to be very small. E.g. if you were a company of cisco's size, with tens of providers, you'd do PI multi-homing.

Cheers,
Ole

> From: Chris Bowers
> Sent: Tuesday, March 15, 2016 8:14 AM
> To: [email-masked]' <[email-masked]>
> Subject: multi-homing for provider-assigned IPv6 addresses
> 
> RTGWG,
> 
> We scheduled a significant amount of time in Buenos Aires for a discussion of multi-homing for provider-assigned IPv6 addresses for enterprise networks as well as homenet.  I want to explain the motivation for this and provide some background on the topic.  And hopefully spark some discussion on the list before Buenos Aires, as well.
> 
> RTGWG adopted draft-ietf-rtgwg-dst-src-routing last October in the context of supporting multi-homing for provider-assigned IPv6 addresses in homenet.  In Yokohama, the v6ops WG had a lengthy discussion about the need for a solution to support multi-homing for provider-assigned IPv6 addresses for enterprise networks in general (as opposed to just homenet).  This discussion took place in the context of draft-ietf-v6ops-design-choices.  That discussion can be found at:
> 
> https://youtu.be/VzH7yqqGiGc?t=5835
> 
> This discussion led to the email (copied below) from Fred Baker in his role as v6ops co-chair to the chairs of several working groups in which drafts related to this topic are being discussed.
> 
> Our meeting in Buenos Aires includes a 20 minute time slot to discuss the background and motivation of this request from v6ops.  The topic of multi-homing for provider-assigned IPv6 addresses has a long history with many documents written, so I thought it would be useful to highlight a few of the more recent documents that I found to be particularly useful reading.
> 
> RFC 7157 "IPv6 Multihoming without Network Address Translation"
> https://datatracker.ietf.org/doc/rfc7157/
> 
> draft-ietf-6man-multi-homed-host-06 "Routing packets from hosts in a multi-prefix network"
> https://datatracker.ietf.org/doc/draft-ietf-6man-multi-homed-host/
> 
> In addition, it would obviously be useful read or re-read the RTGWG document on this topic:
> https://datatracker.ietf.org/doc/draft-ietf-rtgwg-dst-src-routing/
> 
> There following expired document is also helpful.
> https://www.ietf.org/archive/id/draft-baker-rtgwg-src-dst-routing-use-cases-01.txt
> 
> And finally, this document is useful to understand a concrete proposal for how src/dst routing information could be carried in a link-state routing protocol.
> https://datatracker.ietf.org/doc/draft-baker-ipv6-isis-dst-src-routing/
> 
> I look forward to a fruitful discussion on this topic on the list and in Buenos Aires.
> 
> Thanks,
> Chris
> 
> Sent: Sunday, November 01, 2015 11:03 PM
> To: [email-masked]; [email-masked]; [email-masked]; [email-masked]; 6man Chairs <[email-masked]>; [email-masked]
> Cc: [email-masked] WG <[email-masked]>; The IESG <[email-masked]>
> Subject: PA Address Multihoming in IPv6
> 
> This email is being sent in accordance with the v6ops charter, which calls for the working group to communicate operational issues and requirements to working groups that are chartered to address them.
> 
> The IETF's current primary recommendation for multihoming of midrange enterprise networks - those that cannot justify the costs and overheads of a PI address and in fact multihome - is to obtain a provider-allocated prefix from each of their upstream networks, and deploy a /64 out of each on each LAN in their networks.
> 
> https://tools.ietf.org/html/rfc4213
> 4213 Basic Transition Mechanisms for IPv6 Hosts and Routers. E. Nordmark,
>     R. Gilligan. October 2005. (Format: TXT=58575 bytes) (Obsoletes
>     RFC2893) (Status: PROPOSED STANDARD) (DOI: 10.17487/RFC4213)
> 
> This has a number of issues, not the least of which is in the back end OSS software, which needs to now scale to a much larger number of prefixes, handle multiple addresses in DNS for servers and perhaps clients, resolve reverse DNS queries, and so on. It also is obviously carrying that much more information in routing.
> 
> One outcome of v6ops' discussions this morning was that PI multihoming demonstrably works, but PA multihoming when the upstreams implement BCP 38 filtering requires the deployment of some form of egress routing - source/destination routing in which the traffic using a stated PA source prefix and directed to a remote destination is routed to the provider that allocated the prefix. The IETF currently has no such recommendation, or consensus that it should have. However, enterprise networks are known to delaying operational deployment of IPv6 in part due to the complexities visited upon them and the cost of the back end software upgrades, and this is part of that issue.
> 
> Without trying to limit the options available to the working groups in question, I'll point out that options currently on the table include the following. There are also current open source implementations of source/destination and source-specific routing in IS-IS, OSPFv3, and Babel.
> 
> https://datatracker.ietf.org/doc/draft-baker-ipv6-isis-dst-src-routing
>  "IPv6 Source/Destination Routing using IS-IS", Fred Baker, David
>  Lamparter, 2015-10-19
> 
> https://datatracker.ietf.org/doc/draft-boutier-babel-source-specific
>  "Source-Specific Routing in Babel", Matthieu Boutier, Juliusz
>  Chroboczek, 2015-05-27
> 
> https://datatracker.ietf.org/doc/draft-ietf-6man-multi-homed-host
>  "Routing packets from hosts in a multi-prefix network", Fred Baker,
>  Brian Carpenter, 2015-10-15
> 
> https://datatracker.ietf.org/doc/draft-ietf-ospf-ospfv3-lsa-extend
>  "OSPFv3 LSA Extendibility", Acee Lindem, Sina Mirtorabi, Abhay Roy, Fred
>  Baker, 2015-10-08
> 
> https://datatracker.ietf.org/doc/draft-ietf-rtgwg-dst-src-routing
>  "Destination/Source Routing", David Lamparter, 2015-10-17,
> 
> https://datatracker.ietf.org/doc/draft-sarikaya-6man-sadr-overview
>  "Source Address Dependent Routing and Source Address Selection for IPv6
>  Hosts: Problem Space Overview", Behcet Sarikaya, Mohamed Boucadair,
>  2015-08-17
> 
> https://datatracker.ietf.org/doc/draft-sarikaya-6man-sadr-ra
>  "IPv6 RA Option for Source Address Dependent Routing", Behcet Sarikaya,
>  2015-06-08
> 
> https://datatracker.ietf.org/doc/draft-sarikaya-dhc-6man-dhcpv6-sadr
>  "DHCPv6 Solution for Source Address Dependent Routing", Behcet Sarikaya,
>  2015-05-08
> 
> https://datatracker.ietf.org/doc/draft-xu-ospf-multi-homing-ipv6
>  "Extending OSPFv3 to Support Multi-homing", Mingwei Xu, Shu Yang,
>  Jianping Wu, Fred Baker, 2015-10-11,
> 
> https://datatracker.ietf.org/doc/draft-baker-rtgwg-src-dst-routing-use-cases
> 
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg

---

From: Chris Bowers <[email-masked]>
Date: Tue, 12 Apr 2016 00:14:25 +0000

David,

I've responded to two of the points raised / questions asked below.  Feel free to break these responses into separate threads or to resend any of the points/questions as separate threads as well.  

===================

Thanks for the example of the ISP-provided cloud device.  I think that I am starting to better understand one of the sources of disagreement on this topic.  If one assumes that a host can magically choose the correct source address for a packet, then I think even this example can be handled with default-only SADR.  However,  if we also want to use SADR to distribute information to the hosts to allow them to choose the correct source address to put on the packet in the first place, then I agree that default-only SADR will not work.  If this is the case, then I think we need to be more explicit about how we expect the host to get information about the correct source address to select.

This is my best understanding at this point about how this could work.  Please correct me if I am wrong.

Using the example below, for a host (H55) which is a few router hops away from the ISP A's cloud device, we want to put the route for (D=cloud, S=ISP-A-PA) into the IGP.  This IGP advertisement is received by a router (R50) on the same LAN as H55.  I assume that R55 would use Neighbor Discovery Router Advertisements to communicate information about  (D=cloud, S=ISP-A-PA) to H55.  I don't think that PIO and RIO in ND Router Advertisements currently have the semantics to advertise associate arbitrary source and destination prefixes, so I think that ND Router Advertisements would need to be extended.  This is basically what draft-sarikaya-6man-sadr-ra-03 is proposing.  

I think that different participants in this discussion may have different ideas about how this should be done.  The homenet architecture principles document (RFC 7368) and draft-ietf-6man-multi-homed-host-06 both point to RFC 6724 as providing a solution for source address selection with multi-homing, but both documents also seem to acknowledge that RFC 6724 doesn't provide a complete answer.  In the discussion in RTGWG on Thursday, Fred Baker said that thinks a Happy Eyeballs RFC 6555 should play a roll as well.  There was also a mention of having the host use ICMP to detect failures.

I think we need to be very explicit about our assumptions about how the host is going to select the source address, especially if we expect the host to use dest/source route information.   This is also consistent with Alvaro Retana's more general request in RTGWG that we consider a complete solution to the PA multi-homing problem, as opposed to just considering individual pieces of a potential solution in isolation.

==========

Responding to a another question below, I can clarify my concerns about the scalability and implementation of dest/source routing that allows pairing of arbitrary destination and source prefixes. Dest/source routing has been proposed as a solution for several different problems or use cases.  So I'll state my concerns for each use case.

1) PA multi-homing for homenet - I don't have much concern regarding scalability and implementation for PA multi-homing for homenet. This is not my area of expertise, but I would expect most forwarding implementations for homenet to be software based.  So it seems like existing equipment should be able to deal with new forwarding requirements, possibly with some performance penalty. 

2) PA multi-homing for mid-size enterprise networks - My main concern with PA multi-homing for mid-size enterprises is that existing routers deployed in these networks will not be able to support forwarding based on pairing of arbitrary destination and source prefixes.  In mid-size enterprise networks, I would imagine that there is a broad range of existing deployed equipment, some of which may have forwarding hardware that cannot be modified to support arbitrary dest/source route lookups on packets.  I think it is likely that more deployed equipment would be able to be upgraded via software to support a more simplified form of dest/source routing. 

3) Dest/source routing in IGPs as a general tool for service provider infrastructure routing - One example of this kind of use case can be found in draft-baker-rtgwg-src-dst-routing-use-cases-01 under " Intra-domain access control".  Slide 4 of https://www.ietf.org/proceedings/95/slides/slides-95-rtgwg-9.pdf also discusses some uses in CERNET2 of dest/source routing in OSPF (draft-xu-ospf-multi-homing-ipv6-00).  These use cases are very different from PA multi-homing.  My main concern here is that there are existing standardized and deployed solutions addressing these use cases.  That doesn't mean that the IETF can't standardize another solution to the same problem, but I think there is generally a higher bar with respect to demonstrating the need for another solution, and analyzing the costs and benefits relative to existing solutions.

4) Dest/source routing in BGP - draft-xu-src-dst-bgp-00 proposes extending dest/source routing to BGP with arbitrary matching of destination and source prefixes.  The draft doesn't discuss the use cases, but on the surface, it seems like some uses of this could result in a significant increase in routes in the global internet.  

Thanks,
Chris

-----Original Message-----
From: David Lamparter [mailto:[email-masked]] 
Sent: Thursday, April 07, 2016 11:55 AM
To: [email-masked]
Cc: Chris Bowers <[email-masked]>; Fred Baker (fred) <[email-masked]>; Anton Smirnov <[email-masked]>
Subject: Implications of default-only SADR (was: Re: multi-homing for provider-assigned IPv6 addresses)

TL;DR:  note at end about "abort as unreachable" as alternative way to get rid of complicated backtracking.  (Jump to second/last ----- line.)

Hi all,

here is another shot at my somewhat confusing (sorry) in-room comment on the policy aspect of limiting to D=::/0.

User (let's say, small-to-medium-ish business, for illustration let's say the internal network has 2-3 OSPF routers or something.)
\- ISP A, assigned PA prefix, internet/default connectivity
\- ISP B, assigned PA prefix, internet/default connectivity

Now, let's say the CEO of the company heard that clouds are great for putting your calendar and mails into.  She decides ISP A's "Cirrus Castellanus" product is great because it comes with some great security and confidentiality bulletpoints.

For the Cirrus Castellanus service, ISP A installs a separate device in the user's network, which routes a specific prefix for the cloud.  Could be some encrypted VPN foobar, physical separate line, doesn't really matter.  Since it's ISP A, they're aware of their PA assignment to the customer and push a route for that into the cloud product.  The cloud product could implement BCP 38, but it could also just not have a route towards the internet - it doesn't need to and it's "secure".

Now, the installation instructions for the product says "this is the prefix your cloud service has, put a route for it in your domain".

So, now the User's netadmin is essentially screwed since that's a D!=default S!=default route.  More correct, it's in fact 2 routes:
- D=cloud, S=ISP-A-PA => via installed cloud termination device
- D=cloud, S=::/0 => unreachable
Which goes to represent that the cloud service is only reachable from ISP-A's PA prefix assigned to the User.

----------
This, I hope, illustrates how this is a significant policy decision to make.  It's IETF consensus (I hope) that NAT is a bad thing.  It also seems to be IETF consensus (as discussed in Fred's slides) that PA is preferable over PI, and we want to keep tables small.  We've received feedback -the v6ops request- that PA has some issues to be solved.  If we now say "we're doing D=::/0 only", we're deciding to solve only part of the problem posed to us.  (see note below about divide&conquer / "do
D!=::/0 separately")

In any case, any solution that resticts to D=::/0 carries an implication
of:
   "If you're using PA, the only thing you can rely on being able to
    route is the internet as a whole.
If we're making this policy assumption (which is really a decision), we need to be aware we're making it.

On Divide&Conquer:
is certainly possible to write down, but complicates the compatibility mechanisms in the routing protocols since level-1 and level-2 routers are not compatible to be in the same link-state/SPF topology.

----------
If we still want the backtracking behaviour (going to less specific D when not finding a S match under a particular D) gone, I'd prefer looking at specifying "terminate as unreachable" instead of completely removing D!=::/0 support.  I haven't thought through use-case impact of this yet, but in any case the backtracking behaviour could be restored by advertising additional routes to the same effect.

(The same approach of installing additional routes can be used to eliminate the performance hit of backtracking by consuming more memory / hardware route resources through installing additional synthetic routes.
I'll add that to the next version of the draft.)

On a closing note, I'll repeat my mic request for feedback:  could people provide feedback which part of backtracking they're concerned about?  Specification complexity?  Implementation complexity?
Performance penalities?  Something else?

-David

---

From: [email-masked]
Date: Tue, 12 Apr 2016 09:01:43 +0200

Chris,

> I think we need to be very explicit about our assumptions about how the host is going to select the source address, especially if we expect the host to use dest/source route information. This is al…

OK, so you asked for it. ;-)

Let me try to quickly outline all the assumptions here.

In a multi-prefix multi-homed case a host has one or more IPv6 addresses from more than one IPv6 prefix.

 - The host assumes that by picking source address it also chooses the exit in the network.
    In the case where a host is directly connected to two independent exit routers, see below 6man draft for next-hop selection.
   This assumption implies that the network is capable of SADR.

 - To achieve the goals set out in RFC3582, the burden of multi-homing is squarely put on the shoulders of the host.
   It must be capable of picking the best path, load-share across multiple paths, detect failures and switch over to a
   functioning path etc...
   That requires either very smart applications, like MOSH. Or it requires improved transport, like MPTCP or perhaps SCTP.

   Initial path selection, is either use them all (like MPTCP), or throw spaghetti on the wall and pick the ones that stick.
   It is expected that the host maintains some heuristics over time, so it doesn't necessarily have to try Sn*Dn combinations for each connection. RFC6724 gives you the "cheap" version of this by falling back to 'topologically closest' S,D.

- As you can see the assumptions put on the routing layer is pretty trivial compared to what the host has to do.

- Plan B is ILNP or resort to 3rd party multi-home proxies like what LISP can do.

General considerations:
 RFC7157 - IPv6 Multihoming without Network Address Translation

Host source address selection:
 RFC6724 - Default Address Selection for Internet Protocol Version 6 (IPv6)
 RFC7078 - Distributing Address Selection Policy Using DHCPv6
 RFC6731 - Improved Recursive DNS Server Selection for Multi-Interfaced Nodes

Host next-hop selection:
  draft-ietf-6man-multi-homed-host-06 - Routing packets from hosts in a multi-prefix network

Carrying edge specific meta information with prefixes, aka provisioning domains:
 RFC7556 - Multiple Provisioning Domain Architecture

Best regards,
Ole

---

From: Chris Bowers <[email-masked]>
Date: Tue, 12 Apr 2016 14:19:06 +0000

Ole,

Do you think that Neighbor Discovery needs to be extended to support the advertisement of (S,D) routes?

Currently ND Router Advertisements allow the advertisement of source prefixes in PIOs and destination prefixes in RIOs, but I don't see any existing mechanism to associate a particular source prefix with a particular destination prefix.  draft-sarikaya-6man-sadr-ra-03 offers one way to associate a source prefix with a destination prefix in Router Advertisements.

If not, what mechanism do you think should be used to communicate (S,D) routes to the host?

Thanks,
Chris

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, April 12, 2016 2:02 AM
To: Chris Bowers <[email-masked]>
Cc: David Lamparter <[email-masked]>; [email-masked]
Subject: Re: Implications of default-only SADR (was: Re: multi-homing for provider-assigned IPv6 addresses)

Chris,

General considerations:

Host next-hop selection:

Best regards,
Ole

---

From: [email-masked]
Date: Tue, 12 Apr 2016 19:24:33 +0200

Chris,

> Do you think that Neighbor Discovery needs to be extended to support the advertisement of (S,D) routes?

No, I'm hoping we can avoid that.

> If not, what mechanism do you think should be used to communicate (S,D) routes to the host?

In the general case explicit S,D routes to hosts are not needed.

There are two cases, first when a host is connected to two uncoordinated routers (e.g. ISP A, ISP B), in that case draft-ietf-6man-multi-homed-host clarifies that a host should associate source prefix with the default from router discovery. I.e. an implicit instead of explicit S,D route.

In the case, e.g. homenet it is assumed that all routers are part of the same IGP, and if host picks 'wrong' router, the only consequence is an extra hop.

If a host really really wants to participate in routing, it can of course run a routing protocol.

Do you see anything missing?

Best regards,
Ole

---

From: Chris Bowers <[email-masked]>
Date: Tue, 12 Apr 2016 19:16:00 +0000

Ole,

Consider the topology below.  See https://www.ietf.org/mail-archive/web/rtgwg/current/msg05472.html for a more detailed description of the topology.  For H31 to send a packet to destination B3, H31 must choose a source address from within subnet A3x.
                                             -----------------
   LAN-X        LAN-Y          ,-----.      /                 \
          +--+         +--+  ,'       `.   :                   :

For this example, we assume that the R1-R4 originate the following (D,S) routes in the IGP.
R1 originates a route for (D=::/0, S=A1).
R2 originates a route for (D=B2, S=A2).
R3 originates routes for (D=::/0, S=A3) and (D=B3, S=A3).
R4 originates routes for (D=::/0, S=A4) and (D=B4, S=A4).

R7 and R8 receive these routes via the IGP.  With the existing mechanisms in Neighbor Discovery Router Advertisements, R7 and R8 can advertise the following PIOs and RIOs.
PIOs = A4x, A2x, A1x, A3x
RIOs = B2, B3, B4, B1

I have intentionally changed the order of the prefixes in the set of PIOs and RIOs to emphasize that there is no required ordering or relationship between prefixes in PIOs and RIOs.

With only this information, I do not see how H31 can correctly choose a source address in A3x when it needs to send a packet with destination address B3. If this analysis is correct, then it seems like a mechanism like draft-sarikaya-6man-sadr-ra-03 is needed.

Thanks,
Chris

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]
Sent: Tuesday, April 12, 2016 12:25 PM
To: Chris Bowers <[email-masked]>
Cc: David Lamparter <[email-masked]>; [email-masked]
Subject: Re: Implications of default-only SADR (was: Re: multi-homing for provider-assigned IPv6 addresses)

Chris,

Best regards,

Ole

---

From: [email-masked]
Date: Tue, 12 Apr 2016 21:36:32 +0200

Chris,

> Consider the topology below.  See https://www.ietf.org/mail-archive/web/rtgwg/current/msg05472.html for a more detailed description of the topology.  For H31 to send a packet to destination B3, H31…

[very nice ASCII art ruined by my stupid MUA]

> With only this information, I do not see how H31 can correctly choose a source address in A3x when it needs to send a packet with destination address B3. If this analysis is correct, then it seems…

I take it B1, B2, B3 and B4 are walled gardens.

What I think you are suggesting is that the host could use a S,D FIB for source address selection. At least type D hosts could. https://tools.ietf.org/html/draft-pfister-6man-sadr-ra-01

RFC7078 is meant to solve this case though.

Best regards,
Ole

---

From: Chris Bowers <[email-masked]>
Date: Tue, 12 Apr 2016 20:47:00 +0000

Ole,

So here is my summary of the situation.

1) If DHCPv6 with extensions in RFC7078 is used to help the host to select the correct source address, then we could handle even walled garden use cases with default-only SADR.  See details for how this works in http://www.ietf.org/mail-archive/web/rtgwg/current/msg05472.html .  Another assumption would be that DHCP relay could be used so we wouldn't need the (D!=::/0 , S) routes in to reach R7 and R8.  

2) However, if we don't use DHCPv6 with RFC7078 for host source address selection, then Neighbor Discover Router Advertisements need to be extended using a new mechanism like draft-sarikaya-6man-sadr-ra-03 or draft-pfister-6man-sadr-ra-01.  In this case, default-only SADR is not sufficient to support walled gardens because we are relying on the (D!=::/0 , S) routes to be carried in the IGP so that R7 and R8 can then put the (D!=::/0 , S) information into the new ND Router Advertisements.  

Is this more or less accurate?

Thanks,
Chris

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, April 12, 2016 2:37 PM
To: Chris Bowers <[email-masked]>
Cc: David Lamparter <[email-masked]>; [email-masked]
Subject: Re: Implications of default-only SADR (was: Re: multi-homing for provider-assigned IPv6 addresses)

Chris,

Best regards,
Ole

---

From: Jeffrey Haas <[email-masked]>
Date: Wed, 13 Apr 2016 08:57:09 -0400

> ISP-A's PA prefix assigned to the User.

And it's not too far away from cases where private (e.g. RFC 1918) type
addresses are only reachable or differently reachable via a given provider;
e.g. company VPN.

The general presumption for default is "you can reach the whole of the
Internet".  There's lots of circumstances where IP space really isn't "the
whole of the Internet".  While some of these issues can be addressed by
having the host just try access via its available networks, many of them
probably shouldn't be done this way.

-- Jeff

---

From: [email-masked]
Date: Wed, 13 Apr 2016 23:47:51 +0200

Chris,

> Is this more or less accurate?

Yes, that's reasonably accurate. Although of course not the full picture since we have decades of innovation in this space.
- You also have ICMPv6 Destination Unreachable, Source address failed ingress/egress policy that could be used as a trigger by the host to try a different source, independent of what the network signals.
- Or MP-TCP which presumably would also try some subset of all combinations.
- Or draft-baker-happier-eyeballs-00 which throws spaghetti on the wall to see what sticks.

Cheers,
Ole