Subject: Re: Generic anycast addresses...

From: Michael Richardson <[email-masked]>
Date: Sat, 01 Jun 2019 10:05:08 -0400

Mark Smith <[email-masked]> wrote:
    >> hop 1 is the router at home, and hope 2 is the access router at my ISP, and
    >> hop 3 is my ISP's core router connect to upstream peers.
    >>
    >> Should hop 1 (home router) or hop 2 (access node) were to blackhole route
    >> fc00::/6 (ULA-R and ULA-C), would that affect your use case?  Or could the
    >> anycast service possibly be at the ISP?

    > I'm guessing you don't have internal ULA address space, which is why you're
    > more successful than if you did.

I do have a lot of ULA, but I didn't try from within those segments, because I
was not at my office and I couldn't get there from the conference I was at.

    >> Should home routers install routes to 2000::/3 when they see "default"
    >> rather
    >> than "::/0"?  I have made that argument, but I see the other point about
    >> how limiting it could be in the future.

    > A ULA address at an ISP would only be reachable to customers who only have
    > GUA addresses.

    > If a customer has ULA internally, a ULA source address would be preferred
    > over a GUA source to reach an ISP ULA (anycast) destination. That ULA
    > source address would cause the packet to be dropped by a BCP38 filter at
    > the ISP access router, assuming they have them, as they should.

    > Even if the ISP doesn't have BCP 38 filters, the ISP's routing is very
    > unlikely to have routes back to all customers' ULA address spaces.

This is a good point you make.
So should home routers do BCP38 filtering on their external interface?
(Better to drop it early)

    > So you need global scope anycast addresses for working source address
    > selection. Except that defeats the goal of having anycast destined packets'
    > travel being restricted to a domain smaller than global when that is
    > required or desirable.

Do you think we need another scope of address?

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Ted Lemon <[email-masked]>
Date: Sat, 1 Jun 2019 07:07:06 -0700

> (Better to drop it early)

Yes.

---

From: Ole Troan <[email-masked]>
Date: Sat, 1 Jun 2019 17:11:44 +0200

> I was looking through the IANA registry for anycast addresses with an idea of what I wanted, and was surprised to learn that no such thing exists.   I’m curious if what I want is something that’s already been shot down in flames, or something for which no energy has existed to do.
> 
> Right now it appears that anycast addresses either aren’t special (that is, they are just IP addresses in someone’s prefix) or are link-specific (e.g., the subnet router anycast address, which if I understand it correctly is constructed of <local-prefix>::0).
> 
> What I am looking for is an anycast address that won’t match any local prefix, so that it filters out towards an egress router and is caught somewhere along the way, or worst case, at the egress.   I can see where this would have gone down in flames, since we don’t want anycast packets to keep going toward the backbone and create congestion, so that might explain why this hasn’t happened.   But we do have the notion of scopes, e.g. for multicast, and that would seem to apply for anycast as well.   We do allow multicast in scopes larger than the local subnet, and AFAIK this has not melted the Internet.
> 
> The actual use case I have for this is wanting to be able to have a constrained device send a unicast discovery or announcement which can be assumed to be caught and handled by infrastructure.
> 
> So, is this something that’s been talked about and abandoned as a terrible idea, or abandoned because nobody wanted to do the process to make it happen, or is it (seems unlikely) an innovation on my part?   Or is it already done and I just managed to not find the document describing it?

You want something like: https://tools.ietf.org/html/draft-ietf-ipv6-dns-discovery-07
?

Would this be for boot-strapping service discovery, so one would only need a single "service" address ever?

Ole

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 2 Jun 2019 09:06:32 +1200

>     > more successful than if you did.

For example:

C:\WINDOWS\system32>tracert fd2e:82a1:f3c4::1

Tracing route to fd2e:82a1:f3c4::1 over a maximum of 30 hops

  1     2 ms     2 ms     2 ms  fritzy [fd63:45eb:dc14:0:be05:43ff:fe8e:ce39]
  2  General failure.

Trace complete.

fritzy is my CPE, of course.

> Do you think we need another scope of address?

I think we need a better definition of scope boundaries, which clearly cannot be well defined just by bits in an address prefix. But I have a whole draft about that, and it's not particularly an IPv6 problem.

    Brian

---

From: Mark Smith <[email-masked]>
Date: Sun, 2 Jun 2019 10:35:00 +1000

> Do you think we need another scope of address?

I think we need a formal, multi-scoped anycast address space.

Anycast also has enough properties in common with multicast that I think it
should be more than just a configured special case address within the
unicast address space. I think it should be a distinct class in between
unicast and multicast.

Unicast: 1:1
Anycast: 1:Any
Multicast: 1:Many

IPv6 Formal Anycast Addresses and Functional Anycast Addresses

https://tools.ietf.org/id/draft-smith-6man-form-func-anycast-addresses-00.txt

Regards,
Mark.

---

From: Sander Steffann <[email-masked]>
Date: Sun, 2 Jun 2019 11:14:15 +0200

Hi Mark,

> Anycast also has enough properties in common with multicast that I think it should be more than just a configured special case address within the unicast address space. I think it should be a distinct class in between unicast and multicast.
> 
> Unicast: 1:1
> Anycast: 1:Any
> Multicast: 1:Many

I agree. Explicit is better than implicit.

> https://tools.ietf.org/id/draft-smith-6man-form-func-anycast-addresses-00.txt

Thanks!
Sander

---

From: Toerless Eckert <[email-masked]>
Date: Sun, 2 Jun 2019 15:03:00 +0200

As you are repeating your argument to make your case, let me
respectfully also disagree again, but also suggest what IMHO would be
a safer and more flexible approach.

IMHO, it would be lovely to primarily think of standardizing whats
IMHO been most missing piece for more proliferation of anycast, and
thats an appropriate prototol for anycast members to signal their membership. 
That would get a us a lot further proliferating benefits of anycast.
If defined appropriately rich to indicate scope/zone for example, IMHO we would
solve the scope problem as well without having to creating more and
more specialized address spaces. 

There are also other policies for redundant IP addresses
than anycast (nearest member). For example, what i called
"prioritycast", where you want the current highest-priority member of
the group to be the destination. And in a time where delay becomes for
many applications as important if not more so than throughput, "nearest"
could have different bandwidth vs. delay interpretations and routing
policies.

I can see how there will be an ongoing demand to support lazy, oops:
easy "discovery" options with well-known addresses, but lets not assume
that we should or could come up with a generally applicable better working
address encoded scoping solutions than we did in the past or assume that
one specific form of redundany (IGP SPF "shortest") is the only necessary policy.

Aka: I think for each application that needs one or a limited number
of well-known addresses an IANA allocation could be given from a TBD
larger block of well-known "functional unicast addresses" for which
the default policy is not to route, and the policy for each allocation
is to define (via an RFC) how to route/forward it. Effectively this would
mean this block must not be forwarded via aggregated routes, because the
route policy of each allocation should support to be differernt: notion
of scope/zones/anycast/priority/....

This would allow Ted to come up with a routing policy for the
adress(es) he wants to have and his definition of what an appropriate
"scope/zone" is, without us (IETF) having to agree that that is a universally
reusable notion of scope/zone.

The only downside for an app like the one from Ted would be that he
wouldn't get on riding on the default route for free but that whoever
wants to use the address has to explicitly add the right routes and routing
policy. Of course this may only be true in the future depending of which
address space we carve out this address space from - if its carved out
of existing global address space it would be routed via default-route by
legacy equipment. 

Cheers
    Toerless

---

From: Nick Hilliard <[email-masked]>
Date: Sun, 2 Jun 2019 14:31:19 +0100

> thats an appropriate prototol for anycast members to signal their membership.

as an observation, this would introduce host-level state into the 
network, which has scalability consequences.

Nick

---

From: Mark Smith <[email-masked]>
Date: Mon, 3 Jun 2019 09:42:48 +1000

On Sun., 2 Jun. 2019, 23:03 Toerless Eckert, <[email-masked]> wrote:

> As you are repeating your argument to make your case,

What am I supposed to do when people ask about things already covered in my
draft?

Here's the link again.

https://tools.ietf.org/html/draft-smith-6man-form-func-anycast-addresses-00

let me

>From my draft,

Hmm, there
   really needs to be any "Anycast Listener Discovery" protocol similar
   or same as MLD.).

---

From: joel jaeggli <[email-masked]>
Date: Sun, 2 Jun 2019 16:47:59 -0700

> thats an appropriate prototol for anycast members to signal their membership. 

You mean like BGP or are you refering to something else?

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------
>

---

From: Mark Smith <[email-masked]>
Date: Mon, 3 Jun 2019 10:07:12 +1000

> network, which has scalability consequences.

This is a well understood problem, and is called out in RFC 1546.

This issue would be the same one created by ES-IS, except that ES-IS has
all hosts inject their addresses into the routing domain (see RFC 995).
That was designed in the mid-1980s. So if it was a solvable problem then,
it is a much more easily solved problem now given processing and memory
advances since.

Of course, the solution is well known, being route summarisation or
aggregation (and solved for IS-IS with Level-1 levels).

"Forwarding towards anycast addresses is the same as forwarding
   towards unicast addresses, which uses the longest match rule BCP
198 <https://tools.ietf.org/html/bcp198>
   [RFC7608 <https://tools.ietf.org/html/rfc7608>].  Longest match
forwarding facilitates summarisation of
   forwarding information, where a single more general forwarding route

Smith                    Expires April 25, 2019                 [Page 8]

------------------------------

  <https://tools.ietf.org/html/draft-smith-6man-form-func-anycast-addresses-00#page-9>Internet-DraftIPv6
Formal And Functional Anycast Addresses  October 2018

   can summarise a number of more specific forwarding routes.
   Summarisation saves entries in forwarding tables outside of the
   summarised forwarding domain, provides simpler destination based
   filtering for security purposes, and facilitates easier destination
   address based traffic analysis.

   The use of route summarisation with anycast addresses effectively
   creates an anycast domain that is being identified and summarised by
   the anycast summary route.  Outside of the anycast domain, a single
   summary route exists, covering all anycast addresses within the
   domain.  Within the anycast domain, individual routes for individual
   anycast addresses exist.

   When designing a new Anycast Identifier field format and structure,
   the following guidelines should be followed.  These guidelines should
   allow a set of more specific anycast "routes to be summarised as well
   as improving operator usability.

"Note that this guildline should not take

Smith                    Expires April 25, 2019                 [Page 9]

------------------------------

  <https://tools.ietf.org/html/draft-smith-6man-form-func-anycast-addresses-00#page-10>Internet-DraftIPv6
Formal And Functional Anycast Addresses  October 2018

      precedence over any previous measures to faciliate more specific
      anycast route summarisation."

"External to the anycast domain, the
      identifying 64 bit prefix can be used to create a single summary
      route for the anycast function or service identifier space, which
      will help routing scaling for anycast functions or services."

Regards,
Mark.

---

From: Mark Smith <[email-masked]>
Date: Mon, 3 Jun 2019 11:02:03 +1000

> fritzy is my CPE, of course.

Likely implementing RFC 7084.

"In this role, the IPv6 CE router is responsible for ensuring that
   traffic using its ULA addressing does not go out the WAN interface
   and does not originate from the WAN interface."

> I think we need a better definition of scope boundaries, which clearly
> cannot be well defined just by bits in an address prefix.

That seems to have been adequate for multicast, and I think we're in a
similar problem space.

Multicast scopes are defining forwarding domains (replacing the use of the
TTL field) and are an input into source address selection. They don't
specify anything else. Things such as if the multicast group ID is IANA
defined or not is formally encoded somewhere else (T flag bit).

But I have a whole draft about that, and it's not particularly an IPv6
> problem.

I think your draft is covering the topic of network domains much more
broadly.

The forwarding domain of a packet is a specific example of the more general
idea of a network domain.

There are of course domain types that don't have much or anything to do
with a packet forwarding domain, such as a payload encryption domain.

Regards,
Mark.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 3 Jun 2019 09:44:14 +0200 (CEST)

> https://tools.ietf.org/html/draft-smith-6man-form-func-anycast-addresses-00

I have now read RFC1546 and now I understand some of the confusion. Did 
the concept of "send SYN to anycast address and get SYN-ACK back from 
actual unicast address" ever get implemented anywhere?

Anyhow, the RFC1546 use of anycast never caught on from what I have seen. 
I have been doing IP networking for 25 years and I've done numerous BGP 
anycast deployment for different services, and this just involves treating 
them as unicast addresses (configured on each host in the anycast cluster) 
but that happens to be ECMP:ed to multiple hosts. When one says "anycast" 
to 99.9% of the networking world, this is what they'll think of. This is 
what draft-smith-6man-form-func-anycast-addresses seems to call "informal 
anycast addresses". It's good (and absolutely needed) to make the 
distinction here, to avoid confusion.

Now, if we're going to do the whole "functional anycast addresses", is the 
expectation that it's going to add the example RFC1546 behaviour for TCP 
sessions, thus needing changes to all stateful L4 protocols in the host 
stack? That's a huge change, and I'm pretty sure the security requirements 
section would be huge? If not, what does the whole scoping actually add? 
What does all this information actually add over well-known addresses out 
of regular unicast space that we're already using?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Mon, 03 Jun 2019 10:49:20 -0400

> I have now read RFC1546 and now I understand some of the confusion. Did the
    > concept of "send SYN to anycast address and get SYN-ACK back from actual
    > unicast address" ever get implemented anywhere?

I don't think so.
Maybe some BSD 4.0 could support it, but it would be fly in the face of most
firewalls.  It would rely only on the sequence number to match things up.
That would be pretty weak, particularly at the time RFC1546 was published!

If IPsec was ubiquitous in the way imagined once, I believe that we could
make IKEv2 to anycast, respond from unicast work well securely enough.
If well integrated with the stack, the SYN would go out with the correct
unicast destination learnt from IKEv2.  Alas, IPsec is not ubiquitous like
this :-(

    > section would be huge? If not, what does the whole scoping actually add? What
    > does all this information actually add over well-known addresses out of
    > regular unicast space that we're already using?

It means that we can effectively drop them when they get to the edge of the
scope.   It means that a scoped anycast can be used in the home network for
something, and then *also* used at the ISP for a similar use without
confusing things.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 03 Jun 2019 10:55:30 -0400

I upgraded my home router from CeroWRT to OpenWRT 18.06 on the weekend.
(It was way overdue, but complicated by the complexity of my network,
and the fact that CeroWRT introduced some very useful firewalling which 18.06
did not adopt)

My ISP does DHCPv6-PD correctly now, but alas, isn't providing the correct
prefix.  In debugging, I noticed that it is now using Source-Address specific
routing:

root@budwrt18:/etc/config# ip -6 route
default from 2607:f0b0:7:1d0::/64 via fe80::fe5b:39ff:fe89:7b00 dev pppoe-wan  metric 512
default from 2607:f0b0:7:8350::/64 via fe80::fe5b:39ff:fe89:7b00 dev pppoe-wan  metric 512

(why two prefixes, I'm unclear)

The upshot is that all other prefixes will fall flat at the border, which is
rather what we want.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 3 Jun 2019 17:08:33 +0200 (CEST)

On Mon, 3 Jun 2019, Michael Richardson wrote:

> My ISP does DHCPv6-PD correctly now, but alas, isn't providing the 
> correct prefix.  In debugging, I noticed that it is now using 
> Source-Address specific routing:

Yes, openWrt has done that for a while. I am surprised cerowrt didn't.

> (why two prefixes, I'm unclear)

Is your ISP only handing you /64 and no /56? This is what I have (I have 
another router in my home that gets /62 from my openwrt router):

default from 2001:9b0:XXXX:XX00::/56 via fe80::e681:84ff:fe41:740f dev eth1.2 proto static metric 512 pref medium

2001:9b0:21a:3100::/64 dev br-lan proto static metric 1024 pref medium

2001:9b0:XXXX:XX08::/62 via fe80::6666:b3ff:fede:fffd dev br-lan proto static metric 1024 pref medium

unreachable 2001:9b0:XXXX:XX00::/56 dev lo proto static metric 2147483647 error -113 pref medium

If I would have had multiple uplinks and had two different ISPs both 
handing me /56:es, this would have properly made source based routing to 
work to avoid BCP38 filtering on the upstream.

> The upshot is that all other prefixes will fall flat at the border, 
> which is rather what we want.

My biggest beef with the scoping is that we're going to have to agree on 
topology, what a "site" is, etc. This has been historically hard.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 03 Jun 2019 14:46:30 -0400

Mikael Abrahamsson <[email-masked]> wrote:
    >> prefix.  In debugging, I noticed that it is now using Source-Address
    >> specific routing:

    > another router in my home that gets /62 from my openwrt router):

I'm not sure.
I have a /56 statically routed to me, but that was done before they could
support PD.   I don't know why I have two different /64s delegated, and I've
already asked about this.

    >> The upshot is that all other prefixes will fall flat at the border, which
    >> is rather what we want.

    > topology, what a "site" is, etc. This has been historically hard.

I know, but I think that it's okay if we get it right automatically for 80%
of the cases, as they probably cover 99% of the usage.

---

From: Ted Lemon <[email-masked]>
Date: Mon, 3 Jun 2019 14:48:10 -0400

What size prefix are you requesting?

Sent from my iPhone

> On Jun 3, 2019, at 2:46 PM, Michael Richardson <[email-masked]> wrote:
> 
>

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 4 Jun 2019 01:31:15 +0200

On Sun, Jun 02, 2019 at 02:31:19PM +0100, Nick Hilliard wrote:
> as an observation, this would introduce host-level state into the network,
> which has scalability consequences.

The host-level state also exists today, when you just setup anycast
address routing for a couple of e.g.: anycast DNS servers. Its just more
complicated because its not fully automated: you need to hav an operator
or SDN controller create those anycast route prefixes and then make sure
that the host config is aligned.

Aka: not disagreeing with your startement about state, just with the
statement that host<->network signaling changes it. It just automates
it and removes manual, unaligned config issues, accelerates deployments,
....

Cheers
    Teroelss

---

From: Christian Huitema <[email-masked]>
Date: Mon, 3 Jun 2019 16:42:16 -0700

> That would be pretty weak, particularly at the time RFC1546 was published!

There is support for this kind of behavior in QUIC with the "preferred
address" transport parameter, although it is quite different from what
is stated in RFC1546. From the spec, "QUIC allows servers to accept
connections on one IP address and attempt to transfer these connections
to a more preferred address shortly after the handshake. This is
particularly useful when clients initially connect to an address shared
by multiple servers but would prefer to use a unicast address to ensure
connection stability." In that scenario, the full handshake is performed
over the anycast, but the client tried to opens a new path after that to
the "preferred address" of the server. If the path verification
succeeds, the exchange moves to the new address.

-- Christian Huitema

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 4 Jun 2019 01:45:30 +0200

> it is a much more easily solved problem now given processing and memory

I also think we learned a lot more about possible policy parameters to
direct the desired routing for such an address. It likely also needs
to include ome way to ask for allocation of an address.

Aka: Yes, we know a lot more, but if we want to make it a lot better
than what we would have done 20..30 years ago, we still need to brainstorm
agree on several aspects.

> aggregation (and solved for IS-IS with Level-1 levels).

You can only aggregate when the policies actually align, so i would not
in general bet that aggregation is going to be so important. I would
rather count on the total number of required anycast addresses to be
small enough to be easily supported without aggregation.

> >From my draft,

[..]

Thanks, probably a lot more details to be discussed easier
offline/in-person.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Michael Richardson <[email-masked]>
Date: Mon, 03 Jun 2019 20:50:25 -0400

> to a more preferred address shortly after the handshake. This is

That's really cool.
It's just sad that we have to do it at layer 6 rather than layer 3 :-(

It's nice that we finally have a layer-5 security layer that seems to
becoming ubiquitous.

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 4 Jun 2019 15:06:26 +1200

> It's just sad that we have to do it at layer 6 rather than layer 3 :-(

If you think about it, shim6 was really an attempt to do this sort of thing
at layer 3. I never thought of it that way, but shim6 to an anycast address
could trigger an immediate address switch to a unicast address.

I'm not suggesting that this is a practical approach - but it's very close
to a transport-independent solution.

    Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 4 Jun 2019 08:19:17 +0200 (CEST)

> already asked about this.

"tcpdump -i pppoe -vvv port 546 or port 547". Send SIGUSR1 to odhcp6c to 
make it renew the lease, then you can check what's going on.

---

From: Sander Steffann <[email-masked]>
Date: Tue, 4 Jun 2019 14:19:41 +0200

Hi Brian,

> to a transport-independent solution.

It might be interesting to look at it again in this context. Who knows :)

Cheers,
Sander

---

From: Christian Huitema <[email-masked]>
Date: Tue, 4 Jun 2019 05:54:25 -0700

> It might be interesting to look at it again in this context. Who knows :)

I am not sure that we could safely engineer these functions in a shim
layer like Shim6. QUIC can do it because it is an encrypted transport.
Path continuity can be verified and proven by demonstrating knowledge of
the encryption keys, which are negotiated end to end as part of the
transport setup.

That said, we are a bit far from the original discussion of generic
anycast addresses.

-- Christian Huitema

---

From: Toerless Eckert <[email-masked]>
Date: Tue, 4 Jun 2019 20:31:04 +0200

Christian:

I would certainly like a mechanism to resolve from anycast to unicast
that is independent of a specific transport protocol like QUIC. I guess
the interesting new requirement that QUIC introduced into that equation
is that such resolution should avoid any additional round-trip.

One way to do this would be to have a generic simple anycast resolution
mechanism with one request and one reply and both can carry a
transport protocol specific payload. Only thing it costs is that those
two packets reduce the transport size available to the first request/reply,
not sure how problematic this would be. But at least it would be lovely
not having to reinvent that wheel for QUIC, TCP, STCP and the vairiety
of UDP transports we have.

> the encryption keys, which are negotiated end to end as part of the

I guess for the anycast resolution request/reply, a none should be
sufficient. Whether the transport uses encryption or not is unaffected
by this. What the transport is affected by is the notion that the
transport connection ID does not use the anycast address but the
resulting unicast address.

> That said, we are a bit far from the original discussion of generic

Easily resolved by chnge of subject ;-)

Cheer
    Toerless

---

From: Christian Huitema <[email-masked]>
Date: Tue, 4 Jun 2019 21:51:30 -0700

> is that such resolution should avoid any additional round-trip.

It is not so much "round trip" as "make before break".

> of UDP transports we have.

I am not sure of that at all. QUIC can innovate because the
implementations are in user space, and thus do not depend on specific
platform upgrades. Changes can be tailored to the need of the
application. Also, QUIC embeds TLS. In any case, there already are
multipath solutions for TCP and SCTP, so there is probably no need to
invent a new one.

> resulting unicast address.

Nonce based solutions are not robust against man-on-the-side attacks,
which is actually a big concern in several application domains. In
practice, you want something like TLS to protect path decisions.

I think that transitions from anycast to unicast are forms of path
management, and that path management belongs in the transport layer.

-- Christian Huitema

---

From: Brian Haberman <[email-masked]>
Date: Tue, 11 Jun 2019 10:04:18 -0400

> of UDP transports we have.

There have been on/off discussions for years among a group of IETF
participants about using the Home-Address option in MIP6 to do such
anycast to unicast mapping.

Brian