Subject: Metadata over IPv6

From: Brian Haley <[email-masked]>
Date: Tue, 17 Dec 2019 11:41:55 -0500

Hi ipv6-list,

I was an IPv6 contributor many moons ago, and remembered this list when 
looking into something on a new project, was hoping to get some IPv6 
advice (below).

The current project I'm working on, Openstack Neutron, is an SDN 
networking project focused on delivering networking-as-a-service (NaaS) 
in virtual compute environments.

One thing that happens when a virtual machine boots is it typically asks 
for metadata, thing like ssh keys, and other configuration information 
required to make it functional.  It does this via requests to the URL 
http://169.254.169.254/latest/meta-data/... ([0] has more complete 
info).  This link-local IPv4 address was defined by AWS and is widely 
used across all types of clouds.

This works fine for dual-stack hosts, but more and more we're seeing 
IPv6-only networking scenarios that we don't support metadata with, so 
our community is looking to define an IPv6 address to use for this 
service.  My question to the list is - do you see a problem with us just 
defining an IPv6 link-local address for this same service?  Or do you 
think we need to propose a spec for it, in order to get IANA to reserve 
it?  We're trying to use a similar address, fe80::a9fe:a9fe 
(169.254.169.254 in hex - see [1] for more details), so it essentially 
looks the same.  We did think about using DNS to discover it, but for 
now just using a hard-coded link-local seems like a quicker way forward.

Thanks for any comments or advice!

-Brian Haley

[0] 
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
[1] https://review.opendev.org/#/c/315604/

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 17 Dec 2019 17:09:45 +0000

Brian,

> (169.254.169.254 in hex - see [1] for more details),

That is the same as the AERO address format, which has been documented for
many years:

https://datatracker.ietf.org/doc/draft-templin-intarea-6706bis/
https://www.rfc-editor.org/info/rfc7421

For embedded IPv4 addresses/prefixes, however, we would use a modified
IPv4-compatible encapsulation as follows:

  fe80::ffff:a9fe:a9fe

(In other words, bits 64-95 set to 0x0000ffff signify that bits 96-127 embed
an IPv4 address/prefix).

Please have a look at the AERO address format.

Thanks - Fred

> -----Original Message-----
> From: ipv6 [mailto:[email-masked]] On Behalf Of Brian Haley
> Sent: Tuesday, December 17, 2019 8:42 AM
> To: [email-masked]
> Subject: Metadata over IPv6
> 
> Hi ipv6-list,
> 
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 17 Dec 2019 17:32:41 +0000

> IPv4-compatible encapsulation as follows:

I'm sorry; I intended to say "modified IPv4-mapped encapsulation". "IPv4-compatible"
is deprecated by RFC 4291 and would contain all-zero's instead of 0x0000ffff in bits 64-95.
AERO uses IPv4-mapped for embedded IPv4 addresses and all-zero's for administratively
assigned AERO addresses.

Also, fe80::ffff:a9fe:a9fe can be written as fe80::ffff:169.254.169.254 - the two address
representations are equivalent.

Thanks - Fred

---

From: Brian Haley <[email-masked]>
Date: Tue, 17 Dec 2019 14:11:12 -0500

Hi Fred,

Thanks for the response, I do have some questions.

> AERO uses IPv4-mapped for embedded IPv4 addresses and all-zero's for administratively

So in my case I don't think we want to define an IPv4-mapped 
encapsulation, just a new on-link anycast address that something on the 
link will answer to, it doesn't have to be the router since there might 
not actually be one.

> representations are equivalent.

I wouldn't mind using the AERO format, but reading the draft you listed 
it doesn't look like fe80::ffff/96 is being reserved by IANA?  Just that 
"Relay, Server and Proxy AERO addresses are allocated from the range 
fe80::/96, and MUST be managed for uniqueness." - since we do control 
the MAC we could do this I suppose as we want this link-local on the 
proxy, but what we're doing isn't really akin to AERO (IMO).

Thanks,

-Brian

---

From: Michael Richardson <[email-masked]>
Date: Tue, 17 Dec 2019 14:23:50 -0500

Brian Haley <[email-masked]> wrote:
    > IPv6-only networking scenarios that we don't support metadata with, so our
    > community is looking to define an IPv6 address to use for this service.  My
    > question to the list is - do you see a problem with us just defining an IPv6
    > link-local address for this same service?  Or do you think we need to propose
    > a spec for it, in order to get IANA to reserve it?

Well, nobody went to IANA to reserve 169.254.169.254 did they, so why change? :-)

I do think that it should be a link-local address. I don't think you'll get
any push back on that.  How and if to reserve it is a question.
I don't see a good reason to use mDNS to discover it, but others might.

I think that Fred's suggestion to use fe80::ffff:a9fe:a9fe is a good one.
I didn't look at this document to see how they reserved it, but whatever he
did is probably a good idea.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Tue, 17 Dec 2019 19:38:19 +0000

Hi Brian,

> From: Brian Haley [mailto:[email-masked]]
> Sent: Tuesday, December 17, 2019 11:11 AM
> To: Templin (US), Fred L <[email-masked]>
> Cc: [email-masked]
> Subject: Re: Metadata over IPv6
> 
> Hi Fred,
> 

There is already a link-local Subnet Router Anycast address - it is fe80::.

> it doesn't look like fe80::ffff/96 is being reserved by IANA?

fe80::ffff/96 is a format that has significance on AERO links. AERO is an IPv6-over-foo
document that specifies the construction of link-local addresses used on the link. The
format may be useful for other applications and/or link types besides just AERO,
though, so it would probably make sense to adopt a consistent format. RFC7421
cites the AERO format.

> Just that
> fe80::/96, and MUST be managed for uniqueness."

You are referring to something different here that does not apply to the use case
of embedding an IPv4 address in an IPv6 address - so, this part of the AERO spec
is out of scope for what we are discussing here.

Thanks - Fred

> since we do control

---

From: Brian Haley <[email-masked]>
Date: Tue, 17 Dec 2019 14:55:21 -0500

> There is already a link-local Subnet Router Anycast address - it is fe80::.

Ack.  In our case there might not be a router, a network can be 
completely isolated with only a DHCP server attached, which can also be 
used as a proxy to metadata - it would return a route to the fe80::$META 
address via itself in that case.

> though, so it would probably make sense to adopt a consistent format. RFC7421

Ah, Ok.  So does everything on the link use the same address format?  We 
are dealing with pre-existing deployments where we can't change that.  I 
do like the fact that AERO basically has the format we're looking for, 
fe80::ffff:169.254.169.254, which make it obvious this is the metadata 
service IP.

-Brian

---

From: Mark Smith <[email-masked]>
Date: Wed, 18 Dec 2019 07:32:23 +1100

On Wed, 18 Dec 2019, 06:55 Brian Haley, <[email-masked]> wrote:

> >>> Also, fe80::ffff:a9fe:a9fe can be written as
> fe80::ffff:169.254.169.254 - the two address

I think it would be better to reserve a proper IPv6 anycast address for
this from within the upper 128 IID values, unless there is a specific
reason to tie it to the IPv4 address version of the service.

https://www.iana.org/assignments/ipv6-anycast-addresses/ipv6-anycast-addresses.xhtml

Coupling things to IPv4 ways, even just symbolically, increases momentum
against giving up IPv4.

It's better to do things the native "IPv6 way".

Amazon shouldn't be hijacking a IPv4 Link-Local addresses for special
purposes like that. If somebody else does the same thing, then there are
two unofficial meanings for the same address. The argument that Amazon has
already reserved that address doesn't fly when they actually officially
haven't. Amazon should have requested a properly reserved anycast address
from IANA.

People used 1/8 for that sort of thing. That caused problems when APNIC
started using it for its proper purpose - public Internet addressing.

Traffic in Network 1.0.0.0/8
http://www.potaroo.net/studies/1slash8/1slash8.html

Regards,
Mark.

---

From: Fred Baker <[email-masked]>
Date: Tue, 17 Dec 2019 12:36:27 -0800

Thanks for the question.

What comes quickly to mind is RFC 1546, which describes a local "Host Anycasting Service". Related, https://tools.ietf.org/html/rfc4291#section-2.6 and https://tools.ietf.org/html/rfc4291#section-2.7, which respectively talk about local anycast and scoped multicast (one of the scopes being "local").

The bit pattern of a link-local address might be useful. That said, I would want this to not be "one-of", known by a few that happen to use a proprietary service (AWS), but consistent with the IPv6 addressing architecture and openly documented. From that perspective, I should think I might want a short RFC (I can help with drafting, if you like) that identifies the bit pattern and talks about it a bit. 

When documented in that way, one could imagine IANA picking it up. They do that.

> On Dec 17, 2019, at 8:41 AM, Brian Haley <[email-masked]> wrote:
> 
> Hi ipv6-list,
> 
> I was an IPv6 contributor many moons ago, and remembered this list when looking into something on a new project, was hoping to get some IPv6 advice (below).
> 
> The current project I'm working on, Openstack Neutron, is an SDN networking project focused on delivering networking-as-a-service (NaaS) in virtual compute environments.
> 
> One thing that happens when a virtual machine boots is it typically asks for metadata, thing like ssh keys, and other configuration information required to make it functional.  It does this via requests to the URL http://169.254.169.254/latest/meta-data/... ([0] has more complete info).  This link-local IPv4 address was defined by AWS and is widely used across all types of clouds.
> 
> This works fine for dual-stack hosts, but more and more we're seeing IPv6-only networking scenarios that we don't support metadata with, so our community is looking to define an IPv6 address to use for this service.  My question to the list is - do you see a problem with us just defining an IPv6 link-local address for this same service?  Or do you think we need to propose a spec for it, in order to get IANA to reserve it?  We're trying to use a similar address, fe80::a9fe:a9fe (169.254.169.254 in hex - see [1] for more details), so it essentially looks the same.  We did think about using DNS to discover it, but for now just using a hard-coded link-local seems like a quicker way forward.
> 
> [0] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html

---

From: Brian Haley <[email-masked]>
Date: Tue, 17 Dec 2019 16:05:30 -0500

Hi Fred,

So it seems you and Mark are of the same mindset that I should request a 
new anycast allocation for this, which I'm fine with because it becomes 
official, even if it does take more time.  It should be a very short draft.

And just for my own sanity, this anycast address would apply to an 
address of any scope, correct?  For example, given 7c is the next one 
available:

fe80::7c
2001:db8::7c

I just want to make sure I cover the case where there isn't any 
configured IPv6 address besides the link-local.

Thanks,

-Brian

---

From: Bob Hinden <[email-masked]>
Date: Tue, 17 Dec 2019 13:35:38 -0800

> On Dec 17, 2019, at 12:32 PM, Mark Smith <[email-masked]> wrote:
> 

……….

> I think it would be better to reserve a proper IPv6 anycast address for this from within the upper 128 IID values, unless there is a specific reason to tie it to the IPv4 address version of the service.
> 
> Coupling things to IPv4 ways, even just symbolically, increases momentum against giving up IPv4.
> 
> It's better to do things the native "IPv6 way”.

I agree, if this is intended for an IPv6-only network, then it would be much better to avoid using an encoded IPv4 address.

Getting an IPv6 anycast assigned seems like a reasonable approach to me.

Bob

> Amazon shouldn't be hijacking a IPv4 Link-Local addresses for special purposes like that. If somebody else does the same thing, then there are two unofficial meanings for the same address. The argument that Amazon has already reserved that address doesn't fly when they actually officially haven't. Amazon should have requested a properly reserved anycast address from IANA.
> 
> People used 1/8 for that sort of thing. That caused problems when APNIC started using it for its proper purpose - public Internet addressing.
>

---

From: [email-masked]
Date: Tue, 17 Dec 2019 22:54:55 +0100

Hi Brian,

> So it seems you and Mark are of the same mindset that I should request a new anycast allocation for this, which I'm fine with because it becomes official, even if it does take more time.  It should be a very short draft.
> 
> And just for my own sanity, this anycast address would apply to an address of any scope, correct?  For example, given 7c is the next one available:
> 
> fe80::7c
> 2001:db8::7c
> 
> I just want to make sure I cover the case where there isn't any configured IPv6 address besides the link-local.

Is you problem solvable using "service discovery" (e.g. DNS-SD)?

Best regards,
Ole

---

From: Fred Baker <[email-masked]>
Date: Tue, 17 Dec 2019 14:23:57 -0800

> On Dec 17, 2019, at 1:54 PM, [email-masked] wrote:
> 

That would be https://tools.ietf.org/html/rfc6763
6763 DNS-Based Service Discovery. S. Cheshire, M. Krochmal. February
     2013. (Format: TXT, HTML) (Updated by RFC8553) (Status: PROPOSED
     STANDARD) (DOI: 10.17487/RFC6763)

Right? 

8553 changes '_' to '-'.

---

From: Kerry Lynn <[email-masked]>
Date: Tue, 17 Dec 2019 17:55:50 -0500

>      STANDARD) (DOI: 10.17487/RFC6763)

Specifically, the flavor of DNS-SD you'd want to use is Multicast DNS
(mDNS):
https://tools.ietf.org/html/rfc6762

Kerry

---

From: Brian Haley <[email-masked]>
Date: Tue, 17 Dec 2019 18:06:34 -0500

>> Is you problem solvable using "service discovery" (e.g. DNS-SD)?

It *might* be, but I can't guarantee DNS will always be present, so it 
would be better to have a well-known address.  Typically this part of 
initialization in a cloud image is done in cloud-init, which is right 
after DHCP completes, and the address it uses is hard-coded in the 
binary itself to make things simple.

That's not to say mDNS couldn't be used, but having to deploy something 
in order to do a single lookup for 'metadata.local' could add a lot of 
overhead since there could be thousands of private networks involved, 
each with it's own daemon.

-Brian

---

From: [email-masked]
Date: Wed, 18 Dec 2019 10:35:51 +0100

> It *might* be, but I can't guarantee DNS will always be present, so it would be better to have a well-known address.  Typically this part of initialization in a cloud image is done in cloud-init, which is right after DHCP completes, and the address it uses is hard-coded in the binary itself to make things simple.
> 
> That's not to say mDNS couldn't be used, but having to deploy something in order to do a single lookup for 'metadata.local' could add a lot of overhead since there could be thousands of private networks involved, each with it's own daemon.

We have been around the "service discovery" block a few times. Not one of the industry's proudest moments.
Not to rehash all the alternatives here.

If I understand you correctly, the problem you like to solve is "How can I find the Openstack(?) provisioning URL for this link?".

I'm little uneasy about reserving a well-known interface-id/address for the purpose. Isn't it a layer violation? I'm just a little concerned about collectivily taking the cost too. And scale as well, if lots of "applications" are required to be provisioned this way.

There are probably 10-15 ways of doing it.
If you do DHCP anyway, why couldn't the URL be passed as an option in DHCP?

Cheers,
Ole

---

From: Alexandre Petrescu <[email-masked]>
Date: Wed, 18 Dec 2019 17:55:38 +0100

> it?  We're trying to use a similar address, fe80::a9fe:a9fe 

As I look at it, I think a 96 prefix length is pertinent for that address.

Alex

---

From: Brian Haley <[email-masked]>
Date: Wed, 18 Dec 2019 18:08:52 -0500

> If I understand you correctly, the problem you like to solve is "How can I find the Openstack(?) provisioning URL for this link?".

I'd have to ask specifics of the cloud-init maintainers, but from what I 
know and can glean from source code it wants a url:

DEF_MD_URL = "http://169.254.169.254"

This can vary depending on the cloud, and each image for a specific 
cloud will have a "baked-in" url for this.

> If you do DHCP anyway, why couldn't the URL be passed as an option in DHCP?

Yes, there are many ways to do this, it all seems to rely on a 
well-known link-local address being reachable though.  I'm reluctant to 
suggest a DHCP option since it will add complexity since more parts will 
needed to be changed to support it.

-Brian

---

From: [email-masked]
Date: Thu, 19 Dec 2019 14:26:27 +0100

Brian,

> I'd have to ask specifics of the cloud-init maintainers, but from what I know and can glean from source code it wants a url:
> 
> This can vary depending on the cloud, and each image for a specific cloud will have a "baked-in" url for this.
> 
> Yes, there are many ways to do this, it all seems to rely on a well-known link-local address being reachable though.  I'm reluctant to suggest a DHCP option since it will add complexity since more parts will needed to be changed to support it.

At the top of this email you stated: "...which is right after DHCP completes".

Cheers,
Ole

---

From: Brian Haley <[email-masked]>
Date: Thu, 19 Dec 2019 09:21:41 -0500

> At the top of this email you stated: "...which is right after DHCP completes".

Hi Ole,

Yes, but since this is a different process running, how does it know 
that option was returned and how does it get the value?  The DHCP client 
hopefully put it somewhere I would hope.  I'd have to ask the cloud-init 
developers what they think, I was thinking a well-known IPv6 address 
would be received better.

Thanks!

-Brian

---

From: Alexandre Petrescu <[email-masked]>
Date: Thu, 19 Dec 2019 15:25:37 +0100

Le 17/12/2019 à 17:41, Brian Haley a écrit :
> required to make it functional.

Do the ssh keys depend on IP address or not? (username@address, or 
something else).

Is the 'other conf info req'd to make it functional' something closer to 
IP layer or closer to app layer?

These could help think whether we look more towards an IP kind of 
solution (reserve ll address) or more like an upper layer service kind 
of solution.

Alex

   It does this via requests to the URL