Subject: solicit feedback for adding Interconnection among different Cloud DCs to draft-ietf-rtgwg-net2cloud-problem-statement-03

From: Linda Dunbar <[email-masked]>
Date: Tue, 17 Sep 2019 23:24:43 +0000

RTGwg,

During IETF105, we got comments that draft-ietf-rtgwg-net2cloud-problem-statement-03 should expand to cover Interconnection between Cloud DCs owned and operated by different Cloud Operators, in addition to the current focusing on interconnecting Enterprises <-> Cloud DC.

Here is what we would like to add to the draft. Want to get some feedback on the mailing list. Thank you.
Linda

4.  Multiple Clouds Interconnection
4.1. Multi-Cloud Interconnection
Enterprises today can instantiate their workloads or applications in Cloud DCs owned by different Cloud providers, e.g. AWS, Azure, GoogleCloud, Oracle, etc. Interconnecting those workloads involves three parties: The Enterprise, its network service providers, and the Cloud providers.
All Cloud Operators offer secure ways to connect enterprises' on-prem sites/DCs with their Cloud DCs. For example, Google Cloud has Cloud VPN, AWS has VPN/CloudHub, and Azure has VPN gateway.
Some Cloud Operators allow enterprises to connect via private networks. For example, AWS's DirectConnect allows enterprises to use 3rd party provided private Layer 2 path from enterprises' GW to AWS DirectConnect GW. Microsoft's ExpressRoute allows extension of a private network to any of the Microsoft cloud services, including Azure and Office365. ExpressRoute is configured using Layer 3 routing. Customers can opt for redundancy by provisioning dual links from their location to two Microsoft Enterprise edge routers (MSEEs) located within a third-party ExpressRoute peering location. The BGP routing protocol is then setup over WAN links to provide redundancy to the cloud. This redundancy is maintained from the peering data center into Microsoft's cloud network.
Google's Cloud Dedicated Interconnect offers similar network connectivity options as AWS and Microsoft. One distinct difference, however, is that Google's service allows customers access to the entire global cloud network by default. It does this by connecting your on-premises network with the Google Cloud using BGP and Google Cloud Routers to provide optimal paths to the different regions of the global cloud infrastructure.
All those connectivity options are between Cloud providers' DCs and the Enterprises, but not between cloud DCs.  For example, to connect applications in AWS Cloud to applications in Azure Cloud, there must be a third-party gateway (physical or virtual) to interconnect the AWS's Layer 2 DirectConnect path with Azure's Layer 3 ExpressRoute.
It is possible to establish IPsec tunnels between different Cloud DCs, for example, by leveraging open source VPN software such as strongSwan, you create an IPSec connection to the Azure gateway using a shared key. The strong swan instance within AWS not only can connect to Azure but can also be used to facilitate traffic to other nodes within the AWS VPC by configuring forwarding and using appropriate routing rules for the VPC. Most Cloud operators, such as AWS VPC or Azure VNET, use non-globally routable CIDR from private IPv4 address ranges as specified by RFC1918. To establish IPsec tunnel between two Cloud DCs, it is necessary to exchange Public routable addresses for applications in different Cloud DCs. [BGP-SDWAN] describes one method. Other methods are worth exploring.
In summary, here are some approaches, available now (which might change in the future), to interconnect workloads among different Cloud DCs:

  1.  Utilize Cloud DC provided inter/intra-cloud connectivity services (e.g., AWS Transit Gateway) to connect workloads instantiated in multiple VPCs. Such services are provided with the cloud gateway to connect to external networks (e.g., AWS DirectConnect Gateway).
  2.  Hairpin all traffic through the customer gateway, meaning all workloads are directly connected to the customer gateway, so that communications among workloads within one Cloud DC must traverse through the customer gateway.
  3.  Establish direct tunnels among different VPCs (AWS' Virtual Private Clouds) and VNET (Azure's Virtual Networks) via client's own virtual routers instantiated within Cloud DCs. DMVPN (Dynamic Multipoint Virtual Private Network) or DSVPN (Dynamic Smart VPN) techniques can be used to establish direct Multi-point-to-Point or multi-point-to multi-point tunnels among those client's own virtual routers.

Approach a) usually does not work if Cloud DCs are owned and managed by different Cloud providers.
Approach b) creates additional transmission delay plus incurring cost when exiting Cloud DCs.
For the Approach c), DMVPN or DSVPN use NHRP (Next Hop Resolution Protocol) [RFC2735] so that spoke nodes can register their IP addresses & WAN ports with the hub node. The IETF ION (Internetworking over NBMA (non-broadcast multiple access) WG standardized NHRP for connection-oriented NBMA network (such as ATM) network address resolution more than two decades ago.
There are many differences between virtual routers in Public Cloud DCs and the nodes in an NBMA network. NHRP cannot be used for registering virtual routers in Cloud DCs unless an extension of such protocols is developed for that purpose, e.g. taking NAT or dynamic addresses into consideration. Therefore, DMVPN and/or DSVPN cannot be used directly for connecting workloads in hybrid Cloud DCs.
Other protocols such as BGP can be used, as described in [BGP-SDWAN].

4.2. Desired Properties for Multi-Cloud Interconnection
Different Cloud Operators have different APIs to access their Cloud resources. It is difficult to move applications built by one Cloud operator's APIs to another. However, it is highly desirable to have a single and consistent way to manage the networks and respective security policies for interconnecting applications hosted in different Cloud DCs.
The desired property would be having a single network fabric to which different Cloud DCs and enterprise's multiple sites can be attached or detached, with a common interface for setting desired policies. SDWAN is positioned to become that network fabric enabling Cloud DCs to be dynamically attached or detached. But the reality is that different Cloud Operators have different access methods, and Cloud DCs might be geographically far apart. More Cloud connectivity problems are described in the subsequent sections.

The difficulty of connecting applications in different Clouds might be stemmed from the fact that they are direct competitors. Usually traffic flow out of Cloud DCs incur charges. Therefore, direct communications between applications in different Cloud DCs can be more expensive than intra Cloud communications.

---

From: "Andrew G. Malis" <[email-masked]>
Date: Sat, 21 Sep 2019 09:02:17 -0400

RTGwg,

As a co-author, may we interpret no responses as "quiet consensus" for the
update?

Thanks,
Andy

On Tue, Sep 17, 2019 at 7:24 PM Linda Dunbar <[email-masked]>
wrote:

> RTGwg,
>
>
>
> During IETF105, we got comments that
> draft-ietf-rtgwg-net2cloud-problem-statement-03 should expand to cover
> Interconnection between Cloud DCs owned and operated by different Cloud
> Operators, in addition to the current focusing on interconnecting
> Enterprises <-> Cloud DC.
>
>
>
> 4.  Multiple Clouds Interconnection  4.1. Multi-Cloud Interconnection
>
> Enterprises today can instantiate their workloads or applications in Cloud
> DCs owned by different Cloud providers, e.g. AWS, Azure, GoogleCloud,
> Oracle, etc. Interconnecting those workloads involves three parties: The
> Enterprise, its network service providers, and the Cloud providers.
>
> All Cloud Operators offer secure ways to connect enterprises’ on-prem
> sites/DCs with their Cloud DCs. For example, Google Cloud has Cloud VPN,
> AWS has VPN/CloudHub, and Azure has VPN gateway.
>
> Some Cloud Operators allow enterprises to connect via private networks.
> For example, AWS’s DirectConnect allows enterprises to use 3rd party
> provided private Layer 2 path from enterprises’ GW to AWS DirectConnect GW.
> Microsoft’s ExpressRoute allows extension of a private network to any of
> the Microsoft cloud services, including Azure and Office365. ExpressRoute
> is configured using Layer 3 routing. Customers can opt for redundancy by
> provisioning dual links from their location to two Microsoft Enterprise
> edge routers (MSEEs) located within a third-party ExpressRoute peering
> location. The BGP routing protocol is then setup over WAN links to provide
> redundancy to the cloud. This redundancy is maintained from the peering
> data center into Microsoft's cloud network.
>
> Google’s Cloud Dedicated Interconnect offers similar network connectivity
> options as AWS and Microsoft. One distinct difference, however, is that
> Google’s service allows customers access to the entire global cloud network
> by default. It does this by connecting your on-premises network with the
> Google Cloud using BGP and Google Cloud Routers to provide optimal paths to
> the different regions of the global cloud infrastructure.
>
> All those connectivity options are between Cloud providers’ DCs and the
> Enterprises, but not between cloud DCs.  For example, to connect
> applications in AWS Cloud to applications in Azure Cloud, there must be a
> third-party gateway (physical or virtual) to interconnect the AWS’s Layer 2
> DirectConnect path with Azure’s Layer 3 ExpressRoute.
>
> It is possible to establish IPsec tunnels between different Cloud DCs, for
> example, by leveraging open source VPN software such as strongSwan, you
> create an IPSec connection to the Azure gateway using a shared key. The
> strong swan instance within AWS not only can connect to Azure but can also
> be used to facilitate traffic to other nodes within the AWS VPC by
> configuring forwarding and using appropriate routing rules for the VPC.
> Most Cloud operators, such as AWS VPC or Azure VNET, use non-globally
> routable CIDR from private IPv4 address ranges as specified by RFC1918. To
> establish IPsec tunnel between two Cloud DCs, it is necessary to exchange
> Public routable addresses for applications in different Cloud DCs.
> [BGP-SDWAN] describes one method. Other methods are worth exploring.
>
> In summary, here are some approaches, available now (which might change in
> the future), to interconnect workloads among different Cloud DCs:
>
>    1. Utilize Cloud DC provided inter/intra-cloud connectivity services
>    (e.g., AWS Transit Gateway) to connect workloads instantiated in multiple
>    VPCs. Such services are provided with the cloud gateway to connect to
>    external networks (e.g., AWS DirectConnect Gateway).
>    2. Hairpin all traffic through the customer gateway, meaning all
>    workloads are directly connected to the customer gateway, so that
>    communications among workloads within one Cloud DC must traverse through
>    the customer gateway.
>    3. Establish direct tunnels among different VPCs (AWS’ Virtual Private
>    Clouds) and VNET (Azure’s Virtual Networks) via client’s own virtual
>    routers instantiated within Cloud DCs. DMVPN (Dynamic Multipoint Virtual
>    Private Network) or DSVPN (Dynamic Smart VPN) techniques can be used to
>    establish direct Multi-point-to-Point or multi-point-to multi-point tunnels
>    among those client’s own virtual routers.
>
>
>
> For the Approach c), DMVPN or DSVPN use NHRP (Next Hop Resolution
> Protocol) [RFC2735] so that spoke nodes can register their IP addresses &
> WAN ports with the hub node. The IETF ION (Internetworking over NBMA
> (non-broadcast multiple access) WG standardized NHRP for
> connection-oriented NBMA network (such as ATM) network address resolution
> more than two decades ago.
>
> There are many differences between virtual routers in Public Cloud DCs and
> the nodes in an NBMA network. NHRP cannot be used for registering virtual
> routers in Cloud DCs unless an extension of such protocols is developed for
> that purpose, e.g. taking NAT or dynamic addresses into consideration.
> Therefore, DMVPN and/or DSVPN cannot be used directly for connecting
> workloads in hybrid Cloud DCs.
>
> Different Cloud Operators have different APIs to access their Cloud
> resources. It is difficult to move applications built by one Cloud
> operator’s APIs to another. However, it is highly desirable to have a
> single and consistent way to manage the networks and respective security
> policies for interconnecting applications hosted in different Cloud DCs.
>
> The desired property would be having a single network fabric to which
> different Cloud DCs and enterprise’s multiple sites can be attached or
> detached, with a common interface for setting desired policies. SDWAN is
> positioned to become that network fabric enabling Cloud DCs to be
> dynamically attached or detached. But the reality is that different Cloud
> Operators have different access methods, and Cloud DCs might be
> geographically far apart. More Cloud connectivity problems are described in
> the subsequent sections.
>
>
>
> The difficulty of connecting applications in different Clouds might be
> stemmed from the fact that they are direct competitors. Usually traffic
> flow out of Cloud DCs incur charges. Therefore, direct communications
> between applications in different Cloud DCs can be more expensive than
> intra Cloud communications.
>
>
>

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 21 Sep 2019 18:12:24 +0200

Hi Andrew,

Having been directly involved in interconnecting global enterprise to
public clouds for over 2.5 years and having read your document I must ask -
what is the objective here ?

Say ... Direct Connect it works seamlessly both from direct attachment of
my routers to cloud edge or via private peering offered as a service via
most if not all carriers to your doors or DCs.

Take - SDWAN - the one I have been personally using for few years now - as
day one product focused on seamless interconnect to public cloud. It has
multi tenancy build as root of the controller design from the very start.
That offers flexible interconnect at scale with not only fixed sites but
also mobile users.

IPSec to vGW also works pretty solid.

So I am not even asking what is missing ... but first what are you trying
to really accomplish by this draft ?

Many thx,
Robert

On Sat, Sep 21, 2019 at 3:02 PM Andrew G. Malis <[email-masked]> wrote:

>> Enterprises today can instantiate their workloads or applications in
>> Cloud DCs owned by different Cloud providers, e.g. AWS, Azure, GoogleCloud,
>> It is possible to establish IPsec tunnels between different Cloud DCs,
>> for example, by leveraging open source VPN software such as strongSwan, you
>> In summary, here are some approaches, available now (which might change
>> in the future), to interconnect workloads among different Cloud DCs:
>>
>>    3. Establish direct tunnels among different VPCs (AWS’ Virtual
>>    Private Clouds) and VNET (Azure’s Virtual Networks) via client’s own
>>    virtual routers instantiated within Cloud DCs. DMVPN (Dynamic Multipoint
>>    Virtual Private Network) or DSVPN (Dynamic Smart VPN) techniques can be
>>    used to establish direct Multi-point-to-Point or multi-point-to multi-point
>>    tunnels among those client’s own virtual routers.
>>
>>
>>
>> There are many differences between virtual routers in Public Cloud DCs
>> and the nodes in an NBMA network. NHRP cannot be used for registering
>> virtual routers in Cloud DCs unless an extension of such protocols is
>> developed for that purpose, e.g. taking NAT or dynamic addresses into
>> consideration. Therefore, DMVPN and/or DSVPN cannot be used directly for
>> connecting workloads in hybrid Cloud DCs.
>>
> _______________________________________________
> rtgwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtgwg
>

---

From: Linda Dunbar <[email-masked]>
Date: Mon, 23 Sep 2019 15:32:54 +0000

Robert,

Very good question.

All Cloud connectivity options are between Cloud providers’ DCs and the Enterprises, but not between cloud DCs (unless you instantiate your own virtual routers in different Cloud DCs and administer the IPsec tunnels among them yourselves, which by itself is a task) .

AWS’s DirectConnect allows enterprises to use 3rd party provided private Layer 2 path from enterprises’ GW to AWS DirectConnect GW. Microsoft’s ExpressRoute allows extension of a private network to any of the Microsoft cloud services, including Azure and Office365. ExpressRoute is configured using Layer 3 routing.
Therefore, to connect applications in AWS Cloud to applications in Azure Cloud, there must be a third-party gateway (physical or virtual) to interconnect the AWS’s Layer 2 DirectConnect path with Azure’s Layer 3 ExpressRoute, or hairpinned to the customer’s own GW.

Google’s Cloud Dedicated Interconnect offers similar network connectivity options as AWS and Microsoft. One distinct difference, however, is that Google’s service allows customers access to the entire global cloud network by default. It does this by connecting your on-premises network with the Google Cloud using BGP and Google Cloud Routers to provide optimal paths to the different regions of the global cloud infrastructure.

The purpose of adding this section (as requested by RTGwg Chair and Alia) is to make this problem clear as the document is about Network to interconnect Cloud. Many Enterprises today can instantiate their workloads or applications in Cloud DCs owned by different Cloud providers. Interconnecting those workloads involves three parties: The Enterprise, its network service providers, and the Cloud providers.

Hope my explanation is clear.

Thank you.

Linda

From: Robert Raszuk <[email-masked]>
Sent: Saturday, September 21, 2019 11:12 AM
To: Andrew G. Malis <[email-masked]>
Cc: Linda Dunbar <[email-masked]>; [email-masked]; RTGWG <[email-masked]>
Subject: Re: solicit feedback for adding Interconnection among different Cloud DCs to draft-ietf-rtgwg-net2cloud-problem-statement-03

Hi Andrew,

Having been directly involved in interconnecting global enterprise to public clouds for over 2.5 years and having read your document I must ask - what is the objective here ?

Say ... Direct Connect it works seamlessly both from direct attachment of my routers to cloud edge or via private peering offered as a service via most if not all carriers to your doors or DCs.

Take - SDWAN - the one I have been personally using for few years now - as day one product focused on seamless interconnect to public cloud. It has multi tenancy build as root of the controller design from the very start. That offers flexible interconnect at scale with not only fixed sites but also mobile users.

So I am not even asking what is missing ... but first what are you trying to really accomplish by this draft ?

Many thx,
Robert

On Sat, Sep 21, 2019 at 3:02 PM Andrew G. Malis <[email-masked]<mailto:[email-masked]>> wrote:
RTGwg,

As a co-author, may we interpret no responses as "quiet consensus" for the update?

Thanks,
Andy

On Tue, Sep 17, 2019 at 7:24 PM Linda Dunbar <[email-masked]<mailto:[email-masked]>> wrote:
RTGwg,

4.  Multiple Clouds Interconnection
4.1. Multi-Cloud Interconnection
Enterprises today can instantiate their workloads or applications in Cloud DCs owned by different Cloud providers, e.g. AWS, Azure, GoogleCloud, Oracle, etc. Interconnecting those workloads involves three parties: The Enterprise, its network service providers, and the Cloud providers.
All Cloud Operators offer secure ways to connect enterprises’ on-prem sites/DCs with their Cloud DCs. For example, Google Cloud has Cloud VPN, AWS has VPN/CloudHub, and Azure has VPN gateway.
Some Cloud Operators allow enterprises to connect via private networks. For example, AWS’s DirectConnect allows enterprises to use 3rd party provided private Layer 2 path from enterprises’ GW to AWS DirectConnect GW. Microsoft’s ExpressRoute allows extension of a private network to any of the Microsoft cloud services, including Azure and Office365. ExpressRoute is configured using Layer 3 routing. Customers can opt for redundancy by provisioning dual links from their location to two Microsoft Enterprise edge routers (MSEEs) located within a third-party ExpressRoute peering location. The BGP routing protocol is then setup over WAN links to provide redundancy to the cloud. This redundancy is maintained from the peering data center into Microsoft's cloud network.
Google’s Cloud Dedicated Interconnect offers similar network connectivity options as AWS and Microsoft. One distinct difference, however, is that Google’s service allows customers access to the entire global cloud network by default. It does this by connecting your on-premises network with the Google Cloud using BGP and Google Cloud Routers to provide optimal paths to the different regions of the global cloud infrastructure.
All those connectivity options are between Cloud providers’ DCs and the Enterprises, but not between cloud DCs.  For example, to connect applications in AWS Cloud to applications in Azure Cloud, there must be a third-party gateway (physical or virtual) to interconnect the AWS’s Layer 2 DirectConnect path with Azure’s Layer 3 ExpressRoute.
It is possible to establish IPsec tunnels between different Cloud DCs, for example, by leveraging open source VPN software such as strongSwan, you create an IPSec connection to the Azure gateway using a shared key. The strong swan instance within AWS not only can connect to Azure but can also be used to facilitate traffic to other nodes within the AWS VPC by configuring forwarding and using appropriate routing rules for the VPC. Most Cloud operators, such as AWS VPC or Azure VNET, use non-globally routable CIDR from private IPv4 address ranges as specified by RFC1918. To establish IPsec tunnel between two Cloud DCs, it is necessary to exchange Public routable addresses for applications in different Cloud DCs. [BGP-SDWAN] describes one method. Other methods are worth exploring.
In summary, here are some approaches, available now (which might change in the future), to interconnect workloads among different Cloud DCs:

a.      Utilize Cloud DC provided inter/intra-cloud connectivity services (e.g., AWS Transit Gateway) to connect workloads instantiated in multiple VPCs. Such services are provided with the cloud gateway to connect to external networks (e.g., AWS DirectConnect Gateway).

b.      Hairpin all traffic through the customer gateway, meaning all workloads are directly connected to the customer gateway, so that communications among workloads within one Cloud DC must traverse through the customer gateway.

c.      Establish direct tunnels among different VPCs (AWS’ Virtual Private Clouds) and VNET (Azure’s Virtual Networks) via client’s own virtual routers instantiated within Cloud DCs. DMVPN (Dynamic Multipoint Virtual Private Network) or DSVPN (Dynamic Smart VPN) techniques can be used to establish direct Multi-point-to-Point or multi-point-to multi-point tunnels among those client’s own virtual routers.

4.2. Desired Properties for Multi-Cloud Interconnection
Different Cloud Operators have different APIs to access their Cloud resources. It is difficult to move applications built by one Cloud operator’s APIs to another. However, it is highly desirable to have a single and consistent way to manage the networks and respective security policies for interconnecting applications hosted in different Cloud DCs.
The desired property would be having a single network fabric to which different Cloud DCs and enterprise’s multiple sites can be attached or detached, with a common interface for setting desired policies. SDWAN is positioned to become that network fabric enabling Cloud DCs to be dynamically attached or detached. But the reality is that different Cloud Operators have different access methods, and Cloud DCs might be geographically far apart. More Cloud connectivity problems are described in the subsequent sections.

_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg<https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Frtgwg&data=02%7C01%7Clinda.dunbar%40futurewei.com%7C0a7a4fd99362411e339508d73eae83a1%7C0fee8ff2a3b240189c753a1d5591fedc%7C1%7C0%7C637046791575634028&sdata=YoOwPflDYuERizHdLWocgYU9BEsv7xOr9%2FBP0CD37tk%3D&reserved=0>