Connect VMware managed TGW to your AWS TGW in the same region using a "peering VPC"

-

Gilles Chekroun


Lead VMware Cloud on AWS Solutions Architect
---
In many designs we are facing customers that already have a TGW in a specific AWS region and VPCs attached to it.
Adding an SDDC group in the same region is problematic since AWS doesn't support TGW peering in the same region.

If the SDDC Group is in a different region, the VMC software (M15 for EA and M16 for GA) will support that but it's a very rare case and so far my Customers have TGW in the same region.

On my "physical" last Re:Invent conference in Vegas in 2019, I talked to an AWS Network engineer that indicated that we can do transitive routing via a VPC attached to two TGWs in the same region.
Yes, a VPC can be attached up to 5 different TGWs in the same region.

The setup is quite easy and simple. The throughput via this "peering VPC" is great since all attachments are VPC attachments at 50Gbps.

Nothing is required in the Peering VPC only 1 subnet in each AZ you want to connect - see below. Until AWS will provide a TGW peering INTRA region and until we will be able to add this functionality in our SDDC Group software, this is a very valid alternative.

VMC 1.12 release update

VMware decided to add a new feature in the 1.12 release that will allow the Customer to program static route in the VMware managed TGW. This is a mandatory feature for adding Security VPC, Transit VPC or Peering VPC.
A Feature Flag needs to be added called nsxGroupL3ConnectivitySecurityVpc

Lab Setup

SDDC Grouping

The SDDC is running 1.14.0.8. It's attached to an SDDC Group called "peeringVPC".

The SDDC Is attached to the Group and the AWS account is coded.

When this is done, the VMC console is sharing the VMware Managed TGW with the Customer console under RAM. The Customer needs to accept the share.

The Shared TGW will appear at the Customer Console.

AWS Setup

VPCs

We will create 3 VPCs:
  • VPC110 with CIDR 172.110.0.0/16
  • VPC120 with CIDR 172.120.0.0/16
  • Peering VPC with CIDR 172.0.0.0/16

Subnets

For each VPC we create 2 subnets in 2 AZs. The subnets are /24 with range .10 and .20 as below.

EC2s

On the VPC110 and 120 we will create EC2s in 2 AZs so we can ping them.

Customer TGW

The customer TGW is in the same Region (Oregon)


We keep Route table association and Propagation Disabled but in fact, Association could be left enable. 
Propagation should not be enable on the Peering VPC attachment.

Customer TGW Attachments

Every customer VPC is attached to the Customer TGW.
The Peering VPC is attached to BOTH the Customer TGW and the VMware TGW. See the AWS_side and VMC_Side below:


Static route on Customer VPC

We have added a 0.0.0.0/0 pointing to the Peering VPC. This will send all VPC traffic back to the VMC side.
The other 2 routes are propagated from the VPCs attachements associations.


Customer VPCs Route Table

In this test a simple "send everything to the TGW" will translate to a 0.0.0.0/0 pointing to the Customer TGW.

Peering VPC route Table

Here we need to try and summarise the routes.

I am using 172.0.0.0/8 as a "global" representation for ALL the customer side and for the return path a 0.0.0.0/0 to the VMware TGW.

VMC Setup

VMware TGW static route

Accept the Peering VPC attachment to the VMware TGW and as a  final step send all 172.0.0.0/8 routes to the VMware TGW.

To do that, we need the SDDC release 1.12 or up that allows us to program static routes and add this to the "allowed Prefixes" field.

VMware Side route table

SDDC FW rules

Open the proper CGW FW rules to allow traffic from and to the vTGW

Connectivity tests

VMC to Customer VPCs

From the VM in VMC with IP 12.12.12.10 let's ping the EC2 in VPC110 at 172.110.10.100

and also on the other VPC120 and other AZ at 172.120.20.100

Customer VPCs to VMC

For that we need to add an IGW let say on VPC110 and restrict my home IP to reach the EC2 public IP like:

Login to EC2 on VPC110 and ping the VM back to VMC at 12.12.12.10

Performance and throughput

The VPC attachments are all 50Gbps to all TGW.

The VMC Hosts have a 25Gbps interface. We will install an UBUNTU VM in VMC and a large EC2 (M5.24xlarge) on VPC110

VMC side iperf3 Server

Set MTU at 8500 and start iPref3 server.
Packets with a size larger than 8500 bytes that arrive at the transit gateway are dropped.

AWS side iperf3 Client

Check MTU (default is 9001) and set it to 8500 and use the command 

 iperf3 -c 12.12.12.100 -P 30 -w 416K -V


Without any tuning: 12.1 Gbps sending and 12.1 Gbps receiving

AWS Side deployment with Terraform

You can find all the Terraform code for the AWS side here. We still don't have Terraform provider for SDDC Grouping so some manual setup needs to be done on the VMC side.

Thanks for reading.

Comments

  1. UnknownSeptember 9, 2021 at 2:40 PM

    Thanks for this article. It's hugely disappointing that I can't do BGP between the VTGW and the TG.

    Forcing customers to create static routes isn't very manageable. My customer has simlar IP ranges on both sides preventing easy super-netting and has internet based routes on both sides so I'm left struggling to find a manageable solution.

    ReplyDelete
    Replies
    1. GillesSeptember 9, 2021 at 3:36 PM

      Unfortunately, AWS doesn't support BGP between TGWs even on native peering between regions. That's a limitation I agree, but this is what we have and static routes can also help you to do traffic engineering in a controlled way.

      Delete
    2. GillesSeptember 9, 2021 at 3:39 PM

      In your case, BGP will not help is you have overlapping IPs on each side anyway

      Delete
  2. UnknownNovember 18, 2021 at 8:59 PM

    Nice post Giles. Is this a supported design? It looks like transitive peering.

    ReplyDelete
    Replies
    1. Gilles ChekrounNovember 18, 2021 at 10:50 PM

      Yes it’s supported until AWS will propose an intra region peering. Maybe soon. Let see

      Delete

Post a Comment

Populars

Egress VPC and AWS Transit Gateway (Part1)

-
Image
Gilles Chekroun
 Lead VMware Cloud on AWS Solutions Architect --- Usually my blog posts are customer driven and recently I have been working on a design that would include an Egress VPC and AWS Transit Gateway. This customer is going to use both VMware Managed Transit Gateway and also AWS Transit Gateway. I will split this post in 3 parts: The Egress VPC - this article Adding a VMC SDDC to the Egress VPC here Adding VMware Managed Transit Gateway here Why do we need an Egress VPC? Numerous posts on AWS site  will describe how to build an Egress VPC and the subtleties of the various route tables of the TGW and the Egress VPC itself. The main goal is to have ONE Internet gateway only  that will allow workloads to go out to internet on the Egress VPC. One of the most important point is redundancy and multi-availability zones. Applications usually reside in private subnets, while NAT Gateways reside in a public subnet. NAT Gateways To focus the Internet access to a single point, we can cre
Read more

AWS Transitive routing with Transit Gateways in the same region

-
Image
Gilles Chekroun Lead VMware Cloud on AWS Specialist --- At the AWS Re:invent 2019 conference, the long waited TGW peering functionality was announced and available in a few AWS regions. This is an INTER-REGION peering only meaning that the Transit Gateways need to be in different regions. Hoping that AWS will soon release an INTRA-REGION capability I discussed with a few AWS Solutions Architects in Las Vegas, among them Tom Adamski , about using a VPC as a bridge between two TGWs in the same region. Tom assured me that it is possible opening for the first time the transitive routing capability in AWS networking. Well . . . I needed to test that and see by myself. Test bed For a simple test, I will use 2 TGWs in the same regions with 2 VPCs attached each and another VPC as "bridge" connected to both TGWs. Yes you can connect a VPC up to 5 TGWs . The whole idea is to use this "bridging VPC" and point the default route of the TGWs to it. To do that I c
Read more

Costs analysis for Data Transfer via VMware Managed TGW

-
Image
Gilles Chekroun
 Lead VMware Cloud on AWS Solutions Architect --- Every customer design is different but all of them should include a costs analysis specifically when using VMware managed Transit Gateway or AWS Transit Gateway. The AWS page here has very useful information for us to be able to understand costs. Transit Gateway Costs TGW pricing is split in 2 components: A fixed price for infrastructure connectivity per hour. Depending on the attachments, various account owners are charged.  A variable price for processing data via the TGW. Price is per GB. The sending account is charged. Prices depend on regions and range from $0.05 to $0.09 per attachment per hour. Who is charged? See AWS page VPC attachments The VPC account owner is charged and is billed hourly. VPN attachments The TGW account owner is billed hourly. Site-to-Site VPN connection pricing still applies in addition to the VPN TGW attachments. Direct Connect Gateway attachments The DXGW account owner is billed hourly.
Read more