Considerations on vTGW to TGW Peering Link

  Gilles Chekroun


Lead VMware Cloud on AWS Solutions Architect

---

Recently I was talking with a Customer on designing their VMware Cloud on AWS environment and linking it to their existing AWS infrastructure.

The idea was to have 2 SDDCs, Test and Prod, and link them with a VMware Managed TGW and peer it to their existing TGW.

The customer requirements were:

• VMs in each SDDC should have internet access from the SDDC out
• VMs from Test should be able to talk to VMs in Prod
• VMs in any SDDC should be able to access their own vCenter but also the other one
• Some VMs any SDDC should have a Public IP and NAT rule to be accessible from outside
• VMware traffic should stay within VMware SDDCs
• Other traffic, like on-prem, should go via the Customer TGW

Test Setup

Step1 - Create SDDC Group

Create an SDDC Group and attach the 2 SDDCs Prod and Test

Make sure the proper FireWall rules are open in each SDDC Compute GW.

Step 2 - Attach the Customer TGW

In the SDDC Groups, under External TGW TAB, add the details from the customer side:

• AWS Account
• TGW ID
• TGW Region
• SDDC Region
• Peer Link prefixes (up to 250)

Note the RFC 1918 prefixes on the Peering Link

Step 3 - Accept the peering attachment in AWS console

Accept the peering attachment and wait until "AVAILABLE" status is shown.

Verify the connectivity under SDDC Groups in VMC console

Note that it can take 10-15 mins until "CONNECTED" status

Step 4 - Check the routing tables

In the SDDC Groups, look for the Routing Tab.

Check also each SDDC Learned routes in the "Networking & Security" "Transit Connect" area.

Step 5 - Time for some Ping tests

VMs in SDDC_Prod and SDDC_Test can :

• ping each other
• ping the respective vCenters
• ping external IP like amazon.com
• use Public IP to SSH or HTTP/HTTPS

Prod Virtual Machine IP : 10.123.160.2

Test Virtual Machine IP  : 10.123.192.5

From Prod Virtual Machine:

Ping SDDC_Test VM and amazon.com

Ping SDDC_Test vCenter

Request a public IP and create a NAT rule

Verify access from outside

What about a 0.0.0.0/0 on the Peering Link?

Having RFC 1918 address on the peering link is great and covers about 2.3 Billion addresses.

Since we have all static routes entries in the routing tables, the most specific entries will take precedence.

If we code a 0.0.0.0/0 on the peering link, many things happen:

• VMs in any SDDC can still communicate
• VMs in any SDDC can still access the vCenters
• VMs in any SDDC can NOT ping external IPs like amazon.com
• Accessing VMs from outside with Public IPs will not be allowed

Basically anything that is NOT SDDC prefixes will be sent to the customer TGW.

Usually that is what customers wants and they will have the ability to control internet access on an Egress VPC and/or inspect traffic using a NGFW product.

vTWG route table

Conclusion

According to what the Customer wants to achieve it is very important to plan the Peering Link prefixes properly.

Today VMware Managed TGW has a soft limit of 250 prefixes.

All routes are static and even if the Customer TGW has a route table populated with all their prefixes, there is no protocol on the Peering Link that will allow these routes to be transferred to the vTGW.

It's been over 3 years that AWS is eluding about supporting BGP on the peering link and even strongly suggest to use different Autonomous System Numbers for each TGW, but until now, it's still static routes.

AWS. . . are you here???