AWS Transit Gateway and VMware Cloud on AWS

-
Gilles Chekroun
Lead VMware Cloud on AWS Specialist
---

With the release of AWS transit Gateway, connecting VPCs to VMware Cloud on AWS became much more easy.
This blog post will go through the detailed setup for the TGW and Firewall rules on VMC.

NSX-T SDDC and the Transit Gateway

Since November 2018, all new SDDCs deployed will be NSX-T based.
I am assuming that the reader is familiar with that deployment and will have an SDDC ready.
The goal of the AWS Transit Gateway is to allow easy, scalable and performant connectivity between multiple VPCs.
Our lab setup will be something like the schema below.

Lab Setup

On VMware Cloud on AWS side we have two Logical segments (192.168.1.0/24 and .2.0/24). On segment 1, a small Linux machine (.9) and ubuntu machine (.17). On segment 2 just a small Linux machine (.2) so we can do ping tests.
On the side AWS side, I deployed two VPCs (VPC2 and VPC3) with CIDR of 172.100.0.0/16 and 172.101.0.0/16.
In each one I have a small EC2 instance (.159 on VPC2 and .174 on VPC3).
The AWS Transit Gateway connects the two VPC but also the VMC side over VPN.

Transit Gateway Deployment

Let's go to AWS console in Frankfurt and start to deploy the TGW:

Create Attachments

Verify Attachments
The fist one is over VPN and we will see that in a minute. The other two are VPC attachments.

Update Route Tables in VPCs with TGW 

Although the TGW is learning all routes, it doesn't update the route tables in the VPC attached and we need to do that manually. That gives some level of control over the connectivity as well.
Verify EC2 connectivity from VPC2 to VPC3 and back

Setup VPN Attachment

On the AWS console, create a VPN Attachment on the Transit Gateway
We can leave all "Tunnel Options" empty and download the VPN configuration for the VMC side.
Config file important parameters:
By default, AWS creates 2 tunnels. The Parameter for the first tunnel are in the picture above and similarly for the second tunnel.

Configure VPN on VMC side

Check BGP routes learned in the TGW routing table
Check connectivity from DSL02 to EC2 in VPC2 and back

 Check connectivity from DSL02 to EC2 in VPC3 and back
Ping from EC2 in VPC3 to vCenter

Traffic Engineering

The default Firewall rule for VPN in VMC is "Drop".
I removed that rule on the previous tests so all communication is open.
We can now put it back and selectively open specific flows of communications

Create Groups by IP addresses

Allow "segment1" to VPC2 only

Similarly, add multiple FW rules to allow specific traffic to specific locations
Remember that FW rule is one way - we need the reverse way too.

Conclusion

The AWS Transit Gateway is really a very good way for customers that have large number of VPC and want interconnection. The previous "Transit VPC" design was very difficult to manage and to scale. We have now a beautiful solution that combines VMware Cloud on AWS and AWS Transit Gateway in a flexible and scalable design.

Comments

  1. DedicatedHosting4uMay 4, 2019 at 12:50 PM

    thanks for sharing informative information.

    ReplyDelete
  2. Khalil MebarkiaMay 7, 2019 at 11:26 AM

    Dear Gilles,
    I would like to thank you for this tutorial.
    I'm trying to do the same steps as you explained here. I'm having issues while the configuration VPN on VMC side. I downloaded the config file from AWS, I did exactly the steps you mentioned. However, I always have the first tunnel's IPSEC IS UP and the status is down in AWS, and in VMC the status is up and BGP status is in Progress. The second tunnel is always down in AWS as well as in VMC with the following error:

    ([Routing] Subnet should not overlap with other logical router port of same logical router. Subnet [169.254.13.168/30] overlaps with logical router port(s) [LRPort/cd8ad34b-42da-4483-9d91-4556d35b5199].Found errors in the request. Please refer to the related errors for details.
    [Routing] Subnet should not overlap with other logical router port of same logical router. Subnet [169.254.13.168/30] overlaps with logical router port(s) )

    Could you please help?

    ReplyDelete
    Replies
    1. Gilles ChekrounMay 7, 2019 at 12:33 PM

      Make sure the VTI firewall rule is open. Default is drop.
      Make sure the ASN on VMC match the one defined in AWS. Email me for more details and Thanks for the comment

      Delete
  3. AnonymousJune 5, 2019 at 10:50 AM

    base on your diagram when you work with transit gateway Is the connection using ENI is unnecessary ?

    ReplyDelete
    Replies
    1. Gilles ChekrounJune 5, 2019 at 11:31 AM

      Today VMC connectivity to TGW is only with route based IPSec VPN. Although you can setup multiple tunnels and use ECMP to aggregate, you cannot really reach the 25 GBPs of the ENI.

      Delete
  4. AnonymousJune 5, 2019 at 5:38 PM

    thank you very much for the quick answer

    ReplyDelete
  5. followNovember 22, 2019 at 2:57 PM

    Thanks! AMAZING...

    ReplyDelete
  6. JohnnyFebruary 12, 2020 at 8:44 PM

    I thought you access your Native AWS services across the ENI's? Wouldn't you incur egress charges across the VPN tunnel to the TGW?

    ReplyDelete
    Replies
    1. Gilles ChekrounFebruary 12, 2020 at 8:55 PM

      Yes it’s correct. The idea here was to connect to TGW with VPN. Yes you will pay charges for TGW process and VPN as well.

      Delete
    2. JohnnyFebruary 12, 2020 at 10:16 PM

      Thanks for the response. I'm trying to design our VMC environment and I want to connect my SDDC to an existing VPC with native AWS services in it. There is already a TGW connected to this VPC with a VPN connection to our on-prem environment.

      Delete
    3. Gilles ChekrounFebruary 12, 2020 at 10:29 PM

      You can use this VPC as the connected VPC for your SDDC. You will be able to have VMs in your SDDC accessing AWS services in that VPC via the ENI. There is no transitive routing so if you need VMs to access your on-prem we will have to do a different design. Email me : gchekroun at VMware dot com Depending on your timeframe, we have new designs that will be helpful

      Delete
  7. zwergiSeptember 18, 2020 at 7:43 AM

    Hello Gilles! I just read this article and asked me if the VPN connection to the TGW is mandatory? We want to create our VMC and I want to use an existing VPC as connected VPC for my SDDC. On the other hand I have a TGW and it would be great if there is some possibility to use the VPC attachments versus the TGW? So the connection speed is not limited to 1.25Gbps (I know we can create an ECMP multi link). What do you think? :-) Thanks & Regards

    ReplyDelete
    Replies
    1. Gilles ChekrounSeptember 18, 2020 at 1:54 PM

      @zwergi - Today the only way to connect to a TGW is VPN. Very soon (weeks) we will bring a new Networking concept called SDDC groups with our own managed TGW. You will be then able to connect your SDDC as VPC attachment to the VMware TGW.
      VPN supports 4 tunnels and with ECMP that will give you about 5Gbps.
      Cheers

      Delete
  8. UnknownNovember 8, 2020 at 4:19 PM

    Hi Gilles, just gone through this article. Very well explained. Thanks!

    ReplyDelete

Post a Comment

Populars

Egress VPC and AWS Transit Gateway (Part1)

-
Image
Gilles Chekroun
 Lead VMware Cloud on AWS Solutions Architect --- Usually my blog posts are customer driven and recently I have been working on a design that would include an Egress VPC and AWS Transit Gateway. This customer is going to use both VMware Managed Transit Gateway and also AWS Transit Gateway. I will split this post in 3 parts: The Egress VPC - this article Adding a VMC SDDC to the Egress VPC here Adding VMware Managed Transit Gateway here Why do we need an Egress VPC? Numerous posts on AWS site  will describe how to build an Egress VPC and the subtleties of the various route tables of the TGW and the Egress VPC itself. The main goal is to have ONE Internet gateway only  that will allow workloads to go out to internet on the Egress VPC. One of the most important point is redundancy and multi-availability zones. Applications usually reside in private subnets, while NAT Gateways reside in a public subnet. NAT Gateways To focus the Internet access to a single point, we can cre
Read more

AWS Transitive routing with Transit Gateways in the same region

-
Image
Gilles Chekroun Lead VMware Cloud on AWS Specialist --- At the AWS Re:invent 2019 conference, the long waited TGW peering functionality was announced and available in a few AWS regions. This is an INTER-REGION peering only meaning that the Transit Gateways need to be in different regions. Hoping that AWS will soon release an INTRA-REGION capability I discussed with a few AWS Solutions Architects in Las Vegas, among them Tom Adamski , about using a VPC as a bridge between two TGWs in the same region. Tom assured me that it is possible opening for the first time the transitive routing capability in AWS networking. Well . . . I needed to test that and see by myself. Test bed For a simple test, I will use 2 TGWs in the same regions with 2 VPCs attached each and another VPC as "bridge" connected to both TGWs. Yes you can connect a VPC up to 5 TGWs . The whole idea is to use this "bridging VPC" and point the default route of the TGWs to it. To do that I c
Read more

Considerations on vTGW to TGW Peering Link

-
Image
   Gilles Chekroun
 Lead VMware Cloud on AWS Solutions Architect --- Recently I was talking with a Customer on designing their VMware Cloud on AWS environment and linking it to their existing AWS infrastructure. The idea was to have 2 SDDCs, Test and Prod, and link them with a VMware Managed TGW and peer it to their existing TGW. The customer requirements were: VMs in each SDDC should have internet access from the SDDC out VMs from Test should be able to talk to VMs in Prod VMs in any SDDC should be able to access their own vCenter but also the other one Some VMs any SDDC should have a Public IP and NAT rule to be accessible from outside VMware traffic should stay within VMware SDDCs Other traffic, like on-prem, should go via the Customer TGW Test Setup Step1 - Create SDDC Group Create an SDDC Group and attach the 2 SDDCs Prod and Test Make sure the proper FireWall rules are open in each SDDC Compute GW. Step 2 - Attach the Customer TGW In the SDDC Groups, under External TGW TAB, add t
Read more