AWS Transit Gateway and VMware Cloud on AWS

Gilles Chekroun
Lead VMware Cloud on AWS Specialist
---

With the release of AWS transit Gateway, connecting VPCs to VMware Cloud on AWS became much more easy.
This blog post will go through the detailed setup for the TGW and Firewall rules on VMC.

NSX-T SDDC and the Transit Gateway

Since November 2018, all new SDDCs deployed will be NSX-T based.
I am assuming that the reader is familiar with that deployment and will have an SDDC ready.
The goal of the AWS Transit Gateway is to allow easy, scalable and performant connectivity between multiple VPCs.
Our lab setup will be something like the schema below.

Lab Setup

On VMware Cloud on AWS side we have two Logical segments (192.168.1.0/24 and .2.0/24). On segment 1, a small Linux machine (.9) and ubuntu machine (.17). On segment 2 just a small Linux machine (.2) so we can do ping tests.
On the side AWS side, I deployed two VPCs (VPC2 and VPC3) with CIDR of 172.100.0.0/16 and 172.101.0.0/16.
In each one I have a small EC2 instance (.159 on VPC2 and .174 on VPC3).
The AWS Transit Gateway connects the two VPC but also the VMC side over VPN.

Transit Gateway Deployment

Let's go to AWS console in Frankfurt and start to deploy the TGW:

Create Attachments

Verify Attachments
The fist one is over VPN and we will see that in a minute. The other two are VPC attachments.

Update Route Tables in VPCs with TGW 

Although the TGW is learning all routes, it doesn't update the route tables in the VPC attached and we need to do that manually. That gives some level of control over the connectivity as well.
Verify EC2 connectivity from VPC2 to VPC3 and back


Setup VPN Attachment

On the AWS console, create a VPN Attachment on the Transit Gateway
We can leave all "Tunnel Options" empty and download the VPN configuration for the VMC side.
Config file important parameters:
By default, AWS creates 2 tunnels. The Parameter for the first tunnel are in the picture above and similarly for the second tunnel.

Configure VPN on VMC side

Check BGP routes learned in the TGW routing table
Check connectivity from DSL02 to EC2 in VPC2 and back

Check connectivity from DSL02 to EC2 in VPC3 and back
Ping from EC2 in VPC3 to vCenter

Traffic Engineering

The default Firewall rule for VPN in VMC is "Drop".
I removed that rule on the previous tests so all communication is open.
We can now put it back and selectively open specific flows of communications

Create Groups by IP addresses

Allow "segment1" to VPC2 only

Similarly, add multiple FW rules to allow specific traffic to specific locations
Remember that FW rule is one way - we need the reverse way too.

Conclusion

The AWS Transit Gateway is really a very good way for customers that have large number of VPC and want interconnection. The previous "Transit VPC" design was very difficult to manage and to scale. We have now a beautiful solution that combines VMware Cloud on AWS and AWS Transit Gateway in a flexible and scalable design.

Comments

  1. thanks for sharing informative information.

    ReplyDelete
  2. Dear Gilles,
    I would like to thank you for this tutorial.
    I'm trying to do the same steps as you explained here. I'm having issues while the configuration VPN on VMC side. I downloaded the config file from AWS, I did exactly the steps you mentioned. However, I always have the first tunnel's IPSEC IS UP and the status is down in AWS, and in VMC the status is up and BGP status is in Progress. The second tunnel is always down in AWS as well as in VMC with the following error:

    ([Routing] Subnet should not overlap with other logical router port of same logical router. Subnet [169.254.13.168/30] overlaps with logical router port(s) [LRPort/cd8ad34b-42da-4483-9d91-4556d35b5199].Found errors in the request. Please refer to the related errors for details.
    [Routing] Subnet should not overlap with other logical router port of same logical router. Subnet [169.254.13.168/30] overlaps with logical router port(s) )

    Could you please help?

    ReplyDelete
    Replies
    1. Make sure the VTI firewall rule is open. Default is drop.
      Make sure the ASN on VMC match the one defined in AWS. Email me for more details and Thanks for the comment

      Delete
  3. base on your diagram when you work with transit gateway Is the connection using ENI is unnecessary ?

    ReplyDelete
    Replies
    1. Today VMC connectivity to TGW is only with route based IPSec VPN. Although you can setup multiple tunnels and use ECMP to aggregate, you cannot really reach the 25 GBPs of the ENI.

      Delete
  4. thank you very much for the quick answer

    ReplyDelete
  5. I thought you access your Native AWS services across the ENI's? Wouldn't you incur egress charges across the VPN tunnel to the TGW?

    ReplyDelete
    Replies
    1. Yes it’s correct. The idea here was to connect to TGW with VPN. Yes you will pay charges for TGW process and VPN as well.

      Delete
    2. Thanks for the response. I'm trying to design our VMC environment and I want to connect my SDDC to an existing VPC with native AWS services in it. There is already a TGW connected to this VPC with a VPN connection to our on-prem environment.

      Delete
    3. You can use this VPC as the connected VPC for your SDDC. You will be able to have VMs in your SDDC accessing AWS services in that VPC via the ENI. There is no transitive routing so if you need VMs to access your on-prem we will have to do a different design. Email me : gchekroun at VMware dot com Depending on your timeframe, we have new designs that will be helpful

      Delete
  6. Hello Gilles! I just read this article and asked me if the VPN connection to the TGW is mandatory? We want to create our VMC and I want to use an existing VPC as connected VPC for my SDDC. On the other hand I have a TGW and it would be great if there is some possibility to use the VPC attachments versus the TGW? So the connection speed is not limited to 1.25Gbps (I know we can create an ECMP multi link). What do you think? :-) Thanks & Regards

    ReplyDelete
    Replies
    1. @zwergi - Today the only way to connect to a TGW is VPN. Very soon (weeks) we will bring a new Networking concept called SDDC groups with our own managed TGW. You will be then able to connect your SDDC as VPC attachment to the VMware TGW.
      VPN supports 4 tunnels and with ECMP that will give you about 5Gbps.
      Cheers

      Delete
  7. Hi Gilles, just gone through this article. Very well explained. Thanks!

    ReplyDelete

Post a Comment

Populars

AWS Transit Gateway and Multiple Accounts

Using Terraform with multiple providers in multiple phases to deploy and configure VMware Cloud on AWS