AWS Direct Connect (DX)

12 min readApr 5, 2023

Direct Connect is a cloud service provided by Amazon Web Services (AWS) that enables businesses to establish a dedicated network connection between their data centers and AWS cloud. With Direct Connect, organizations can establish a private, high-bandwidth, and low-latency connection to the AWS cloud, bypassing the public internet. This enables businesses to enjoy the benefits of AWS cloud, such as scalability, flexibility, and cost-effectiveness, while maintaining the security and performance of their on-premises infrastructure. In this blog post, we will explore the benefits of Direct Connect and how it can help businesses achieve greater agility and competitiveness.

Overview

AWS Direct Connect links your internal network to an AWS Direct Connect location over a standard Ethernet fiber-optic cable. One end of the cable is connected to your router, the other to an AWS Direct Connect router.

Components

Connections
Create a connection in an AWS Direct Connect location to establish a network connection from your on-premises to an AWS Region
Virtual interfaces
Create a virtual interface to enable access to AWS services. A public virtual interface enables access to public services, such as Amazon S3. A private virtual interface enables access to your VPC

Network Requirements

Customer network must use single-mode fiber
1. 1000BASE-LX transceiver for 1 gigabit Ethernet
2. 10GBASE-LR (1310 nm) transceiver for 10 gigabit
3. 100GBASE-LR4 for 100 gigabit Ethernet
802.1Q VLAN encapsulation must be supported
Auto-negotiation for a port must be disabled
Customer device must support Border Gateway Protocol (BGP) and BGP MD5 authentication
(Optional) Configure Bidirectional Forwarding Detection (BFD) on Customer network.

Connection Types

Dedicated Connection : A physical Ethernet connection associated with a single customer
Hosted Connection : A physical Ethernet connection that an AWS Direct Connect Partner provisions on behalf of a customer

Dedicated Connection

Port Speed available are 1 Gbps, 10 Gbps and 100 Gbps (Depends on DX location)
You cannot change the port speed after you create the connection request.
You can create multiple(max 50) Virtual Interfaces (VIFs) (public or private) per connection
Billing automatically starts when the port is active or 90 days after the LOA (Letter of Authorization) has been issued, whichever comes first

Hosted Connection

Port Speed available are 50 Mbps, 100 Mbps, 200 Mbps, 300 Mbps, 400 Mbps, 500 Mbps, 1 Gbps, 2 Gbps, 5 Gbps, and 10 Gbps (Depends on DX location)
You cannot change the port speed after you create the connection request
You can create only one VIF (public or private) per connection

Cross Connects

Cross Connects can be used when you already have equipment located in an AWS Direct Connect location

Virtual Interfaces (VIF)

It provides logical connectivity
There are 3 types of VIFs
1. Private VIF — A private virtual interface should be used to access an Amazon VPC using private IP addresses
2. Public VIF — A public virtual interface can access all AWS public services using public IP addresses
3. Transit VIF — A transit virtual interface should be used to access one or more Amazon VPC Transit Gateways associated with Direct Connect gateways. You can use transit virtual interfaces with any AWS Direct Connect dedicated or hosted connection of any speed

Public VIFs and Private VIFs Architecture

Transit VIFs Architecture

VIF Parameters

To create a VIFs, we need AWS DX connection or a LAG (Link aggregation groups)
VIF Type : Public or Private or Transit
VIF Name : Any name you prefer
VIF Owner : your AWS account or other AWS account (hosted VIF)
Gateway Type (applicable to only Private VIF only)
1. Virtual Private Gateway
2. Direct Connect Gateway
VLAN
1. Not duplicated on same DX connection (1–4094)
2. For Hosted connection VLAN ID is already configured by the partner
BGP address Family — IPv4 or IPv6
BGP Peer IP Address
1. For Public VIF — Public IPs(/30) allocated to you for the BGP session. Request to AWS if you don’t have it
2. For Private VIF- Specify private IPs in the range 169.254.0.0/16. By default, AWS provides address space with /29 which will be allocated from 169.254.0.0/16
3. In case of IPv6, Amazon automatically allocates you a /125 IPv6 CIDR
BGP ASN
1. For VIF creation, Public or Private BGP ASN
2. Public ASN must be owned by the customer and assigned by IANA
3. Private ASN can be set by you and must be between 64512 to 65534
BGP MD5 authentication key. If not AWS automatically generates it for you
For Public VIFs, you can advertise prefixes to Amazon over a BGP
Jumbo Frames can be enabled only for Private and Transit VIF only
1. Private VIF : 9001 MTU (Default is 1500 MTU)
2. Transit VIF : 8500 MTU

Public VIF

Enables your network to connect to all AWS Public IPs globally(All AWS regions)
you can access services outside VPC e.g, S3, SQS, DynamoDB and other public endpoints like AWS managed VPN Public IPs
To create a public VIF with IPv4 addresses, you need to provide both the AWS router public IPs and your side of the router public IPs with /30 CIDR
In case you don’t have your own Public IPs for peering, raise a support case to get it from AWS owned IP ranges (AWS provides /31 range)
You must also specify the IPv4 address prefixes you want to advertise
AWS verifies with Internet registries that these IP prefixes are owned by you and you are authorized to advertise those prefixes
AWS advertises all Amazon prefixes over a BGP session. These include prefixes for AWS services like EC2, S3 and Amazon.com
From customer router to AWS maximum 1000 route prefixes can be advertised per Border Gateway Protocol (BGP) session

Private VIF

Enables your network to connect to resources inside VPC using Private IPs for resources like EC2, RDS, Redshift over Private IPs
You must attach your VPC to VGW and associate VGW with Private VIF
Private VIF and VGW must be in the same AWS Region
On BGP session, customer router will receive all the prefixes of VPC
You can only advertise only 100 prefixes to AWS
All the routes can be automatically be propagated into subnet route table
The propagated routes will take precedence over default route to internet via IGW
Support MTU of 1500 (default) and 9001 for propagates routes
VPC Gateway endpoints cannot be accessed through Private VIF
VPC DNS resolver cannot be accessed through Private VIF

Direct Connect Gateway

A direct connect gateway can be connected to multiple VPCs in the same or different AWS regions and in the same or different AWS accounts using Virtual Private Gateway (CGW)

Direct Connect Gateway is associated with private VIF
Direct Connect Gateway is used for VPC and on-premises connectivity and cannot be used for public endpoint
Private VIF and Direct Connect Gateway must be owned by same AWS account
No transitive routing for VPC to VPC communication using DX Gateway. For such communication, we need to use Transit Gateway
One DX Gateway can connect to 10 VGWs(VPCs)

Direct Connect Gateway with Transit Gateway (TGW)

Transit VIF is used for connecting DX Gateway to Transit Gateway
You can only use 1 Transit VIF per DX connection
You can connect only have 3 TGWs (Max) attached to a single DX Gateway
You can only advertise 20 Routes per TGW
To have a transitive routing between VPCs when using Direct Connect Gateway. You must have a peering between Transit Gateways to achieve it
You cannot have a transitive routing between customer sites because DX Gateway does not support it. Instead, you can use Transit Gateway peering to achieve transitive routing between two different on-premises data center having individual Direct Connect for each site.
Transit VIF cannot be created on a hosted connection with a capacity less than 1 Gbps

Direct Connect with SiteLink

If you want two on-premises datacenter located in different location to connect each other then you can use Direct connect with SiteLink to access on-premises locally using low-latency and consistent performance

You can only enable SiteLink on a Private VIF or Transit VIF
SiteLink supports on any combination of a dedicated or a hosted DX connection with different port speeds
SiteLink supports Full mesh topology connectivity with Private and Transit VIF

SiteLink support Partial mesh topology connectivity with Private and Transit VIF

Cost is calculated per hour and data transfer cost

DX Routing Policies and BGP communities

Direct Connect traffic flow decisions are influenced by Routing Policies
Inbound routing Policies — from on-premises to AWS
Outbound routing Policies — from AWS to on-premises
BGP community tags are used to advertise routes made by Amazon and you
Routing policies and BGP communities are used differently for Public and Private VIF

Public VIF — Routing policies

Inbound

You must specify the public IPv4 prefixes or IPv6 prefixes to advertise over BGP
You must own the public prefixes and they must be registered as such in the appropriate regional internet registry.
Traffic must be destined to Amazon public prefixes. Transitive routing between connections is not supported.
AWS Direct Connect performs inbound packet filtering to validate that the source of the traffic originated from your advertised prefix.

Outbound

AWS routing path decision is made by AS_PATH and Longest Prefix Match
AWS Direct Connect advertises all local and remote AWS Region prefixes
AWS Direct Connect advertises all public prefixes with the well-known NO_EXPORT BGP community
The prefixes advertised by AWS Direct Connect must not be advertised beyond the network boundaries of your connection
If you have multiple AWS Direct Connect connections, you can adjust the load-sharing of inbound traffic by advertising prefixes with similar path attributes

Public VIF — Routing scenarios

Active/Active Connection

Traffic is load-shared between interfaces based on flow. If one connection becomes unavailable, then all traffic is routed through the other connection
If you’re using a public ASN:
1. Allow your customer gateway to advertise the same prefix (public IP or network that you own) with the same Border Gateway Protocol (BGP) attributes on both public virtual interfaces.
2. Configuration permits you to load balance traffic over both public virtual interfaces
If you’re using a private ASN:
1. Load balancing on a public virtual interface isn’t supported

Active/Passive Connection

One connection handles traffic, and the other is on standby. If the active connection becomes unavailable, then all traffic is routed through the passive connection
If you’re using a public ASN:
1. Confirm that your customer gateway is advertising the same prefix (public IP or network that you own) on both BGP sessions
2. Start advertising the on-premises public prefixes with additional AS_Path prepends in the BGP attributes (To influence traffic from AWS to on-premises)
3. Increase the Local Preference (local-pref) to be sure that the on-premises router always chooses the correct path for sending traffic to AWS (To influence traffic from on-premises to AWS)
If you’re using a private ASN:
1. Customer gateway is advertising the longer prefix on your primary connection. The longer prefix is advertised on your primary connection, then traffic is sent to your customer gateway through the primary connection.

Public VIF — BGP Communities

To help control the scope (Regional or global) and route preference of traffic
If you do not apply any community tags, prefixes are advertised to all public AWS Regions (global) by default.
BGP community tag for inbound routing policies
1. 7224:9100—Local AWS Region
2. 7224:9200—All AWS Regions for a continent
a. North America–wide
b. Asia Pacific
c. Europe, the Middle East and Africa
3.7224:9300—Global (all public AWS Regions)
BGP community tag for outbound routing policies
1. 7224:8100—Routes that originate from the same AWS Region in which the AWS Direct Connect point of presence is associated
2. 7224:8200—Routes that originate from the same continent with which the AWS Direct Connect point of presence is associated
3. No tag — Global (all public AWS Regions)
Supported BGP community tags
1. 7224:9100, 7224:9200, 7224:9300 — Inbound
2. 7224:8100, 7224:8200 — Outbound
Supports NO_EXPORT BGP community tag for outbound routing policies
For public virtual interfaces, all routes that AWS Direct Connect advertises to customers are tagged with the NO_EXPORTcommunity tag

Private VIF — Routing Policies

Supports local preference BGP community tags to help control the route preference of traffic on private virtual interfaces and transit virtual interfaces
The Direct Connect home Region location determines the default routing for private and transit virtual interfaces, using the distance from the local Region to the Direct Connect location. If you do not specify a local preference using BGP community tags, the default outbound routing behaviour is based on the Direct Connect location’s relative distance to the originating Region.
You can use local preference BGP community tags to achieve load balancing and route preference for incoming traffic to your network
The following local preference BGP community tags are supported:
7224:7100—Low preference
7224:7200—Medium preference
7224:7300—High preference
Local preference BGP community tags are evaluated before any AS_PATH attribute, and are evaluated in order from lowest to highest preference (where highest preference is preferred)
If you do not specify local preference using BGP community tags, the local preference associated with the 7224:7200—Medium preference community is applied by default
Steps for decision on routing policies
1. Longest prefix
2. If prefixes are same then we would use AS_Path
3. If you cannot control prefix, BGP communities tags in combination with Local preferences for routing decision. Higher preference the traffic is routed to that VIF