EC2 Elastic Fabric Adapter (EFA)

A Complete Guide to High-Performance Networking for HPC and ML Workloads

What is an Elastic Fabric Adapter?

An Elastic Fabric Adapter (EFA) is a network device that you can attach to your Amazon EC2 instance to accelerate High Performance Computing (HPC) and machine learning applications. EFA provides lower and more consistent latency and higher throughput than the TCP transport traditionally used in cloud-based HPC systems.

Key Characteristics

Network device for high-performance computing
Provides OS-bypass capabilities
Supports message passing interface (MPI)
Lower and more consistent latency than TCP
Higher throughput for tightly-coupled applications
Works with standard AWS VPC networking

How EFA Works

OS-Bypass Technology

EFA uses a technique called OS-bypass that enables HPC and machine learning applications to bypass the operating system kernel and communicate directly with the EFA device. This reduces overhead and provides similar performance to on-premises HPC clusters.

Applications communicate directly with EFA device
Bypasses the operating system kernel
Reduces communication latency
Increases packets per second (PPS)
Provides consistent performance at scale

EFA vs. Standard ENI

Standard ENI (Elastic Network Interface)

Uses standard TCP/IP stack
All network traffic passes through OS kernel
Higher latency for inter-node communication
Limited packets per second (PPS)
Suitable for general-purpose workloads
No special software requirements

Use Cases:

Web servers
Databases
Application servers
General-purpose computing

EFA (Elastic Fabric Adapter)

Uses OS-bypass technology
Direct communication between application and device
Lower and more consistent latency
Higher throughput for tightly-coupled workloads
Designed for HPC and ML applications
Requires EFA-enabled AMI and software

Use Cases:

Computational fluid dynamics (CFD)
Weather forecasting
Molecular dynamics
Distributed machine learning
Finite element analysis

Performance Comparison

Metric	Standard ENI	EFA	Improvement
Latency	10-20 μs	3-5 μs	3-5x lower
Message Rate	~1M messages/sec	~5M messages/sec	5x higher
Bandwidth	Up to 100 Gbps	Up to 100 Gbps	Similar
CPU Overhead	High	Low	Significantly reduced
Scalability	Limited	Excellent	Better scaling to thousands of nodes

EFA Architecture Diagram

Prerequisites for Using EFA

Supported Instance Types

EFA is supported on specific high-performance instance types:

Compute Optimized: c5n.18xlarge, c5n.metal, c6gn.16xlarge
Memory Optimized: r5n.24xlarge, r5n.metal, r6i.32xlarge, x2iezn.metal
Accelerated Computing: p3dn.24xlarge, p4d.24xlarge, g4dn.metal, g5.48xlarge
High Performance Computing: hpc6a.48xlarge

Security Group Requirements

EFA requires specific security group rules:

# Inbound and outbound rules for all traffic between instances in the same security group
Type: All traffic
Protocol: All
Port Range: All
Source/Destination: The security group itself

This allows all EFA-enabled instances in the same security group to communicate with each other.

Software Requirements

EFA driver
Libfabric
Open MPI or another MPI implementation
EFA-enabled kernel
AWS-provided EFA AMI or custom AMI with EFA support

Setting Up EFA

Using AWS Management Console

Navigate to EC2 in the AWS Management Console
Launch a new instance
Select an EFA-supported instance type
Choose an EFA-enabled AMI (Amazon Linux 2, Ubuntu, etc.)
Configure network settings
Under "Network Interfaces", select "Add network interface"
Set "Network Card" to the appropriate value
Enable "Elastic Fabric Adapter"
Complete the instance launch process

Using AWS CLI

# Create a security group for EFA
aws ec2 create-security-group \
  --group-name efa-sg \
  --description "Security group for EFA" \
  --vpc-id vpc-12345678

# Add inbound and outbound rules
aws ec2 authorize-security-group-ingress \
  --group-id sg-12345678 \
  --source-group sg-12345678 \
  --protocol all

aws ec2 authorize-security-group-egress \
  --group-id sg-12345678 \
  --destination-group sg-12345678 \
  --protocol all

# Launch an instance with EFA
aws ec2 run-instances \
  --image-id ami-12345678 \
  --count 1 \
  --instance-type p4d.24xlarge \
  --key-name my-key-pair \
  --security-group-ids sg-12345678 \
  --subnet-id subnet-12345678 \
  --network-interfaces "DeviceIndex=0,InterfaceType=efa,AssociatePublicIpAddress=true,Groups=sg-12345678"

Installing EFA Software

Using AWS-Provided AMIs

AWS provides EFA-enabled AMIs with all required software pre-installed:

Amazon Linux 2 with EFA
Ubuntu with EFA
RHEL with EFA
AWS Deep Learning AMIs with EFA

These AMIs are the easiest way to get started with EFA.

Manual Installation

If you need to install EFA on a custom AMI:

# Download the EFA installer
curl -O https://efa-installer.amazonaws.com/aws-efa-installer-latest.tar.gz

# Extract the installer
tar -xf aws-efa-installer-latest.tar.gz

# Navigate to the extracted directory
cd aws-efa-installer

# Run the installer
sudo ./efa_installer.sh -y

# Verify installation
fi_info -p efa

Verifying EFA Functionality

After installation, verify that EFA is working correctly:

# Check if EFA device exists
ls -la /dev/efa*

# Verify EFA module is loaded
lsmod | grep efa

# Check Libfabric EFA provider
fi_info -p efa

# Run a simple MPI test
cd /opt/amazon/openmpi/bin
mpirun -n 2 --host localhost,localhost ./osu_latency

If everything is working correctly, you should see the EFA device, loaded kernel modules, and successful MPI communication.

Optimizing EFA Performance

Placement Groups

Use cluster placement groups to minimize latency between instances:

# Create a cluster placement group
aws ec2 create-placement-group \
  --group-name efa-cluster-pg \
  --strategy cluster

# Launch instances in the placement group
aws ec2 run-instances \
  --image-id ami-12345678 \
  --count 1 \
  --instance-type p4d.24xlarge \
  --placement "GroupName=efa-cluster-pg" \
  --network-interfaces "DeviceIndex=0,InterfaceType=efa,Groups=sg-12345678"

MPI Tuning

Optimize MPI parameters for EFA:

# Example mpirun command with optimizations
mpirun \
  --mca btl ^openib \
  --mca pml ^ucx \
  --mca opal_warn_on_missing_libcuda 0 \
  --mca mtl ofi \
  --mca mtl_ofi_provider_include efa \
  -x FI_EFA_USE_DEVICE_RDMA=1 \
  -x FI_PROVIDER=efa \
  -x RDMAV_FORK_SAFE=1 \
  -np 64 -npernode 8 \
  --hostfile hostfile \
  ./my_application

Instance Type Selection

Choose the right instance type for your workload:

p4d.24xlarge: Best for ML training with 8 NVIDIA A100 GPUs
hpc6a.48xlarge: Cost-optimized for CPU-based HPC workloads
c5n.18xlarge/c5n.metal: Good balance of compute and network performance
r5n.24xlarge/r5n.metal: Memory-intensive HPC applications

Advanced Configurations

Multi-Node Clusters

Configure a hostfile for multi-node MPI jobs:

# Example hostfile
ip-10-0-1-101 slots=8
ip-10-0-1-102 slots=8
ip-10-0-1-103 slots=8
ip-10-0-1-104 slots=8

# Run MPI job across multiple nodes
mpirun -n 32 -npernode 8 --hostfile hostfile ./my_application

For large clusters, consider using a cluster management system like AWS ParallelCluster.

NCCL Configuration for ML

Optimize NCCL for distributed machine learning with EFA:

# Set NCCL environment variables
export NCCL_DEBUG=INFO
export FI_PROVIDER=efa
export FI_EFA_USE_DEVICE_RDMA=1
export NCCL_PROTO=simple
export NCCL_ALGO=ring

# Example PyTorch distributed training
python -m torch.distributed.launch \
  --nproc_per_node=8 \
  --nnodes=4 \
  --node_rank=0 \
  --master_addr=ip-10-0-1-101 \
  --master_port=12345 \
  train.py

Hybrid MPI+OpenMP

Combine MPI for inter-node and OpenMP for intra-node parallelism:

# Set OpenMP threads
export OMP_NUM_THREADS=4

# Run hybrid MPI+OpenMP job
mpirun -n 16 -npernode 4 --hostfile hostfile ./hybrid_application

This approach can reduce MPI communication overhead and improve performance.

Monitoring EFA Performance

CloudWatch Metrics

Monitor EFA performance using CloudWatch:

NetworkIn/NetworkOut: Network traffic volume
NetworkPacketsIn/Out: Network packet count
EfaPacketSent/Received: EFA-specific packet metrics
CPUUtilization: CPU usage (should be lower with EFA)

Application-Level Benchmarks

Use standard HPC benchmarks to measure performance:

# OSU MPI Benchmarks
mpirun -n 2 --host node1,node2 osu_latency
mpirun -n 2 --host node1,node2 osu_bw
mpirun -n 2 --host node1,node2 osu_allreduce -m 8:1048576

# NCCL Tests for ML workloads
./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8

Troubleshooting

Common Issues

Missing EFA device: EFA not properly attached or driver not loaded
Security group issues: Incorrect security group configuration
Software compatibility: Incompatible MPI or Libfabric versions
Performance degradation: Instances not in placement group
Connection failures: Network ACLs or OS firewall blocking traffic

Debugging Steps

# Check EFA device and driver
ls -la /dev/efa*
lsmod | grep efa
dmesg | grep -i efa

# Verify Libfabric configuration
fi_info -p efa
FI_LOG_LEVEL=debug fi_info -p efa

# Test connectivity between nodes
mpirun -n 2 --host node1,node2 hostname

# Check security group rules
aws ec2 describe-security-groups --group-id sg-12345678

AWS Customer Examples

OpenAI (ChatGPT)

OpenAI uses EFA-enabled instances for training large language models like GPT. The distributed training process requires efficient communication between GPU nodes, and EFA's low-latency, high-throughput networking significantly reduces training time. By using EFA with NCCL, OpenAI can scale their training across hundreds of GPUs while maintaining high efficiency.

Toyota Research Institute

Toyota Research Institute uses EFA for computational fluid dynamics (CFD) simulations to optimize vehicle aerodynamics. Their simulations require intensive communication between compute nodes, and EFA's OS-bypass capability allows them to achieve performance similar to on-premises HPC clusters but with the flexibility of the cloud.

Rescale

Rescale, a cloud HPC platform provider, uses EFA to deliver high-performance computing capabilities to their customers. By leveraging EFA, they can offer near-bare-metal performance for tightly-coupled applications like crash simulations, structural analysis, and weather modeling, enabling their customers to run complex simulations in the cloud.

AstraZeneca

AstraZeneca uses EFA for molecular dynamics simulations in drug discovery. These simulations model the interactions between proteins and potential drug compounds, requiring intensive communication between compute nodes. EFA enables them to run these simulations faster, accelerating their drug discovery pipeline.

Common Use Cases

Computational Fluid Dynamics (CFD)

CFD simulations model fluid flow and are used in aerospace, automotive, and weather forecasting:

Domain decomposition splits the simulation across multiple nodes
Each time step requires boundary exchange between domains
EFA reduces communication overhead, enabling larger and more detailed simulations
Popular software: OpenFOAM, ANSYS Fluent, Star-CCM+

# Example OpenFOAM run with EFA
mpirun -n 256 --hostfile hostfile simpleFoam -parallel

Distributed Machine Learning

Training large ML models across multiple GPU nodes:

Data-parallel training splits batches across GPUs
All-reduce operations synchronize gradients
EFA accelerates these collective operations
Frameworks: PyTorch, TensorFlow with Horovod

# Example PyTorch distributed training
horovodrun -np 32 -H node1:8,node2:8,node3:8,node4:8 \
  python train.py --batch-size 64

Molecular Dynamics

Simulating the physical movements of atoms and molecules:

Requires calculating forces between all particles
Domain decomposition with frequent communication
EFA reduces communication latency, enabling longer simulations
Software: GROMACS, NAMD, LAMMPS

# Example GROMACS run with EFA
mpirun -n 128 --hostfile hostfile gmx_mpi mdrun -deffnm protein -ntomp 4

Case Study: Weather Forecasting

Weather forecasting models like the Weather Research and Forecasting (WRF) model are classic examples of applications that benefit from EFA. These models:

Divide the atmosphere into a 3D grid of cells
Calculate atmospheric physics for each cell
Require frequent communication between adjacent cells
Scale to hundreds or thousands of cores

A major weather forecasting center migrated their WRF workload from an on-premises HPC cluster to AWS using EFA-enabled instances. They found:

Performance within 10% of their on-premises cluster
Ability to scale up during severe weather events
30% cost reduction compared to maintaining on-premises hardware
Improved forecast resolution due to on-demand scaling

Implementation Details

Their AWS architecture included:

64 c5n.18xlarge instances with EFA
Instances placed in a cluster placement group
Amazon FSx for Lustre for high-performance storage
Custom AMI with optimized WRF and EFA software
AWS ParallelCluster for cluster management

Performance optimizations:

Domain decomposition optimized for EFA communication patterns
I/O optimizations to reduce storage bottlenecks
Custom MPI parameters for EFA
Hybrid MPI+OpenMP parallelization

Advantages of EFA

Performance Benefits

Lower latency (3-5x improvement over TCP)
Higher message rate (up to 5x more messages per second)
Reduced CPU overhead for networking
Better scaling to thousands of nodes
More consistent performance under load
Improved application efficiency and throughput

Operational Advantages

Works within standard VPC networking
Compatible with existing security groups
No additional cost beyond instance pricing
Supports popular MPI implementations
Pre-configured AMIs available
Enables cloud-based HPC without performance compromise

Business Benefits

Reduces time-to-solution for HPC workloads
Enables cloud migration of performance-sensitive applications
Eliminates need for specialized on-premises HPC hardware
Pay-as-you-go pricing for HPC infrastructure
Ability to scale up for urgent or large workloads
Access to latest hardware without capital investment

Limitations of EFA

Technical Constraints

Limited to specific instance types
Requires special software stack (drivers, libfabric)
Applications may need modification to fully benefit
Limited to a single Availability Zone
Not available in all AWS regions
Cannot be used with containers without special configuration

Operational Challenges

More complex setup than standard EC2 networking
Requires specific security group configuration
Troubleshooting can be more difficult
Limited monitoring tools for EFA-specific metrics
Instance type availability can be constrained
Requires specialized knowledge to optimize

Alternative Considerations

Standard ENI may be sufficient for loosely-coupled workloads
For some workloads, algorithm optimization may be more effective
Specialized cloud providers may offer alternatives
On-premises HPC may still be more cost-effective for steady-state workloads
GPU Direct RDMA not supported (as of 2023)
Some legacy HPC applications may not be compatible

EFA vs. Other HPC Networking Options

Feature	EFA	Standard EC2 Networking	On-Premises InfiniBand	AWS Enhanced Networking
OS-Bypass	Yes	No	Yes	No
Latency	3-5 μs	10-20 μs	1-2 μs	8-15 μs
Setup Complexity	Medium	Low	High	Low
Cost	Instance cost only	Instance cost only	High (hardware + maintenance)	Instance cost only
Scalability	Thousands of nodes	Limited	Thousands of nodes	Moderate
Elasticity	High	High	Low	High

Test Your Knowledge

1. What is the primary advantage of Elastic Fabric Adapter (EFA) over standard EC2 networking?

A) Higher bandwidth

B) Better security

C) OS-bypass capabilities for lower latency

D) Support for more IP addresses

2. Which of the following workloads would benefit most from using EFA?

A) Web server hosting static content

B) Single-node database

C) Content delivery network

D) Distributed computational fluid dynamics simulation

3. What software component is required to use EFA?

A) AWS SDK

B) Libfabric with EFA provider

C) AWS CloudFormation

D) Amazon RDS

4. Which placement strategy is recommended when using EFA for optimal performance?

A) Cluster placement group

B) Spread placement group

C) Partition placement group

D) No placement group is needed

5. Which of the following is a limitation of EFA?

A) It can only be used with Amazon Linux 2

B) It requires a dedicated VPC

C) It is limited to a single Availability Zone

D) It can only be used with p4d instances