Compute, Networking, and Data Ingestion
Module 4 · Design High-Performing Architectures

Compute, Networking, and Data Ingestion

Design high-performing compute, network, and data pipelines: EC2 instance families and placement groups, Lambda and containers, CloudFront vs Global Accelerator, hybrid connectivity, and ingestion and analytics with Kinesis, Glue, Athena, and Lake Formation.

Compute That Fits the Workload

EC2 instance families

Family (examples)Optimized forTypical workload
General purpose (M, T)Balanced CPU and memoryWeb servers, small databases. T instances are burstable
Compute optimized (C)High CPU per GB of memoryBatch processing, gaming servers, video encoding
Memory optimized (R, X)Large memoryIn-memory databases, caches, big data analytics
Storage optimized (I, D)Fast local storageNoSQL databases, data warehousing on local disks
Accelerated computing (P, G, Inf, Trn)GPUs and ML chipsML training and inference, graphics

Instance types ending in g (like m7g) use AWS Graviton processors, which usually give better price-performance for workloads that run on Arm.

Placement groups

StrategyWhat it doesUse it for
ClusterPacks instances close together in one AZLowest latency, highest throughput between nodes (HPC)
SpreadPuts each instance on distinct hardware (max 7 running instances per AZ per group)A small number of critical instances that must not fail together
PartitionGroups instances into partitions on separate racksLarge distributed systems like Hadoop, Cassandra, Kafka

Choosing a compute model

  • EC2: full control, any OS, long-running processes, licensed software
  • ECS or EKS on Fargate: containers without managing servers
  • Lambda: event-driven code, up to 15 minutes per invocation, scales to zero
  • AWS Batch: queues and runs large numbers of batch jobs on EC2 or Fargate

Network Performance

CloudFront vs Global Accelerator

Both use the AWS edge network, but they solve different problems:

Amazon CloudFrontAWS Global Accelerator
What it isContent delivery network that caches at the edgeNetwork accelerator with two static anycast IPs
ProtocolsHTTP/HTTPS (and WebSocket)TCP and UDP, any application
CachingYesNo
Best forWebsites, APIs, video, downloadsGaming, VoIP, IoT, non-HTTP apps; fast Regional failover; clients that need fixed IPs

Connecting VPCs and on-premises

NeedService
Connect two VPCs privatelyVPC peering (not transitive: A–B and B–C does not give A–C)
Connect many VPCs and on-premises networksTransit Gateway (hub and spoke, transitive routing)
Expose one service to other VPCs without full network accessAWS PrivateLink (interface endpoint + Network Load Balancer)
Quick, encrypted on-premises link over the internetSite-to-Site VPN
Consistent, high-bandwidth private link to AWSAWS Direct Connect (takes weeks to provision; not encrypted by default, so add MACsec or a VPN on top when required)
Highly available hybrid linkTwo Direct Connect connections at different locations, or Direct Connect with a VPN backup

Why large networks move from peering to Transit Gateway:

VPC peering: not transitiveVPC AVPC CVPC B✕no routeA–B and B–C peered, but A cannotreach C through BTransit Gateway: hub and spokeTransit GatewayVPC AVPC BVPC CVPC DOn-premisesVPN or DXEvery attachment can reach the otherspeeredpeered

Enhanced networking (the Elastic Network Adapter) gives EC2 higher bandwidth and lower latency, and Elastic Fabric Adapter (EFA) adds OS-bypass networking for tightly coupled HPC and ML jobs.


Data Ingestion and Transformation

Domain 3 includes moving data in, transforming it, and making it queryable. A common serverless pipeline:

Producersapps, IoT, logsKinesis DataStreamsAmazon DataFirehoseReal-timeLambda / FlinkS3 raw zoneJSON / CSVAWS Gluecatalog + ETLS3 curatedParquet, partitionedAthena (SQL)→ BI dashboardsAWS Lake Formationfine-grained access control for the data lake
ServiceRole in the pipelineExam cue
Kinesis Data StreamsReal-time, ordered, replayable stream; you manage consumers”real-time,” “multiple consumers,” “replay”
Amazon Data FirehoseFully managed delivery into S3, Redshift, OpenSearch, and others, with buffering and optional transformation”near real-time,” “load into S3 with no code”
Managed Service for Apache FlinkSQL or Java stream processing”analyze the stream in real time”
AWS GlueServerless ETL, crawlers, and the Data Catalog”transform,” “convert CSV to Parquet,” “catalog”
Amazon AthenaServerless SQL directly on S3; pay per data scanned”ad hoc queries on S3”
AWS Lake FormationBuilds and governs a data lake with table-, column-, and row-level permissions”fine-grained access to the data lake”
Amazon EMRManaged Spark, Hadoop, and more”existing Spark jobs”
AWS DataSyncOnline transfer from on-premises storage (NFS, SMB, HDFS) to S3, EFS, or FSx”migrate file shares”
AWS Storage GatewayHybrid access: on-premises apps use S3-backed file, volume, or tape storage”on-premises apps need cloud storage”

For dashboards, the exam guide lists Amazon Quick, AWS’s business intelligence suite, which now includes what used to be Amazon QuickSight.

⚠️ Common Pitfall: Querying Raw CSV With Athena

Common mistake: a pipeline dumps uncompressed CSV into one S3 prefix and analysts query it with Athena every day.

Athena charges by data scanned. Converting to columnar Parquet, compressing it, and partitioning by date often cuts both cost and query time dramatically, because each query reads only the columns and partitions it needs.


Key Takeaway

Match the instance family to the bottleneck, use placement groups for latency (cluster) or isolation (spread, partition), and pick Lambda, Fargate, or EC2 by how much control you need. Use CloudFront to cache HTTP content and Global Accelerator for static IPs and non-HTTP traffic. Know the ingestion chain: Kinesis or Firehose → S3 → Glue → Athena, governed by Lake Formation.

🧪 Try It: Query S3 With Athena

In a sandbox account:

  1. Upload a CSV file (any public data set, a few MB) to an S3 bucket.
  2. Run an AWS Glue crawler on the prefix to create a table in the Data Catalog.
  3. Query it in Athena and note the “data scanned” figure.
  4. Use an Athena CREATE TABLE AS SELECT (CTAS) query to write the same data as Parquet, partitioned by a date or category column.
  5. Run the same query against the Parquet table and compare data scanned.

Delete the buckets and tables when you’re done.

🧠 Knowledge Check

1. A multiplayer game uses UDP and needs static IP addresses and fast failover between Regions. Which service fits BEST?

2. An HPC application needs the lowest possible network latency between its EC2 instances. What should you use?

3. Clickstream data must be loaded into S3 in near real time with no custom consumer code. Which service fits BEST?

4. A company connects 40 VPCs and two data centers and wants transitive routing with central management. What should they use?