VPC Flow Logs        -> metadata flow-level: siapa bicara ke siapa, port apa, action ACCEPT/REJECT, berapa packet/bytes
VPC Traffic Mirroring -> copy packet-level traffic dari ENI ke monitoring appliance

Flow Logs menjawab: apakah ada flow, arahnya ke mana, diterima/ditolak, volumenya berapa?
Traffic Mirroring menjawab: apa isi packet/protocol behavior yang sebenarnya terjadi?

source address
source port
destination address
destination port
protocol

interface-id
account-id
vpc-id/subnet-id/instance-id jika custom field
packets
bytes
start/end timestamp
action ACCEPT/REJECT
log-status OK/NODATA/SKIPDATA
traffic-path
flow-direction
pkt-srcaddr/pkt-dstaddr

HTTP URL
TLS SNI secara langsung
request body
DNS query detail ke AmazonProvidedDNS
application error message
IAM AccessDenied cause
TCP retransmission detail lengkap

VPC      -> semua ENI di VPC, high coverage, high volume
Subnet   -> ENI di subnet tertentu, bagus untuk subnet class
ENI      -> granular, bagus untuk forensic/debugging target tertentu

1. Enable VPC/subnet-level flow logs untuk critical environments.
2. Publish ke S3 dalam format queryable.
3. Gunakan CloudWatch Logs untuk near-real-time operational debugging bila perlu.
4. Gunakan ENI-level logs temporary untuk incident deep dive.

CloudWatch Logs
Amazon S3
Amazon Data Firehose

CloudWatch Logs -> short retention for ops
S3 Parquet      -> long retention and analytics
Firehose/SIEM   -> security detection pipeline

vpc-id
subnet-id
instance-id
pkt-srcaddr
pkt-dstaddr
flow-direction
traffic-path
region
az-id
sublocation-type
sublocation-id
tcp-flags
type

${version} ${account-id} ${region} ${az-id} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${flow-direction} ${traffic-path} ${pkt-srcaddr} ${pkt-dstaddr} ${tcp-flags} ${type}

2 111122223333 eni-abc123 10.0.12.34 10.0.21.50 49152 443 6 10 8400 1783350000 1783350060 ACCEPT OK

version       = 2
account       = 111122223333
interface-id  = eni-abc123
srcaddr       = 10.0.12.34
 dstaddr      = 10.0.21.50
srcport       = 49152
dstport       = 443
protocol      = 6 TCP
packets       = 10
bytes         = 8400
start/end     = aggregation window
action        = ACCEPT
log-status    = OK

TLS handshake error
HTTP 500
IAM AccessDenied
DNS wrong target
server closed connection
app timeout after accepting connection

Security Group
Network ACL

Client -> Server:443 allowed by SG inbound
Server -> Client:ephemeral response allowed automatically by SG statefulness

inbound 443
outbound ephemeral

1. Cari flow client -> server dstport 443.
2. Cari reverse flow server -> client dstport ephemeral/source 443.
3. Bandingkan action pada kedua arah.
4. Jika forward ACCEPT tapi reverse REJECT, pikirkan NACL/stateless return path.

Jangan berharap flow log muncul detik yang sama.
Jangan menggunakan Flow Logs sebagai satu-satunya detector untuk sub-second issue.
Untuk incident, kombinasikan dengan app logs, ALB/NLB logs, CloudTrail, metric, dan packet capture jika perlu.

OK       -> log record valid
NODATA   -> tidak ada traffic selama aggregation interval
SKIPDATA -> ada flow yang tidak tertangkap karena internal capacity/processing constraint

NODATA tidak sama dengan network blocked.
SKIPDATA tidak boleh diabaikan dalam forensic-sensitive workload.

1. Kurangi scope jika perlu.
2. Gunakan subnet/ENI-specific logs untuk investigation.
3. Pastikan downstream destination tidak menjadi bottleneck.
4. Jangan membuat kesimpulan absolut dari dataset yang kehilangan record.

Traffic instance ke Amazon DNS server
Traffic ke instance metadata 169.254.169.254
Traffic ke Amazon Time Sync 169.254.169.123
DHCP traffic
ARP traffic
Traffic ke reserved IP default VPC router
Traffic mirrored source traffic
Beberapa traffic antara endpoint ENI dan NLB ENI

Jika DNS ke AmazonProvidedDNS tidak muncul di Flow Logs, itu expected.
Jika IMDS call tidak muncul, itu expected.
Jika Anda butuh DNS query visibility, gunakan Route 53 Resolver query logging/DNS Firewall telemetry.
Jika Anda butuh packet payload, gunakan Traffic Mirroring atau host capture.

Internet client cannot reach public ALB or EC2.

1. Identify ENI of target instance or load balancer nodes if visible via logs.
2. Filter dstaddr = private IP of ENI or pkt-dstaddr if needed.
3. Filter dstport = service port.
4. Check action.
5. Check reverse flow to client ephemeral port.
6. Confirm route table has IGW route for public subnet.
7. Confirm NACL allows inbound service port and outbound ephemeral.
8. Confirm SG allows inbound from source.

DNS/public IP wrong
client never reached AWS
route/IGW/public IP association issue
load balancer path, not instance path
upstream firewall blocked

SG/NACL issue at target boundary.

Application/listener/TLS/backend issue.

Service A in private subnet cannot call Service B in private subnet.

A ENI -> local VPC route -> B ENI

Filter A ENI egress:
  srcaddr = A private IP
  dstaddr = B private IP
  dstport = target port

Filter B ENI ingress:
  srcaddr = A private IP
  dstaddr = B private IP
  dstport = target port

Private workload accesses internet or AWS public endpoint through NAT Gateway.

Which source workloads create NAT cost?
Which destinations consume most egress?
Which traffic should move to VPC endpoints?
Is there suspicious exfiltration?

srcaddr
pkt-srcaddr
dstaddr
pkt-dstaddr
dstport
bytes
packets
traffic-path
interface-id
subnet-id

1. Enable VPC/subnet flow logs with packet address fields.
2. Aggregate bytes by srcaddr/dstaddr/dstport.
3. Identify high-volume AWS service/public IP destinations.
4. Move S3/DynamoDB to gateway endpoint where appropriate.
5. Move AWS APIs to interface endpoints where appropriate.
6. Restrict/inspect remaining egress.

SELECT
  srcaddr,
  dstaddr,
  dstport,
  protocol,
  SUM(bytes) AS total_bytes,
  COUNT(*) AS records
FROM vpc_flow_logs
WHERE action = 'ACCEPT'
  AND start_time >= TIMESTAMP '2026-07-06 00:00:00'
GROUP BY srcaddr, dstaddr, dstport, protocol
ORDER BY total_bytes DESC
LIMIT 100;

1. Resolve service hostname from workload.
2. Identify endpoint ENI private IPs.
3. Query Flow Logs for workload -> endpoint ENI:443.
4. Check NAT Gateway bytes drop for that service path.
5. Correlate CloudTrail event with VPC endpoint context where available.

1. Confirm route table has prefix list target for S3/DynamoDB.
2. Query flows from workload to service prefix/list destination.
3. Confirm NAT is not used for that service path.
4. Check bucket/table policy effects separately.

Endpoint policy denial may show network ACCEPT and CloudTrail AccessDenied.

Did the client actually use AAAA record?
Does route table include ::/0 to IGW or egress-only IGW?
Do SG/NACL rules include IPv6 ranges separately?
Is return path allowed by NACL?
Is app binding/listening on IPv6?

source IPv6
destination IPv6
protocol/port
action
bytes/packets

Is this ingress or egress relative to the ENI?
Did traffic go through internet gateway, NAT gateway, transit gateway, or local path?
Which flows are leaving VPC boundaries?

Do not rely on one field alone for compliance conclusions.
Correlate with route tables, ENI metadata, NAT/TGW/endpoint IDs, CloudTrail, and resource tags.

ENI ID -> instance/ECS task/EKS pod/Lambda ENI/endpoint ENI/load balancer ENI
subnet ID -> subnet class/environment/AZ
security group -> app boundary
route table -> route intent
account ID -> workload owner
IP/CIDR -> internal service registry
endpoint ENI -> AWS service endpoint

Whose IP is 10.42.17.93?

payments-api task in prod private app subnet ap-southeast-1a called secretsmanager endpoint on 443 and was accepted.

fields @timestamp, interfaceId, srcAddr, dstAddr, srcPort, dstPort, action, logStatus
| filter dstPort = 443 and action = 'REJECT'
| sort @timestamp desc
| limit 100

fields srcAddr, dstAddr, dstPort, bytes
| stats sum(bytes) as totalBytes by srcAddr, dstAddr, dstPort
| sort totalBytes desc
| limit 50

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, action, bytes
| filter srcAddr = '10.40.12.34'
| filter dstAddr in ['10.40.100.10', '10.40.101.10']
| filter dstPort = 443
| sort @timestamp desc

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, action
| filter action = 'REJECT'
| filter dstPort >= 1024
| sort @timestamp desc
| limit 100

Parquet format
hourly partitioning
Hive-compatible prefixes
lifecycle retention policy
central logging account
bucket policy with least privilege
KMS encryption
partition projection where useful

s3://org-network-logs/AWSLogs/<account-id>/vpcflowlogs/<region>/year=<yyyy>/month=<mm>/day=<dd>/hour=<hh>/

NAT cost attribution
exfiltration detection
incident timeline reconstruction
least-privilege SG/NACL review
endpoint migration validation
traffic baseline per service
regulatory evidence

ingestion volume
CloudWatch storage
S3 storage
Athena query scan size
Firehose/SIEM downstream cost
high cardinality queries
keeping all traffic forever

1. Use S3/Parquet for high-volume long-term analytics.
2. Keep CloudWatch retention short unless justified.
3. Use lifecycle policies.
4. Partition by account/region/date/hour.
5. Use custom fields intentionally.
6. Use subnet/ENI logs for temporary deep debugging.
7. Aggregate detections instead of querying raw logs constantly.

payload/protocol inspection
TLS handshake details
packet retransmission behavior
malware/network threat inspection
IDS/IPS appliance feed
deep troubleshooting of weird TCP/application behavior

Source  -> ENI whose traffic is copied
Target  -> destination for mirrored packets
Filter  -> rules selecting traffic to mirror
Session -> binds source, target, filter, priority, packet length

Network interface of monitoring appliance
Network Load Balancer with UDP listener
Gateway Load Balancer with UDP listener

Your appliance must support the mirrored traffic format.
Security tools must parse encapsulated packets.
Packet length/truncation settings affect what payload is visible.

mirror encapsulation support
MTU/jumbo frame handling
packet truncation length
UDP listener configuration
appliance capture interface
source/target route path

source/destination CIDR
protocol
port range
inbound/outbound direction
accept/reject rule ordering

Mirror only app -> database TCP 5432.
Mirror only suspicious egress destination CIDR.
Mirror only inbound HTTPS to a test host during incident window.
Mirror only DNS to custom resolver.

cost
appliance capacity
privacy
signal/noise
blast radius of telemetry exposure

runs inside instance OS
can capture before/after local firewall depending interface
requires host access
can be tampered with by compromised host/root
cheap for ad-hoc debugging

copies from ENI outside guest userspace
centralized target
good for security appliances
requires supported instance/ENI setup
more operational overhead
sensitive/high-volume telemetry

For one-off developer debugging, host tcpdump may be enough.
For security monitoring and compromised-host assumptions, Traffic Mirroring is stronger.

headers
metadata
internal service names
unencrypted payloads
credentials if protocols are insecure
PII/payment/regulatory data

1. Limit who can create mirror sessions.
2. Use change approval for production mirroring.
3. Filter narrowly.
4. Encrypt/log/monitor access to appliances.
5. Isolate monitoring VPC/account.
6. Apply retention and data handling policy.
7. Audit mirror session creation/modification.

Is mirrored traffic allowed by route/security controls?
Can monitoring target handle aggregate throughput?
Is target in same region?
What happens if appliance fails?
Does mirroring path compete with production traffic capacity?
Who owns monitoring account and data?

Problem: intermittent TLS handshake failures from app to internal API.

app -> api:443 ACCEPT
bytes sometimes small
no REJECT

Traffic Mirroring on app ENI or API ENI for tcp/443 between two CIDRs.

SYN/SYN-ACK/ACK timing
TLS ClientHello/ServerHello
RST/FIN behavior
retransmissions
MTU/path fragmentation symptoms
proxy behavior

NAT Gateway cost spike
DataTransfer-Out increased
Security alert for unusual external traffic

1. Query Flow Logs by bytes grouped by srcaddr,dstaddr,dstport.
2. Map srcaddr to ENI/instance/task/pod owner.
3. Map dstaddr to AWS service/public ASN/domain if possible.
4. Check whether destination should use VPC endpoint.
5. Check CloudTrail for related API activity.
6. Check application deploys/config changes around spike time.
7. Temporarily add Traffic Mirroring if payload/protocol inspection is needed and approved.
8. Apply containment: SG egress restriction, NACL, Network Firewall, route change, app config rollback.
9. Preserve logs and mirror captures according to incident policy.

App cannot connect to database.
Error: connection timeout.

1. Query app ENI -> DB ENI dstport 5432/3306/etc.
2. Query DB ENI ingress from app IP.
3. Query DB ENI -> app ENI reverse flow.
4. Look for REJECT on either direction.
5. If ACCEPT both directions, inspect DB listener/app credentials/TLS.

No flow from app -> app didn't send, DNS wrong, connection pool not attempting.
REJECT at app egress -> app SG/NACL.
REJECT at DB ingress -> DB SG/NACL.
Forward ACCEPT, reverse REJECT -> NACL ephemeral issue.
Both ACCEPT -> not basic network filtering.

Capture app<->DB TCP handshake and TLS startup only, narrow filter, limited window.

Private workload cannot call Secrets Manager/S3/KMS/ECR.

DNS -> Flow Logs -> CloudTrail -> Service-specific logs

1. Resolve hostname from runtime.
2. If interface endpoint, identify endpoint ENI IP.
3. Query Flow Logs workload -> endpoint ENI:443.
4. If gateway endpoint, check route table prefix list and NAT avoidance.
5. If Flow Logs show REJECT, fix SG/NACL.
6. If Flow Logs show ACCEPT and CloudTrail has AccessDenied, fix policy.
7. If no CloudTrail event, request likely did not reach service auth layer.
8. If intermittent, inspect DNS cache, endpoint AZ placement, app retries, service throttling.

Network team edits SG.
App team edits IAM.
Platform team edits endpoint.
Nobody proves packet path.

1. VPC/subnet Flow Logs enabled for prod VPCs.
2. Central S3 log archive with encryption and retention.
3. Query model via Athena or SIEM.
4. ENI/subnet/account/tag enrichment pipeline.
5. NAT egress dashboard.
6. REJECT dashboard for critical subnets.
7. Endpoint adoption dashboard.
8. Runbook templates for timeout/reject/AccessDenied.
9. Traffic Mirroring approved process for incidents.
10. IAM guardrails around log deletion and mirror session creation.

Route 53 Resolver query logs
ALB/NLB logs where applicable
CloudFront logs for edge paths
CloudTrail organization trail
GuardDuty/VPC DNS findings
Network Firewall logs if used
Transit Gateway Flow Logs in multi-VPC architecture

Outbound traffic to unexpected ports from private app subnets.
Inbound traffic to database subnet from non-app subnet.
High bytes to internet via NAT from a single workload.
Rejected traffic spikes after SG/NACL deployment.
Traffic to metadata IP will not appear in Flow Logs; use other telemetry.
Private workloads still using NAT for AWS service endpoints.
Cross-AZ chatter for latency/cost investigation.
Unexpected east-west traffic between bounded domains.

1. Tag-enriched detections beat raw CIDR detections.
2. Baseline by subnet class and workload type.
3. Alert on meaningful deviation, not every reject.
4. Distinguish scanning/noise from production regression.
5. Validate with owners before auto-remediation unless high-confidence.

aws ec2 create-flow-logs \
  --resource-type VPC \
  --resource-ids vpc-0123456789abcdef0 \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::org-network-flow-logs/prod/ \
  --log-format '${version} ${account-id} ${region} ${az-id} ${vpc-id} ${subnet-id} ${instance-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status} ${flow-direction} ${traffic-path} ${pkt-srcaddr} ${pkt-dstaddr} ${tcp-flags} ${type}'

resource "aws_flow_log" "prod_vpc" {
  log_destination      = aws_s3_bucket.flow_logs.arn
  log_destination_type = "s3"
  traffic_type         = "ALL"
  vpc_id               = aws_vpc.prod.id

  log_format = join(" ", [
    "$${version}",
    "$${account-id}",
    "$${region}",
    "$${az-id}",
    "$${vpc-id}",
    "$${subnet-id}",
    "$${instance-id}",
    "$${interface-id}",
    "$${srcaddr}",
    "$${dstaddr}",
    "$${srcport}",
    "$${dstport}",
    "$${protocol}",
    "$${packets}",
    "$${bytes}",
    "$${start}",
    "$${end}",
    "$${action}",
    "$${log-status}",
    "$${flow-direction}",
    "$${traffic-path}",
    "$${pkt-srcaddr}",
    "$${pkt-dstaddr}",
    "$${tcp-flags}",
    "$${type}"
  ])

  tags = {
    Environment = "prod"
    Purpose     = "network-observability"
  }
}

resource "aws_ec2_traffic_mirror_target" "nids" {
  network_load_balancer_arn = aws_lb.nids.arn

  tags = {
    Name = "prod-nids-mirror-target"
  }
}

resource "aws_ec2_traffic_mirror_filter" "https_only" {
  description = "Mirror selected HTTPS traffic for incident analysis"
}

resource "aws_ec2_traffic_mirror_filter_rule" "egress_https" {
  traffic_mirror_filter_id = aws_ec2_traffic_mirror_filter.https_only.id
  rule_number              = 100
  rule_action              = "accept"
  traffic_direction        = "egress"
  protocol                 = 6
  destination_port_range {
    from_port = 443
    to_port   = 443
  }
  source_cidr_block      = "10.40.0.0/16"
  destination_cidr_block = "10.50.0.0/16"
}

resource "aws_ec2_traffic_mirror_session" "app_to_nids" {
  network_interface_id     = aws_instance.app.primary_network_interface_id
  traffic_mirror_target_id = aws_ec2_traffic_mirror_target.nids.id
  traffic_mirror_filter_id = aws_ec2_traffic_mirror_filter.https_only.id
  session_number           = 1
  packet_length            = 128

  tags = {
    Purpose = "temporary-incident-analysis"
  }
}

Flow Logs show flow metadata, not packet payload.
Traffic Mirroring copies packets for deep inspection.
ACCEPT does not mean application success.
REJECT needs direction and ENI context.
Reverse path matters, especially with NACL.
DNS visibility requires DNS-specific logging.
Endpoint debugging requires DNS + Flow Logs + CloudTrail.
Raw IP logs must be enriched with inventory to become operationally useful.
Traffic Mirroring is powerful but sensitive and should be tightly governed.

https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html
https://docs.aws.amazon.com/vpc/latest/userguide/flow-log-records.html
https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-records-examples.html
https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-limitations.html
https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3-create-flow-log.html
https://docs.aws.amazon.com/vpc/latest/mirroring/what-is-traffic-mirroring.html
https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-how-it-works.html
https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-network-limitations.html