Terminal
Terminal/Skill
Helps operate and troubleshoot Amazon MSK Provisioned clusters, covering health reviews, performance issues, consumer lag, broker restarts, and alarms.
By Amazon Web Services
Works with
npx skills add aws/tools-for-devops-agent --skill msk-operationsOperate, troubleshoot, and assess Amazon MSK (Managed Streaming for Apache Kafka) Provisioned clusters — both Standard and Express broker types. This skill covers day-to-day operations (health assessments, monitoring setup) and ad-hoc incident response (performance degradation, consumer lag, storage full, unexpected broker reboots).
Activate this skill when the user asks to:
AWS/Kafka
namespace.Do not activate this skill for MSK Connect, MSK Serverless, or MSK Replicator — those are separate services with their own operational surfaces.
Determine the broker type first — many checks differ between Standard and Express.
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
Check ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType:
kafka. (e.g. kafka.m5.large, kafka.m7g.xlarge) → Standard broker.express. (e.g. express.m7g.large) → Express broker.Standard brokers use customer-managed EBS volumes for storage. You choose
instance types (kafka.m5.*, kafka.m7g.*), provision EBS, and manage storage
scaling. Standard brokers have scheduled maintenance windows.
Express brokers provide fully managed, pay-as-you-go storage with no EBS
provisioning. Instance types are prefixed with express.m7g.*. Express brokers
offer up to 3× more throughput per broker than Standard, and have no maintenance
windows. Express enforces a fixed replication factor of 3 and
min.insync.replicas=2 — you cannot create topics with RF=1.
references/
that mutates cluster state — update-broker-storage, create-configuration,
update-cluster-configuration, update-monitoring, put-metric-alarm,
reboot-broker, and any partition reassignment — is a recommendation for
the operator to run after review. Present these as proposed remediations
with expected impact and preconditions; do NOT execute them, and do NOT
imply that the agent will run them.UnderReplicatedPartitions > 0 (Standard only —
Express brokers do not emit URP). This risks data loss and extended outages.linger.ms=0 is the #1 cause of "high CPU" on MSK. ALWAYS check client
batch configuration before recommending broker scaling.VolumeReadBytes, VolumeWriteBytes, VolumeQueueLength)
when diagnosing Standard broker latency.min.insync.replicas=2 — do NOT
attempt to create topics with RF=1 on Express. If RF=1 is needed, use Standard
brokers.These five checks cover the most common MSK issues. Use them before loading a reference file.
CpuUser + CpuSystem > 60%: Check RequestHandlerAvgIdlePercent
(PER_BROKER monitoring level). If < 30%, request threads are saturated. Check
client batch.size and linger.ms before recommending scaling.
KafkaDataLogsDiskUsed > 85% (Standard only): Recommend to the
operator that EBS be expanded via aws kafka update-broker-storage (do
not execute). Identify high-growth topics via per-topic BytesInPerSec to
size the increase. Express clusters use StorageUsed metric instead and
storage is fully managed.
UnderReplicatedPartitions > 0 (Standard only): Check if a maintenance
operation or broker restart is in progress. If URP is decreasing, wait for
recovery. Do NOT restart brokers or reassign partitions during URP. Express
brokers do not emit this metric — monitor ProduceThrottleTime,
FetchThrottleTime, and consumer lag instead.
Consumer OffsetLag / MaxOffsetLag increasing: Determine if broker-side
(high ProduceTotalTimeMsMean, CPU saturation) or client-side (slow
processing, insufficient consumers). Per-partition lag from
PER_TOPIC_PER_PARTITION monitoring level helps isolate hot partitions.
BytesInPerSec near throughput ceiling: For Standard, check EBS volume
type and calculate: BytesInPerSec × ReplicationFactor vs volume throughput
limit. For Express, check against the per-broker sustained performance limits
in the MSK quotas.
Route to a reference file based on the customer intent. Read the reference in full before answering — do not paraphrase from memory.
| Customer Intent | Reference |
|---|---|
| High CPU, high produce/fetch latency, slow cluster, TrafficShaping | references/troubleshoot-performance.md |
| Consumer lag increasing, rebalance storms, stuck consumer groups | references/troubleshoot-consumer-lag.md |
| Disk filling up, retention planning, tiered storage, EBS scaling | references/manage-storage.md |
| Setting up monitoring level, dashboards, recommended CloudWatch alarms | references/monitor-and-alarm.md |
| Rolling restart impact, patching, Kafka version upgrades, maintenance resilience | references/maintenance-operations.md |
| Producer / consumer configuration, IAM / SCRAM / TLS auth for clients | references/configure-clients.md |
For sizing questions (broker count, instance type choice, monthly cost), refer the user to the Amazon MSK best practices — right-size your cluster documentation. Do not size from memory.
Use this workflow when the user asks for a review, audit, health check, or assessment of an MSK cluster. The routing table above handles ad-hoc troubleshooting; this section produces a consistent, comprehensive report.
Follow the steps in order for each target cluster. Do not skip steps. If a step cannot be completed (e.g. a metric requires a higher monitoring level than the cluster has enabled), record the gap in the report rather than silently omitting the check.
Ask the user which MSK clusters to review. Accept any of:
If no scope is given, default to all configured account regions. Enumerate
clusters with aws kafka list-clusters-v2 per region.
For each cluster:
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
Read ClusterInfo.Provisioned.BrokerNodeGroupInfo.InstanceType. Standard
brokers (kafka.*) and Express brokers (express.*) require different checks
in the later steps — some metrics only exist on one type.
For each cluster, gather:
aws kafka describe-cluster-v2 --cluster-arn <arn>
aws kafka list-nodes --cluster-arn <arn>
aws kafka get-bootstrap-brokers --cluster-arn <arn>
aws kafka list-cluster-operations-v2 --cluster-arn <arn> # last 30 days
aws kafka describe-configuration-revision \
--arn <configuration-arn> --revision <revision> # if a custom config is applied
Capture:
ZoneIds), storage mode (EBS / Tiered), current version.EncryptionInTransit.ClientBroker (TLS / TLS_PLAINTEXT /
PLAINTEXT), EncryptionInTransit.InCluster, EncryptionAtRest.DataVolumeKMSKeyId.ClientAuthentication.Sasl.Iam.Enabled,
ClientAuthentication.Sasl.Scram.Enabled, ClientAuthentication.Tls.Enabled,
ClientAuthentication.Unauthenticated.Enabled.BrokerNodeGroupInfo.ConnectivityInfo.PublicAccess.Type.EnhancedMonitoring (DEFAULT / PER_BROKER /
PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION).LoggingInfo.BrokerLogs (CloudWatch / S3 / Firehose destinations
and their Enabled flags).OpenMonitoring.Prometheus.JmxExporter.EnabledInBroker,
NodeExporter.EnabledInBroker.list-cluster-operations-v2, note any
SECURITY_PATCHING, BROKER_UPDATE, UPDATE_CLUSTER_CONFIGURATION,
UPDATE_STORAGE, or UPDATE_MONITORING events in the review window.The EnhancedMonitoring value from Step 3 determines which checks are
available. At DEFAULT, most per-broker health metrics are still available
(CPU, disk, network, partitions, connections, memory, TrafficShaping), but the
following checks are not possible without upgrading:
ReplicationBytesInPerSec / ReplicationBytesOutPerSec (inter-broker
replication load)RequestHandlerAvgIdlePercent / NetworkProcessorAvgIdlePercent (thread pool
saturation)BwInAllowanceExceeded / BwOutAllowanceExceeded (detailed bandwidth
breaches)VolumeQueueLength (EBS I/O queue depth — Standard only)VolumeReadBytes / VolumeWriteBytes (EBS throughput utilization — Standard
only)IAMTooManyConnections)Record the monitoring level and list any dimensions that will be scored
partially or skipped. Recommend upgrading to PER_BROKER if any dimension is
degraded by the current level.
Namespace: AWS/Kafka. Dimensions: Cluster Name and Broker ID for
per-broker metrics; Cluster Name and Consumer Group and Topic for
consumer lag; Cluster Name only for cluster-wide metrics.
Use one cloudwatch.GetMetricData batch per cluster where possible.
Period: 3600 (1 hour). StartTime: 7 days ago. EndTime: now.
| Metric | Stat | Purpose |
|---|---|---|
ActiveControllerCount | Sum | Must be exactly 1 |
OfflinePartitionsCount | Maximum | Must be 0 |
GlobalPartitionCount | Maximum | Total leader partitions |
GlobalTopicCount | Maximum | Total topics |
kafka.*)| Metric | Stat | Threshold |
|---|---|---|
CpuUser + CpuSystem | Average, Maximum | < 60% avg |
KafkaDataLogsDiskUsed | Average, Maximum | < 70% avg, < 85% max |
PartitionCount | Maximum | ≤ recommended limit for broker size |
LeaderCount | Maximum | Compare across brokers; skew < 10% |
UnderReplicatedPartitions | Maximum | 0 in steady state |
UnderMinIsrPartitionCount | Maximum | 0 |
BytesInPerSec / BytesOutPerSec | Average, Maximum | vs baseline bandwidth |
ConnectionCount | Average, Maximum | Compare across brokers |
HeapMemoryAfterGC | Maximum | < 60% |
TrafficShaping | Sum | Must be 0 |
NetworkRxDropped / NetworkTxDropped / NetworkRxErrors / NetworkTxErrors | Sum | Must be 0 |
ReplicationBytesInPerSec / ReplicationBytesOutPerSec | Average | Requires PER_BROKER |
RequestHandlerAvgIdlePercent / NetworkProcessorAvgIdlePercent | Average | > 30% (PER_BROKER) |
VolumeQueueLength | Average, Maximum | Avg < 1 (PER_BROKER) |
VolumeReadBytes + VolumeWriteBytes | Sum | vs EBS baseline throughput (PER_BROKER) |
express.*)| Metric | Stat | Threshold |
|---|---|---|
CpuUser + CpuSystem | Average, Maximum | < 60% avg |
StorageUsed | Maximum | Fully managed — flag if trending against per-broker quota |
PartitionCount | Maximum | ≤ recommended limit for broker size |
LeaderCount | Maximum | Compare across brokers |
BytesInPerSec / BytesOutPerSec | Average, Maximum | vs Express per-broker ingress/egress quotas |
ProduceThrottleTime / FetchThrottleTime | Maximum | Should be 0 |
ClientConnectionCount | Average, Maximum | vs listener quota (see references/monitor-and-alarm.md) |
Express brokers do NOT emit: UnderReplicatedPartitions,
UnderMinIsrPartitionCount, HeapMemoryAfterGC, TrafficShaping, Volume*
metrics, KafkaDataLogsDiskUsed, ProduceMessageConversionsPerSec,
FetchMessageConversionsPerSec.
Per consumer group (identified from the customer or from the broker logs):
| Metric | Stat | Notes |
|---|---|---|
SumOffsetLag per (Consumer Group, Topic) | Maximum | DEFAULT level |
MaxOffsetLag per (Consumer Group, Topic) | Maximum | DEFAULT level |
EstimatedMaxTimeLag per (Consumer Group, Topic) | Maximum | DEFAULT level |
OffsetLag per (Consumer Group, Topic, Partition) | Maximum | Requires PER_TOPIC_PER_PARTITION |
Collect existing alarms:
aws cloudwatch describe-alarms --namespace AWS/Kafka
Compare against the 13-alarm recommended set (details in
references/monitor-and-alarm.md) and produce a coverage table (present / missing / firing).
Assign a severity to every finding: CRITICAL / HIGH / MEDIUM / LOW / INFO (see Severity Definitions later in this file).
Evaluate across seven dimensions. Some checks are skipped for Express — noted inline.
ACTIVE. MAINTENANCE / UPDATING is transient; anything
else is a finding.PER_BROKER or higher — DEFAULT → MEDIUM.TLS. TLS_PLAINTEXT → MEDIUM (prod: HIGH).
PLAINTEXT → CRITICAL.Unauthenticated.Enabled=true in production → CRITICAL.SERVICE_PROVIDED_EIPS in production → HIGH.ALARM state → HIGH (surface in report header).PartitionCount per broker ≤ recommended limit for the broker instance
type. Above recommended but below max → MEDIUM. Above max → HIGH (blocks
update operations).LeaderCount variance across brokers < 10%. 10-25% → MEDIUM. > 25% → HIGH.UnderReplicatedPartitions > 0 sustained (Standard only) → HIGH. Transient
during a SECURITY_PATCHING / BROKER_UPDATE operation from Step 3 →
INFO — do NOT flag.UnderMinIsrPartitionCount > 0 (Standard only) → CRITICAL.min.insync.replicas >= replication.factor
are a configuration error the CloudWatch metric will not surface. If the
user has provided topic configs, flag any such topic as HIGH.CpuUser + CpuSystem avg < 60%. 60-70% → MEDIUM. > 70% → HIGH.HeapMemoryAfterGC < 60% (Standard only). 60-80% → MEDIUM. > 80% → HIGH.RequestHandlerAvgIdlePercent > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.NetworkProcessorAvgIdlePercent > 30% (PER_BROKER). 10-30% → MEDIUM. < 10% → HIGH.TrafficShaping = 0 (Standard only). Any non-zero → HIGH.BwInAllowanceExceeded / BwOutAllowanceExceeded = 0 (PER_BROKER). Non-zero → HIGH.references/troubleshoot-performance.md for baseline table). 60-70% →
MEDIUM. > 70% → HIGH.ProduceThrottleTime / FetchThrottleTime > 0 → HIGH.Standard only (Express storage is managed — Express clusters only get the
StorageUsed check):
KafkaDataLogsDiskUsed < 70%. 70-85% → HIGH. > 85% → CRITICAL.VolumeQueueLength avg < 1 (PER_BROKER). 1-5 → MEDIUM. > 5 → HIGH.VolumeReadBytes + VolumeWriteBytes) < 60% of instance
baseline (PER_BROKER). 60-70% → MEDIUM. > 70% → HIGH.Express:
StorageUsed per broker vs Express per-broker storage quota (see MSK
Express quotas). Approaching quota → MEDIUM.Generate a separate report artifact per cluster reviewed.
Artifact naming: msk-review-<cluster-name>-<YYYY-MM-DD>.md
Example: msk-review-prod-orders-2026-04-29.md
Report structure:
# MSK Operational Review — <cluster-name>
Account: <account-id> | Region: <region> | Date: <YYYY-MM-DD>
Broker Type: Standard/Express | Instance Type: <type> | Broker Count: <n> | AZs: <n>
Kafka Version: <version> | Monitoring Level: <level>
| Item | Value | | Cluster state / version | … | | Broker type / instance / count / AZs | … | | Storage | mode, size (Standard), provisioned throughput (if any) | | Encryption | in-transit, in-cluster, at-rest (KMS) | | Authentication | IAM / SCRAM / mTLS / Unauthenticated flags | | Public access | DISABLED / SERVICE_PROVIDED_EIPS | | Monitoring level | DEFAULT / PER_BROKER / PER_TOPIC_PER_BROKER / PER_TOPIC_PER_PARTITION | | Logging | destinations enabled |
For each of the 7 dimensions (7.1-7.7):
| # | Finding | Severity | Current State | Recommendation |
If a dimension was skipped or partial due to monitoring level or broker type, say so explicitly in a note above the table for that dimension.
| Metric | Stat | 7-Day Avg | 7-Day Max | Status | Finding |
| # | Recommended Alarm | Metric | Threshold | Priority | Status |
Also list alarms currently in ALARM state with timestamps.
From list-cluster-operations-v2 in Step 3:
| Operation Type | Start Time | End Time | State |
Call out any operations that would explain transient metric anomalies.
| # | Finding | Severity | Dimension | Effort | Impact |
Sorted by severity.
Describe cluster:
aws kafka describe-cluster-v2 --cluster-arn <cluster-arn>
List brokers:
aws kafka list-nodes --cluster-arn <cluster-arn>
Get bootstrap brokers:
aws kafka get-bootstrap-brokers --cluster-arn <cluster-arn>
List recent cluster operations (patching, config updates, storage changes):
aws kafka list-cluster-operations-v2 --cluster-arn <cluster-arn>
Expand Standard broker storage — Operator-run (recommend, do not execute):
aws kafka update-broker-storage \
--cluster-arn <cluster-arn> \
--current-version <cluster-version> \
--target-broker-ebs-volume-info '[{"KafkaBrokerNodeId": "All", "VolumeSizeGB": <target-size>}]'
Get a CloudWatch metric (example: CpuUser per broker):
aws cloudwatch get-metric-statistics \
--namespace AWS/Kafka \
--metric-name CpuUser \
--dimensions Name="Cluster Name",Value="<cluster-name>" Name="Broker ID",Value="<broker-id>" \
--start-time <start> --end-time <end> --period 300 --statistics Average
Create a cluster configuration (server.properties) — Operator-run (recommend, do not execute):
The --server-properties argument MUST be a real Kafka properties file with
one key=value per line, separated by actual newline characters — NOT the
literal two-character escape sequence \n. The MSK API accepts the bytes as-is;
if you pass "k1=v1\nk2=v2" as a single string with escaped newlines, MSK
stores ONE invalid property line and the cluster will fail to apply it.
Recommended pattern: write the properties to a local file with real newlines,
then pass it via fileb:// so the CLI uploads the raw bytes verbatim. Verify by
reading the revision back with describe-configuration-revision and
base64-decoding ServerProperties — you should see one property per line.
cat > server.properties <<'EOF'
auto.create.topics.enable=false
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false
num.io.threads=32
num.network.threads=16
log.retention.hours=168
EOF
aws kafka create-configuration \
--name <config-name> \
--kafka-versions "3.6.0" \
--server-properties fileb://server.properties
| Error | Cause | Fix |
|---|---|---|
aws kafka update-broker-storage returns "storage is optimizing" | Previous storage expansion still in cool-down (minimum 6 hours) | Wait for optimization to complete. Check cluster state with describe-cluster-v2. |
ClusterState is MAINTENANCE | Standard brokers undergoing patching. Express brokers stay ACTIVE during maintenance. | Wait for cluster to return to ACTIVE. Do not perform update operations during MAINTENANCE. |
Consumer receives GROUP_COORDINATOR_NOT_AVAILABLE | Coordinator broker is temporarily unavailable during rolling restart or overloaded | Retry with backoff. Check if maintenance is in progress via list-cluster-operations-v2. |
NotEnoughReplicasException on produce | Fewer brokers in ISR than min.insync.replicas (default: 2) | Check UnderReplicatedPartitions (Standard only). For Express, check ProduceThrottleTime and broker health instead — URP is not available. If a broker is down for maintenance, this is transient. Do NOT lower min.insync.replicas to work around this. |
| Severity | Definition | SLA |
|---|---|---|
| CRITICAL | Immediate risk to availability, security, or data integrity | Fix within 24–48 hours |
| HIGH | Significant gap that could lead to incidents | Fix within 1 week |
| MEDIUM | Notable improvement opportunity | Plan within 30 days |
| LOW | Minor optimization or hardening | Address when convenient |
| INFO | Observation, no action required | N/A |