Terminal
Terminal/Skill
Runs a read only health check on an OpenSearch domain: cluster status, shards, performance, security settings, and cost signals, with prioritized fixes.
By Amazon Web Services
Works with
npx skills add aws/tools-for-devops-agent --skill analytics-opensearch-expertiseThis skill performs a comprehensive, read-only health assessment of Amazon OpenSearch Service domains using AWS APIs available through the customer's local AWS profile. It produces a structured report with findings and actionable recommendations.
Scope: CLI-agent compatible. No external packages, no IDE workspace, no human review gates.
opensearch, opensearch health, domain health, cluster review, opensearch review, opensearch assessment, opensearch check, open search, opensearch performance, opensearch security, opensearch cost
es:DescribeDomaines:DescribeDomainHealthes:GetCompatibleVersionses:DescribeReservedInstanceses:ListDomainNamescloudwatch:GetMetricDataRequired: Domain ARN (e.g., arn:aws:es:us-west-2:123456789012:domain/my-domain)
Parse the ARN to extract:
us-west-2)123456789012)domain/ in field 6Fallback / Target Resolution (apply these rules exactly — never silently substitute a target):
DescribeDomain → proceed with the assessment.ListDomainNames in the user's likely region(s) (cross-region scan if needed) and present the discovered domains as options for the user to choose. Do not pick one yourself unless exactly one domain exists in the entire account — state that you did so.ListDomainNames in the ARN's region, then other regions). Then STOP and confirm — do NOT assess:
prod-es-content / prod-es-users), a silently substituted target produces a confident report about the wrong domain, which is worse than an error.1. Parse domain ARN → region, account, domain name
2. Resolve the target (Target Resolution rules above) — confirm with the user if not an exact match
3. Describe domain configuration (DescribeDomain)
4. Describe domain health (DescribeDomainHealth)
5. Pull CloudWatch metrics (last 24h)
6. Check upgrade eligibility (GetCompatibleVersions; ListVersions fallback)
7. Check reserved instance coverage (DescribeReservedInstances)
8. Analyze findings against thresholds
9. Load the Remediation Reference — read_skill_resource(skill_id='analytics-opensearch-expertise',
path='references/remediation-reference.md') — MANDATORY before writing recommendations
10. Generate report with recommendations
The DevOps Agent AWS tool exposes the LEGACY Elasticsearch-compatible client. Call legacy operation names FIRST; fall back to the modern names only if the legacy call fails:
| Purpose | Try first (legacy) | Modern equivalent |
|---|---|---|
| Domain config | describe_elasticsearch_domain | describe_domain |
| Version catalog | list_elasticsearch_versions | list_versions |
| Upgrade targets | get_compatible_elasticsearch_versions | get_compatible_versions |
| Reserved instances | describe_reserved_elasticsearch_instances | describe_reserved_instances |
es:Describe* and es:List* —
Describe/List operations succeed under either naming; Get* actions (BOTH GetCompatible
variants) are typically denied. This is exactly why the 1.4 ListVersions fallback exists.DescribeDomainHealth has NO legacy equivalent. If it fails, do NOT skip checks 2.2/2.3 —
use CloudWatch instead: Shards.active (Maximum) = active shards, Shards.unassigned
(Maximum) = unassigned, TotalShards ≈ active + unassigned; ClusterStatus.green/yellow/red
for 1.1/3.7. Never report the checks as SKIPPED when this fallback is available.AWS/ES, dimensions DomainName and
ClientId (the account ID).DescribeDomainHealthClusterHealth (green | yellow | red)DescribeDomain → ClusterConfigInstanceCount, ZoneAwarenessEnabled, AvailabilityZoneCountZoneAwarenessEnabled == true:
InstanceCount % AvailabilityZoneCount != 0 → ⚠️ "Data node count ({count}) is not a multiple of AZ count ({az_count}). This causes uneven shard distribution across AZs, reducing fault tolerance."ZoneAwarenessEnabled == false and InstanceCount > 1:
DescribeDomain → ClusterConfigDedicatedMasterEnabled, DedicatedMasterType, DedicatedMasterCountDedicatedMasterEnabled == false AND InstanceCount >= 3 → ⚠️ "No dedicated master nodes. Clusters with 3+ data nodes should use dedicated masters for stability."DedicatedMasterCount == 2 → ⚠️ "Even number of master nodes ({count}) risks split-brain. Use 3 dedicated master nodes."DedicatedMasterCount < 3 → ⚠️ "Fewer than 3 master nodes reduces fault tolerance."GetCompatibleVersions with DomainNameGetCompatibleVersions returns AccessDeniedException — do NOT skip):
call ListVersions and derive upgrade candidates using these two rules:
ListVersions with the SAME major version as the
domain and a LATER minor version (e.g., a 2.11 domain → every later 2.x in the region).ListVersions; never hard-code a specific version number).
Label fallback results as "derived upgrade candidates (ListVersions + documented upgrade-path
rules)" and cite https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html
for the reader. Only mark 1.4 SKIPPED if BOTH APIs are denied.
Note placement: the fallback provenance note belongs INSIDE the 1.4 finding ONLY — do NOT
surface it in the Executive Summary or any report-level note box, and do NOT name API errors
(e.g., AccessDeniedException) anywhere in the report. Phrase it customer-friendly: "Upgrade
candidates were derived from the available version listings and are not API-confirmed — verify
eligible target versions in the console before planning an upgrade."
Render candidates as a SET of direct single-hop targets (comma-separated) — NEVER as an arrow chain (NOT "2.13 → 2.15 → 2.19"): every listed same-major version is directly reachable in ONE hop. Any upgrade recommendation MUST propose a single hop to the latest same-major version (plus one more hop for a major jump). Do NOT invent sequential multi-hop plans or per-hop time estimates.ListVersions/GetCompatibleVersions (e.g., 2.11 with {2.13, 2.15, 2.17, 2.19} available = 4 releases behind) — NEVER the numeric delta between version numbers (2.19 − 2.11 is NOT "8 versions behind").DescribeDomain → EBSOptionsVolumeType, VolumeSize, Iops, ThroughputVolumeType == "gp2" → ⚠️ "Using gp2 volumes. gp3 offers better price/performance with configurable IOPS and throughput."VolumeSize × InstanceCountIopsThrottle, ThroughputThrottle, ReadIOPSMicroBursting, WriteIOPSMicroBursting, ReadThroughputMicroBursting, WriteThroughputMicroBursting (Maximum statistic, last 7 days)IopsThrottle Max > 0 OR ThroughputThrottle Max > 0 in the window → ⚠️ "EBS IOPS/throughput throttling detected. Increase provisioned Iops/Throughput on gp3 via UpdateDomainConfig."Iops/Throughput."DescribeDomainHealthTotalShards, ActiveShardsshards_per_node = TotalShards / InstanceCountDescribeDomain → EngineVersion):
heap_GiB ≈ min(32, instance_RAM_GiB / 2) (e.g., t3.medium.search 4 GiB RAM → ~2 GiB heap; r6g.xlarge.search 32 GiB RAM → ~16 GiB heap).
Instance RAM reference (GiB): t3.small=2 · t3.medium=4 · c6g.large=4 · m6g.large=8 ·
c6g.xlarge=8 · r6g.large=16 · m6g.xlarge=16 · c6g.2xlarge=16 · r6g.xlarge=32 ·
m6g.2xlarge=32 · r6g.2xlarge=64 (heap caps at 32 GiB). For unlisted types use the EC2
equivalent's memory — never guess silently; name the assumed RAM in the report.bp_limit = 25 × heap_GiB shards/node.shards_per_node > hard_limit → 🔴 "Exceeds hard shard limit for engine version {version}."shards_per_node > 4 × bp_limit → 🔴 "Shard density ({shards_per_node}/node) is more than 4× the best-practice limit ({bp_limit}/node for ~{heap_GiB} GiB heap). Severe heap/GC and recovery risk." (This tier is 🔴, not ⚠️ — do not downgrade.)shards_per_node > bp_limit → ⚠️ "Shard density ({shards_per_node}/node) exceeds best-practice threshold of 25 shards/GiB heap ({bp_limit}/node for ~{heap_GiB} GiB heap)."shards_per_node > 0.5 × bp_limit → ℹ️ "Moderate shard density. Monitor JVM heap pressure."shards_per_node <= 0.5 × bp_limit → ✅ PASS. (Verbatim: at or below half the best-practice limit the verdict is PASS, never ℹ️ — low density is not a finding.)DescribeDomainHealthActiveShards, TotalShardsActiveShards == TotalShards (all shards placed and serving)ActiveShards < TotalShards:
TotalShards - ActiveShards_cluster/allocation/explain for unassigned shard root cause."FreeStorageSpace (Minimum statistic, last 24h)free_pct = FreeStorageSpace_min_MB / (VolumeSize_GiB × 1024) × 100
(Dividing the per-node Minimum by VolumeSize × InstanceCount understates free space by a
factor of InstanceCount — this is wrong. Cross-check: the Sum statistic ÷ (VolumeSize ×
InstanceCount × 1024) should give approximately the same percentage.)free_pct < 10% → 🔴 "Free storage below 10%. High watermark may trigger read-only mode."free_pct < 20% → ⚠️ "Free storage below 20%. Approaching low watermark. Plan capacity increase."free_pct < 30% → ℹ️ "Storage utilization above 70%. Monitor trend."free_pct >= 30% → ✅ PASS.Note on percentiles: CloudWatch metrics for OpenSearch (
SearchLatency,IndexingLatency) report aggregate values, not per-request percentiles. We use the Maximum statistic over 1-minute periods as a "worst-case" proxy. True p50/p99 latency requires OpenSearch slow-log analysis or UltraWarm query insights, which is outside the scope of API-only checks.
Time range: Last 24 hours, 1-minute period granularity.
JVMMemoryPressure, Statistic: Maximum — the verdict MUST be computed from the
Maximum statistic. Never substitute Average (an idle cluster commonly shows Avg ~43% while Max
sustains ~74% — these produce different verdicts). Report Average only as context.CPUUtilization, Statistic: Average (sustained) and Maximum (spikes)SearchLatency, Statistic: Maximum (worst-case proxy)IndexingLatency, Statistic: Maximum (worst-case proxy)SearchRate (Sum), IndexingRate (Sum)4xx (Sum), 5xx (Sum)5xx > 0 → ⚠️ "{count} server errors (5xx) in last 24h. Indicates cluster-side failures (overload, circuit breaker, shard failures)."5xx == 0 AND 4xx > 0 → ℹ️ "{count} client errors (4xx). May indicate malformed queries, auth issues, or unauthenticated probes against a public endpoint." (Verbatim: with zero 5xx this is ℹ️ regardless of 4xx volume — never ⚠️. Note spikes/patterns as context only.)5xx == 0 AND 4xx == 0 → ✅ PASS.ClusterStatus.red (Maximum), ClusterStatus.yellow (Maximum)red == 1 datapoints → 🔴 "Cluster entered RED status {count} times in last 24h."yellow == 1 datapoints (and currently not yellow) → ℹ️ "Cluster experienced YELLOW status {count} times in last 24h (now recovered)."SEVERITY OVERRIDE PROHIBITION (apply exactly): The verdict tier for every check in this document comes ONLY from the Logic line that matches the observed values — NEVER from general security judgment. Several security checks below are deliberately rated ⚠️ (not 🔴) by owner decision: a single disabled control in isolation is a Warning; only the fully-compounded 4.4 case is Critical. Rendering 4.1, 4.2, 4.3, 4.5, or 4.6 as 🔴/CRITICAL is a SPEC VIOLATION even if it feels more correct — if your judgment disagrees with a Logic line, the Logic line wins. You may add ONE sentence of context inside the finding; you may never change the verdict tier, the emoji, or the severity word. This applies to the matrix, the executive summary, section headings, and prose equally.
DescribeDomain → EncryptionAtRestOptions.Enabledfalse → ⚠️ "Encryption at rest is disabled. Data on disk is unencrypted — non-compliant with most regulatory frameworks." (⚠️ by owner decision — do NOT escalate to 🔴.)DescribeDomain → NodeToNodeEncryptionOptions.Enabledfalse → ⚠️ "Node-to-node encryption disabled. Inter-node traffic is unencrypted." (⚠️ by owner decision — do NOT escalate to 🔴.)DescribeDomain → DomainEndpointOptions.EnforceHTTPSfalse → ⚠️ "HTTPS not enforced. Clients can connect over HTTP (plaintext)."TLSSecurityPolicy — if not Policy-Min-TLS-1-2-2019-07 or newer → ℹ️ "TLS policy allows older protocol versions."DescribeDomain → VPCOptionsVPCOptions is empty/null AND EnforceHTTPS == false AND AdvancedSecurityOptions.Enabled == false → 🔴 "Domain uses a public endpoint with BOTH transport and access controls disabled — fully compounded exposure. Migrate to VPC or close both gaps." (This is an AND — BOTH controls must be disabled. A single weak control does NOT escalate 4.4; it is already flagged ⚠️ by its own check: HTTPS by 4.3, FGAC by 4.6. The fully-compounded case is the ONLY security condition rated 🔴.)VPCOptions is empty/null (any other combination) → ⚠️ "Domain uses a public endpoint{; weak control: {name the single false one, if any}}. Consider VPC deployment for network isolation."DescribeDomain → AccessPolicies (JSON string)"*" with no IP condition → ⚠️ "Access policy allows unauthenticated access from any IP.""*" with IP condition → ℹ️ "Access restricted by IP/CIDR but no IAM authentication."DescribeDomain → AdvancedSecurityOptionsEnabled, InternalUserDatabaseEnabledEnabled == false → ⚠️ "Fine-grained access control disabled. No index-level or document-level permissions." (⚠️ by owner decision — do NOT escalate to 🔴.)InternalUserDatabaseEnabled == true → ℹ️ "Internal user database enabled. Consider SAML or IAM-based auth for production."DescribeDomain → ClusterConfig.WarmEnabled, ColdStorageOptions.EnabledWarmEnabled == false → ℹ️ "UltraWarm not enabled. If you have infrequently accessed indices (e.g., >30 days old), UltraWarm can reduce cost by ~80% for those indices."ColdStorageEnabled == false AND WarmEnabled == true → ℹ️ "Cold storage not enabled. For rarely accessed data, cold storage offers lowest-cost option."DescribeReservedInstancesFor EVERY finding rated 🔴, ⚠️, or ℹ️, the Prioritized Recommendations section MUST include that check's entry from references/remediation-reference.md (loaded in Execution Flow step 9 via read_skill_resource): copy the "Why it matters" line and "Resolve" steps verbatim (you may instantiate observed values, e.g. the actual master count), and include the "Dive deeper" link(s) EXACTLY as written. NEVER emit a documentation link that is not present in the Remediation Reference — do not construct, recall, or infer URLs from any other source. PASS/✅ checks get no remediation entry.
MANDATORY COVERAGE RULE: The report MUST evaluate and account for EVERY check in this document (1.1–1.4, 2.1–2.4, 3.1–3.7, 4.1–4.6, 5.1–5.3 — 24 checks total). No check may be silently omitted. ID FIDELITY: matrix rows MUST use these exact IDs with these exact meanings — never renumber, split, merge, or invent checks: 1.1 Cluster Status · 1.2 Node Count & AZ Balance · 1.3 Dedicated Masters · 1.4 Engine Version · 2.1 EBS Configuration · 2.2 Shard Count & Density · 2.3 Active vs Total Shards · 2.4 Free Storage · 3.1 JVM Memory Pressure · 3.2 CPU Utilization · 3.3 Search Latency · 3.4 Indexing Latency · 3.5 Search & Indexing Rate · 3.6 HTTP Errors · 3.7 Cluster Status History · 4.1 Encryption at Rest · 4.2 Node-to-Node Encryption · 4.3 HTTPS Enforcement · 4.4 Network Exposure · 4.5 Access Policy · 4.6 FGAC · 5.1 Instance Right-Sizing · 5.2 Storage Tiering · 5.3 Reserved Instances. (Splitting 1.1 into red/yellow/green rows, or using "1.4" for service-software patches, are documented failure modes — do not reproduce them.) CATEGORY ROLL-UP: each category's executive-summary status = the WORST finding in it: any 🔴 → Critical; else any ⚠️ → Warning; else all ✅/ℹ️ → Healthy. Never judgment-based. Apply the exact thresholds written in each check's Logic section verbatim — do not substitute your own thresholds or severity levels, and do not round a verdict up or down a tier based on judgment: the tier is determined solely by which Logic line matches. If a check cannot be evaluated (API error, no data), it MUST appear in the Check Coverage Matrix as SKIPPED with the reason. For every CloudWatch-based check (2.1 throttle sub-check, 2.4, 3.1–3.7, 5.1), the matrix's Observed Value column MUST name the statistic used (e.g., "Max=74.3% (Maximum stat)") — it must match the statistic named in that check's spec. Before finishing, count the matrix rows: if the count is not exactly 24, the report is incomplete — fix it before responding.
Structure the report as:
# OpenSearch Domain Health Assessment
## Domain: {domain_name}
**Region:** {region} | **Engine:** {engine_version} | **Assessed:** {timestamp}
## Summary
| Category | Status | Findings |
|----------|--------|----------|
| Cluster Health | 🟢/🟡/🔴 | {one-line summary} |
| Storage & Shards | 🟢/🟡/🔴 | {one-line summary} |
| Performance | 🟢/🟡/🔴 | {one-line summary} |
| Security | 🟢/🟡/🔴 | {one-line summary} |
| Cost Optimization | 🟢/🟡/⚪ | {one-line summary} |
## Detailed Findings
### 🔴 Critical (act now)
{findings}
### ⚠️ Warnings (plan action)
{findings}
### ℹ️ Informational
{findings}
### ✅ Passing Checks
{list of checks that passed}
## Recommendations (prioritized)
1. {highest priority}
2. ...
## Raw Data Reference
{key metrics and config values for verification}
## Check Coverage Matrix (REQUIRED — one row per check, all 24, in order)
| Check | Verdict | Observed value | Threshold applied |
|-------|---------|----------------|-------------------|
| 1.1 Cluster status | 🟢/⚠️/🔴 | {value} | {rule} |
| 1.2 Node/AZ balance | ... | ... | ... |
| ... (every check through 5.3 — SKIPPED rows must state the reason) |
DescribeDomainHealth returns Processing → domain is being modified. Report current state with caveat.AccessDeniedException → report which check was skipped and what permission is needed.