Terminal
Terminal/Skill
Queries Kubernetes cluster, pod, and node health in Dynatrace, including OOMKills, restarts, evictions, and resource waste.
By Dynatrace
Works with
npx skills add dynatrace/dynatrace-for-ai --skill dt-obs-kubernetesMonitor and analyze Kubernetes infrastructure using Dynatrace DQL. Query cluster resources, monitor workload health, analyze pod placement, optimize costs, and assess security posture.
references/cluster-inventory.md - Clusters,
namespaces, resource distributionreferences/labels-annotations.md - Parse
k8s.object, labels, annotationsreferences/pod-node-placement.md - Node
selectors, affinity, taints, HAWorkloads: K8S_DEPLOYMENT, K8S_STATEFULSET, K8S_DAEMONSET,
K8S_JOB, K8S_CRONJOB, K8S_HORIZONTALPODAUTOSCALER
Infrastructure: K8S_CLUSTER, K8S_NAMESPACE, K8S_NODE, K8S_POD
Configuration: K8S_SERVICE, K8S_CONFIGMAP, K8S_SECRET,
K8S_PERSISTENTVOLUMECLAIM, K8S_PERSISTENTVOLUME, K8S_INGRESS,
K8S_NETWORKPOLICY
Note: HPA, Job, CronJob, and configuration entity types are listed for inventory queries only. For HPA scaling analysis see
references/workload-health.md; for PVC/PV seereferences/pv-pvc.md; for Ingress/NetworkPolicy see the corresponding reference files.
smartscapeNodes - Query K8s entities (current state, no time range needed):
smartscapeNodes K8S_POD
| filter k8s.namespace.name == "production"
| fields k8s.cluster.name, k8s.pod.name
timeseries - Monitor metrics over time (always specify a from: range):
timeseries cpu = sum(dt.kubernetes.container.cpu_usage),
by: {k8s.pod.name, k8s.namespace.name},
from: now()-1h
| fieldsAdd avg_cpu = arrayAvg(cpu)
timeseriesreturns each metric as a time-bucket array. To collapse it to a scalar in a downstreamfieldsAdd, usearrayAvg(series)orarraySum(series)— callingavg()orsum()on a series field outside thetimeseries {}block is not supported.
fetch logs - Analyze log events:
fetch logs
| filter k8s.namespace.name == "production" and loglevel == "ERROR"
k8s.cluster.name, k8s.namespace.name, k8s.pod.name, k8s.node.namek8s.workload.name, k8s.workload.kind, k8s.container.namek8s.object - Full JSON configuration for deep inspectiontags[label] - Access labels and annotationsk8s.workload.kind values (always lowercase in DT — not Pascal-case as in the K8s API):
"deployment", "statefulset", "daemonset", "replicaset", "job", "cronjob"
k8s.object availability:
| Entity type | k8s.object available? |
|---|---|
K8S_POD | Yes |
K8S_NAMESPACE | Yes |
K8S_NODE | Yes |
Workload types (K8S_DEPLOYMENT, etc.) | Yes |
K8S_CLUSTER | No |
CPU: dt.kubernetes.container.cpu_usage, cpu_throttled, limits_cpu,
requests_cpu
Memory: dt.kubernetes.container.memory_working_set, limits_memory,
requests_memory
Operations: dt.kubernetes.container.restarts, oom_kills
Node: dt.kubernetes.node.pods_allocatable, cpu_allocatable,
memory_allocatable, dt.kubernetes.pods
Aggregation rule for requests/limits: Always use
sum()— neveravg()— when aggregatingrequests_cpu,limits_cpu,requests_memory, orlimits_memory.avg()returns a per-container average and silently undercounts multi-container pods and multi-replica workloads. For DaemonSets (one pod per node), this error is proportional to cluster size.
K8S_POD vs CONTAINER: these are different entity types in Dynatrace.
K8S_POD — K8s-native entities with k8s.object JSON, scheduling state, conditions, and K8s metrics. Use this skill.CONTAINER — Host-level container inventory (image, lifetime, host assignment). Use dt-obs-hosts skill instead.The smartscape edge is CONTAINER --(is_part_of)--> K8S_POD. To reach containers from a pod, traverse backward:
smartscapeNodes K8S_POD
| filter k8s.namespace.name == "<namespace>"
| traverse edgeTypes: {is_part_of}, targetTypes: {CONTAINER}, direction: backward, fieldsKeep: {id}
| fields k8s.cluster.name, k8s.namespace.name, k8s.pod.name, container.id=id
No direct smartscape edge exists between SERVICE and K8S_POD. The correlation key is the shared dimension k8s.workload.name. See Service → Pod Drill-Down in references/pod-debugging.md for the full two-step pattern.
List all clusters:
smartscapeNodes K8S_CLUSTER
| fields k8s.cluster.name, k8s.cluster.version, k8s.cluster.distribution
Check node capacity:
timeseries {
current_pods = avg(dt.kubernetes.pods),
max_pods = avg(dt.kubernetes.node.pods_allocatable)
}, by: {k8s.node.name, k8s.cluster.name},
from: now()-1h
| fieldsAdd pod_capacity_pct = (arrayAvg(current_pods) / arrayAvg(max_pods)) * 100
| filter pod_capacity_pct > 80
Identify pods in non-Running state:
smartscapeNodes K8S_POD
| parse k8s.object, "JSON:config"
| fieldsAdd phase = config[status][phase]
| filter not(in(phase, {"Running", "Succeeded"}))
| fields k8s.cluster.name, k8s.namespace.name, k8s.pod.name, phase
Succeededis a healthy terminal phase for completed Job pods — excluding it avoids false positives. Adjust if you specifically want to audit completed Jobs.
Pod-level — find over-provisioned pods (usage < 30%):
timeseries {
cpu_usage = sum(dt.kubernetes.container.cpu_usage),
cpu_requests = sum(dt.kubernetes.container.requests_cpu)
}, by: {k8s.pod.name, k8s.namespace.name, k8s.cluster.name},
from: now()-7d
| fieldsAdd usage_pct = (arrayAvg(cpu_usage) / arrayAvg(cpu_requests)) * 100
| filter usage_pct < 30 and arrayAvg(cpu_requests) > 0
Workload-level — aggregate across all replicas (required for correct DaemonSet accounting):
timeseries {
cpu_usage = sum(dt.kubernetes.container.cpu_usage),
cpu_requests = sum(dt.kubernetes.container.requests_cpu)
}, by: {k8s.workload.name, k8s.workload.kind, k8s.namespace.name, k8s.cluster.name},
from: now()-7d
| fieldsAdd
avg_usage = arrayAvg(cpu_usage),
avg_requests = arrayAvg(cpu_requests)
| fieldsAdd usage_pct = (avg_usage / avg_requests) * 100
| filter usage_pct < 30 and avg_requests > 0
| sort usage_pct asc
Use
sum()for bothcpu_usageandcpu_requestsat every aggregation level. A DaemonSet running on 50 nodes has 50× the per-pod request total;avg()would report 1/50th of the true reserved capacity.
Identify containers without limits:
smartscapeNodes K8S_POD
| parse k8s.object, "JSON:config"
| expand container = config[spec][containers]
| fieldsAdd
container_name = container[name],
cpu_limit = container[resources][limits][cpu],
memory_limit = container[resources][limits][memory]
| filter isNull(cpu_limit) or isNull(memory_limit)
Pod troubleshooting benefits from combining metrics (timeseries) with Kubernetes events (event stream) for a complete picture.
Find pods with OOMKills:
timeseries oom_kills = sum(dt.kubernetes.container.oom_kills),
by: {k8s.pod.name, k8s.namespace.name, k8s.cluster.name},
from: now()-1h
| filter arraySum(oom_kills) > 0
| fieldsAdd total_oom_kills = arraySum(oom_kills)
| sort total_oom_kills desc
Analyze pod restart patterns:
timeseries restarts = sum(dt.kubernetes.container.restarts),
by: {k8s.pod.name, k8s.namespace.name, k8s.cluster.name},
from: now()-1h
| fieldsAdd total_restarts = arraySum(restarts)
| filter total_restarts > 5
For operational events (pod restarts, OOM kills, evictions, scheduling failures), Kubernetes events provide richer context than metrics alone — including event reasons, messages, and timestamps.
When to use Kubernetes events over metrics:
Kubernetes events are available through the get-events-for-kubernetes-cluster
tool (a Dynatrace MCP tool). Call it with findAllK8Events: true to get events
for all clusters, or findAllK8Events: false with either clusterId
(k8s.cluster.uid) or kubernetesEntityId (dt.entity.kubernetes_cluster) to
scope to one cluster. Use history to set the lookback window (e.g. "1h",
"24h", "7d"; max "60d").
Prefer this tool when the user asks about OOM events, pod restarts,
evictions, or cluster-wide event history. Fall back to the fetch events DQL
pattern below if the tool is unavailable.
Important: distinguish event types when filtering results. Kubernetes events cover many categories. When the user asks about a specific event type, filter the results accordingly — do not report unrelated events:
| User Asks About | Relevant Event Reasons | NOT Related |
|---|---|---|
| Pod restarts | BackOff, CrashLoopBackOff, Killing | Readiness probe failures, CPU throttling |
| OOM events | OOMKilling, OOMKilled | Memory pressure warnings |
| Evictions | Evicted, Preempting | Node pressure |
| Scheduling failures | FailedScheduling, Unschedulable | Resource quotas |
For a complete answer, combine both approaches:
Pod restart and operational events can also be queried via DQL from the events table:
fetch events
| filter event.kind == "K8S_EVENT"
| filter event.type == "Warning"
| fields timestamp, k8s.cluster.name, k8s.namespace.name, k8s.pod.name,
event.reason, event.message
| sort timestamp desc
| limit 50
Filter for specific event reasons:
fetch events
| filter event.kind == "K8S_EVENT"
| filter in(event.reason, {"OOMKilling", "BackOff", "Evicted", "FailedScheduling"})
| fields timestamp, k8s.cluster.name, k8s.namespace.name, k8s.pod.name,
event.reason, event.message
| sort timestamp desc
Field names in fetch events: Use event.reason and event.message — not
dt.kubernetes.event.reason. The dt.kubernetes.* prefix is for timeseries metrics,
not the events table. Queries using the wrong prefix return zero results.
Identify privileged containers:
smartscapeNodes K8S_POD
| parse k8s.object, "JSON:config"
| expand container = config[spec][containers]
| fieldsAdd
container_name = container[name],
privileged = container[securityContext][privileged]
| filter privileged == true
Find containers running as root:
smartscapeNodes K8S_POD
| parse k8s.object, "JSON:config"
| expand container = config[spec][containers]
| fieldsAdd
container_name = container[name],
run_as_user = container[securityContext][runAsUser],
run_as_non_root = container[securityContext][runAsNonRoot]
| filter (isNull(run_as_user) or run_as_user == 0) and run_as_non_root != true
Verify pod distribution (HA compliance) for Deployments and StatefulSets:
smartscapeNodes K8S_POD
| filter in(k8s.workload.kind, {"deployment", "statefulset"})
| summarize pod_count = count(),
node_count = countDistinct(k8s.node.name),
by: {k8s.cluster.name, k8s.namespace.name, k8s.workload.name, k8s.workload.kind}
| fieldsAdd ha_compliant = node_count > 1
| filter pod_count >= 2 and not ha_compliant
DaemonSets are intentionally excluded — each pod runs on exactly one node by design, so single-node placement is not an HA violation for them.
Find active DAVIS problems affecting K8s entities:
fetch dt.davis.problems, from:now() - 2h
| filter not(dt.davis.is_duplicate) and event.status == "ACTIVE"
| filter iAny(startsWith(smartscape.affected_entities[][type], "K8S_"))
| fields display_id, event.name, event.category, affected_entity_ids = smartscape.affected_entities[][id]
smartscape.affected_entities is a record array; each record has id, type, and name. Use
[][id] to get the array of Smartscape IDs to look up the affected entity, or
[id] after expand smartscape.affected_entities. Without a preceding expand, [id] returns
null silently. A filter cannot take a bare iterative expression, so wrap it in iAny(...).
| User Question | Best Approach | Why |
|---|---|---|
| "Show me OOM events" | Events tool + metrics | Events give reasons/messages; metrics show trends |
| "Show me pod restart events" | Events tool + timeseries metrics | Events reveal the reason (BackOff, Killing, CrashLoopBackOff); dt.kubernetes.container.restarts metric gives the actual restart counts |
| "How many pod restarts?" | Timeseries metrics | Quantitative data over time |
| "What happened to my pods in the last 48h?" | Events tool | Operational event history with context |
| "Which pods are using the most CPU?" | Timeseries metrics | Resource utilization analysis |
| "List all clusters/namespaces" | smartscapeNodes | Entity discovery and inventory |
| "Are there scheduling failures?" | Events tool | Event reasons explain why |
| "Which workloads are over-provisioned?" | Timeseries metrics, workload-level | Must use sum() for requests; group by k8s.workload.name |
smartscapeNodes — no time range; always returns current entity state.timeseries — always add from: now()-<window>. Default window is
system-determined and often too short for trend analysis. Recommended defaults:
now()-1h for recent spikes, now()-24h for daily patterns, now()-7d for
weekly trends.fetch events — adding from: is recommended for reproducible results;
without it the system default window applies.fetch dt.davis.problems — use from: now()-2h for active problems; extend
to now()-7d to include recently closed problems.limit for explorationk8s.object unless needed - Parsing it increases query cost significantly| Problem | Cause | Solution |
|---|---|---|
| No pod data returned | Wrong entity type or missing cluster filter | Use K8S_POD (not POD); add k8s.cluster.name filter |
k8s.object parsing errors | Complex JSON structure | Use parse k8s.object, "JSON:config" then access nested fields |
| Pod network metrics unavailable | Not available in Grail | Use service mesh metrics or host-level network metrics |
| Large result sets | No time range or cluster filter | Add time range and filter by cluster/namespace early |
| Missing labels in output | Labels accessed incorrectly | Use tags[label_name] to access labels |
k8s.workload.kind filter returns no results | Value is Pascal-case (e.g. "Deployment") | Values are lowercase in DT: "deployment", "statefulset", "daemonset" |
Unavailable Metrics:
Query Considerations:
k8s.object field if not necessary→ references/cluster-inventory.md
k8s.object for detailed configuration inspection→ references/labels-annotations.md
→ references/pod-node-placement.md
→ references/pod-debugging.md
→ references/workload-health.md
→ references/pv-pvc.md
→ references/ingress.md
→ references/network-policies.md
dt.smartscape_source.id with K8S_ prefix filters)