Terminal
Terminal/Skill
Helps StarRocks operators find which process or workload is using CPU on a BE or FE host, then points to the right fix guide.
By StarRocks
Works with
npx skills add starrocks/starrocks-debug-skills --skill cpu-saturationInvestigation guide for BE or machine CPU saturation. This skill focuses on attribution: which process, which module, and which workload is consuming CPU. After attribution, follow the referenced deep-dive skill for root-cause analysis.
Before using the decision tree, establish what the CPU is doing and which process owns it.
| Metric | Meaning |
|---|---|
us (user) in top | User-space CPU — StarRocks computation, compression |
sy (system) in top | Kernel CPU — system calls, IO, interrupts; high sy with us suggests thread overhead |
wa (iowait) in top | CPU waiting for disk IO — not CPU saturation, disk bottleneck |
st (steal) in top | CPU stolen by hypervisor (VM environments only) — noisy neighbor |
| Load average | Running + runnable threads; >vCPU count indicates oversubscription |
How to retrieve:
# Snapshot CPU breakdown
top -b -n 1 | head -5
# Output: %Cpu(s): 85.2 us, 8.3 sy, 0.0 ni, 3.1 id, 2.1 wa, 0.0 hi, 1.3 si, 0.0 st
# Per-process CPU (10-second sample)
pidstat -u 2 5
# Load average vs CPU count
uptime
# Load average: 64.2, 58.1, 52.3 → compare to: nproc (e.g., 32 CPUs → overloaded)
nproc
| Metric | Meaning |
|---|---|
starrocks_be_cpu_task_query | CPU time consumed by query pipeline threads |
starrocks_be_cpu_task_load | CPU time consumed by import/flush threads |
starrocks_be_cpu_task_compaction | CPU time consumed by compaction threads |
starrocks_be_cpu_task_clone | CPU time consumed by tablet clone (rebalance) |
How to retrieve:
# Pull all BE CPU task metrics
curl -s "http://<be_host>:<be_http_port>/metrics" | grep "starrocks_be_cpu"
# Loop across all BEs to find the hotspot node
for be in <be1> <be2> <be3>; do
echo "=== $be ==="
curl -s "http://$be:8040/metrics" | grep "starrocks_be_cpu_task"
done
| Metric / Observable | Meaning |
|---|---|
java process CPU in top | FE planning storm or GC pressure |
GC CPU ratio from fe.gc.log | High GC CPU (>30% of elapsed) indicates heap pressure |
jstack — many RUNNABLE threads in optimizer code | Query planning CPU storm |
How to retrieve:
# FE Java process CPU
top -b -n 1 | grep java
# Per-thread breakdown for FE JVM
top -Hp $(pgrep -f 'StarRocksFE\|fe.pid') | head -20
# Convert thread ID (decimal) to hex → cross-reference with jstack
# FE thread dump
jstack $(cat fe/bin/fe.pid) > /tmp/fe_jstack.log
grep -c "RUNNABLE" /tmp/fe_jstack.log
# High RUNNABLE count relative to vCPUs → planning storm or GC
# GC pressure
grep "GC" fe/log/fe.gc.log | tail -20
| Hot function prefix | Likely module |
|---|---|
starrocks::pipeline:: | Query pipeline executor |
starrocks::vectorized:: | Vectorized evaluation (scan / aggregate / join) |
starrocks::MemTable:: | Import memtable flush |
starrocks::Compaction:: | Compaction |
starrocks::SegmentWriter:: | Import segment write |
starrocks::lake:: | Shared-data operations |
snappy:: / lz4:: / zstd:: | Compression (flush or compaction) |
tcmalloc:: / malloc | Memory allocator contention |
starrocks::SchemaChange | Schema change task |
starrocks::clone | Tablet clone (rebalance) |
How to retrieve:
# Live hot functions (needs root or perf permission)
perf top -p $(pidof starrocks_be) -d 5
# Format: [overhead%] [symbol_name] [DSO]
# Focus on the top 5 functions; the prefix identifies the module
# Quick thread snapshot without perf
pstack $(pidof starrocks_be) > /tmp/be_pstack_$(date +%s).log
grep -oE "starrocks::[A-Za-z:]+" /tmp/be_pstack_$(date +%s).log | sort | uniq -c | sort -rn | head -20
Key rules for interpretation:
wa (iowait) is NOT CPU saturation — it means CPU is idle waiting for disk. Diagnose with iostat -x 1, then follow compaction/SKILL.md or import/SKILL.md.starrocks_be_cpu_task_* metrics are available only in v3.4+. On older versions, use perf top or pidstat for function-level attribution.st > 0) rather than a StarRocks-internal cause.| Observation combination | Most likely root cause | First checkpoint |
|---|---|---|
us high; query latency up; import and compaction normal | Query-caused (Section 3a) | SHOW PROC '/current_queries'; perf top shows vectorized:: |
us high; import latency up; perf top shows MemTable:: / snappy:: | Import compression CPU (Section 3b) | starrocks_be_cpu_task_load metric |
us high; compaction score dropping; query slow; perf top shows Compaction:: | Compaction over-provisioned (Section 3c) | starrocks_be_cpu_task_compaction; reduce threads |
ALL metrics degrade simultaneously; st > 0 in VM or non-StarRocks process in top | Non-StarRocks process or hypervisor steal (Section 5) | ps aux --sort=-%cpu; top %st |
java process CPU high; FE queries stuck in Planning | FE planning storm or GC (Section 4) | jstack + top -Hp; fe.gc.log |
wa (iowait) high; us normal | IO bottleneck, not CPU | iostat -x 1; see compaction/SKILL.md |
sy high; many threads in top -Hp | Thread scheduling overhead | Reduce pipeline_dop or compaction thread count |
starrocks_be_cpu_task_clone elevated | Tablet rebalance consuming CPU | SHOW PROC '/cluster_balance'; throttle clone |
CPU saturation observed (load average high or top shows 100% on a node)
│
├─ Step 1: Which process owns the CPU?
│ → see Section 1
│
├─ Process = starrocks_be
│ └─ Step 2: Which BE module?
│ → see Section 2
│ ├─ Queries → Section 3a + query/SKILL.md
│ ├─ Import → Section 3b + import/SKILL.md
│ ├─ Compaction → Section 3c + compaction/SKILL.md
│ └─ Background task / unknown → Section 3d
│
├─ Process = java (FE)
│ → FE Full GC or planning storm → Section 4 + node/SKILL.md
│
└─ Process = non-StarRocks (backup agent, kernel, etc.)
→ Section 5: OS-level attribution
# Snapshot: top CPU consumers by process
top -b -n 1 | head -20
# More detail: per-thread breakdown for a single process
top -Hp $(pidof starrocks_be) | head -30
# Or use pidstat for sustained view
pidstat -u 2 10
| Observation | Next Step |
|---|---|
starrocks_be is top consumer | Section 2 |
java (FE) is top consumer | Section 4 |
| Other process (mysqld, rsync, python...) | Section 5 |
kworker / ksoftirqd / kernel threads | Network interrupt or disk IRQ — check sar -I ALL, iostat |
CPU wa (iowait) is high, not us | IO bottleneck, not CPU — see compaction/SKILL.md or import/SKILL.md |
-- Any active long-running queries?
SHOW PROC '/current_queries';
-- Look for QueryTime > 30s or MemUsageBytes > 1GB
-- Any active imports?
SELECT LABEL, STATE, PROGRESS, TYPE, LOAD_START_TIME
FROM information_schema.loads
WHERE STATE IN ('LOADING', 'PREPARED', 'PREPARED')
ORDER BY LOAD_START_TIME ASC;
# CPU breakdown by task type (query / load / compaction / clone / schema change)
curl -s http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep "starrocks_be_cpu"
Key metrics:
starrocks_be_cpu_task_query — query pipeline CPUstarrocks_be_cpu_task_load — import write CPUstarrocks_be_cpu_task_compaction — compaction CPUstarrocks_be_cpu_task_clone — tablet clone CPU# Live hot functions (needs root or perf permission)
perf top -p $(pidof starrocks_be) -d 5
# Or: pstack for a quick thread snapshot without perf
pstack $(pidof starrocks_be) > /tmp/be_pstack_$(date +%s).log
Confirm signals:
starrocks_be_cpu_task_query is the dominant rising metricperf top shows starrocks::pipeline:: or starrocks::vectorized:: as hot functionsSHOW PROC '/current_queries' shows active queries with high ScanRows or MemUsageBytesMechanism: Query pipeline threads consume CPU proportional to data volume and operator complexity. Full-table scans (no predicate pushdown), large hash joins (hash table exceeds CPU cache), and high-cardinality aggregations all create sustained CPU pressure. Without resource group limits, a single large query can saturate all cores.
# Find queries with highest scan rows in audit log (last 1 hour)
python3 scripts/analyze_logs.py "$(date -d '1 hour ago' '+%Y-%m-%d %H:%M:%S')" "$(date '+%Y-%m-%d %H:%M:%S')" "ScanRows" 5 fe.audit.log
# Or find by query execution time
grep 'QueryTime' fe.audit.log | awk -F'QueryTime=' '{print $2}' | cut -d'|' -f1 | sort -rn | head -10
SET is_report_success = true;
-- Re-run the query, then check profile:
-- http://<fe_ip>:8030 -> Queries -> Profile
Common query CPU causes:
| Symptom in Profile | Cause | Fix |
|---|---|---|
HASH_JOIN_NODE dominates CPU | Large hash table probe | Force [SHUFFLE]; collect statistics |
AGGREGATE_NODE dominates | High-cardinality aggregation | Check streaming pre-agg; add pre-filter |
OLAP_SCAN_NODE CPU high, BytesRead huge | Full scan, no pushdown | Fix predicate type mismatch; add index |
| Many pipeline threads near 100% | Too many concurrent heavy queries | Add resource group; set query_mem_limit |
→ Deep-dive: query/SKILL.md
Confirm signals:
starrocks_be_cpu_task_load is the dominant elevated metricperf top shows starrocks::MemTable::, starrocks::SegmentWriter::, snappy::, or zstd:: as hot functionsinformation_schema.loadsMechanism: Import flush compresses memtable data before writing to disk segments. Compression (especially zstd) is CPU-intensive. With many concurrent stream loads, flush threads saturate available cores. Query pipeline threads then compete for CPU time slices, causing query latency to rise even without any change in query patterns.
# Check import thread CPU contribution
curl -s http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep "starrocks_be_cpu_task_load"
# Check active imports
curl -s http://<BE_IP>:<BE_HTTP_PORT>/api/running_load | python3 -m json.tool 2>/dev/null
Common import CPU causes:
| Observation | Cause | Fix |
|---|---|---|
| Many concurrent stream loads | Memtable flush compression saturating cores | Reduce concurrency; use lz4 instead of zstd for memtable |
flush_thread_num_per_store threads all busy | Flush threads competing with query | Balance flush_thread_num_per_store vs query concurrency |
perf top shows snappy:: or zstd:: hot | Compression CPU during flush | Tune compression_type for the table |
| Import CPU high, query CPU also high | Both competing for CPU | Use resource groups to cap import CPU |
→ Deep-dive: import/SKILL.md
Confirm signals:
starrocks_be_cpu_task_compaction is the dominant elevated metricperf top shows starrocks::Compaction:: or starrocks::SegmentWriter:: as hot functionsMechanism: Compaction threads merge rowsets into larger segments, which involves decompressing source segments, merging sorted data, and recompressing the output. With cumulative_compaction_num_threads_per_disk set too high (e.g., to clear a historical backlog), compaction monopolizes CPU cores, reducing the time slices available to query pipeline threads.
# Check compaction CPU contribution
curl -s http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep "starrocks_be_cpu_task_compaction"
# Check compaction activity
curl -s http://<BE_IP>:<BE_HTTP_PORT>/api/compaction/show | python3 -m json.tool 2>/dev/null
# Check compaction score
SELECT tablet_id, num_rowset, num_segment, data_size
FROM information_schema.be_tablets
ORDER BY num_rowset DESC LIMIT 10;
Common compaction CPU causes:
| Observation | Cause | Fix |
|---|---|---|
| Compaction CPU consistently >30% of BE | Too many compaction threads | Reduce cumulative_compaction_num_threads_per_disk |
| CPU spike during compaction of large tablets | Base compaction of multi-GB rowset | Schedule off-peak; throttle base compaction IO |
| Compaction CPU high + query slow | CPU contention | Use resource groups or cgroup to cap compaction CPU |
# Temporarily reduce compaction threads (dynamic — no restart)
curl -XPOST "http://<BE_IP>:<BE_HTTP_PORT>/api/update_config?cumulative_compaction_num_threads_per_disk=1"
# Or via SQL (v3.2+, applies to all BEs)
# UPDATE information_schema.be_configs SET value = 1
# WHERE name = 'cumulative_compaction_num_threads_per_disk';
→ Deep-dive: compaction/SKILL.md
Confirm signals:
perf top shows hot functions that don't match query/import/compaction prefixesSHOW PROC '/current_queries' shows no long-running queriesinformation_schema.loads shows no active importsMechanism: Schema changes, tablet clone (rebalance), and memory allocator contention (tcmalloc) can cause sustained CPU usage without appearing in the query or import workload metrics. Clone operations copy tablet data between BEs during rebalancing; schema changes rebuild column segments. Both generate significant IO and CPU workload.
If perf top shows hot functions that don't match query/import/compaction prefixes:
# Check schema change tasks
SHOW ALTER TABLE COLUMN;
# Check clone tasks
SHOW PROC '/cluster_balance';
# Check routine load tasks
SHOW ROUTINE LOAD;
# Memory allocator contention (tcmalloc)
curl -s http://<BE_IP>:<BE_HTTP_PORT>/memz | head -30
| Hot function | Likely task | Action |
|---|---|---|
starrocks::SchemaChange | Concurrent schema change | Pause or serialize schema changes |
starrocks::clone | Tablet clone (rebalance) | Throttle clone tasks |
tcmalloc / malloc | Memory allocator contention (many threads) | Check brpc_num_threads; reduce thread count |
starrocks::DictColumn / GlobalDictCache | Low-cardinality dict build | Usually self-resolving; monitor |
Confirm signals:
top shows java process (FE) in top CPU consumersjstack shows many RUNNABLE threads in CBO optimizer or statistics codefe.gc.log shows frequent Full GC eventsMechanism: FE Java process CPU saturation has two distinct causes: (1) query planning storm where many concurrent complex queries overwhelm CBO planner threads, and (2) Full GC where Java's garbage collector consumes CPU cleaning heap. Both appear as high java process CPU but have different jstack signatures.
# Identify hot Java threads
top -Hp $(pgrep -f 'StarRocksFE\|fe.pid') | head -20
# Convert thread ID to hex, then cross-reference jstack
jstack $(cat fe/bin/fe.pid) > /tmp/fe_jstack.log
# Look for threads in RUNNABLE state consuming CPU
# Check if GC is the cause
grep "GC" fe/log/fe.gc.log | tail -20
| Observation | Cause | Fix |
|---|---|---|
Many RUNNABLE threads in CBO/optimizer code | Query plan storm (high QPS of complex queries) | Add plan cache; reduce concurrent complex queries |
GC CPU > 30% (%cpu in GC log) | Heap pressure / Full GC | Increase FE -Xmx; check for memory leaks |
StatisticsCollectJob threads hot | Statistics collection concurrent with traffic | Reduce statistic_collect_concurrency; schedule off-peak |
→ Deep-dive: node/SKILL.md — Section 3 (FE GC) / Section 4 (thread pool)
Confirm signals:
ps aux --sort=-%cpu shows a non-StarRocks process in the top CPU consumersst (steal) > 0 in top (VM environments)Mechanism: Co-located processes (backup agents, Spark jobs, monitoring scanners) compete for CPU with StarRocks. In VM environments, hypervisor steal (st) directly reduces the CPU available to StarRocks without appearing as a StarRocks-internal issue. The distinguishing signal is simultaneous degradation of ALL metrics — if only one metric category is affected, the cause is StarRocks-internal.
# Identify the process
ps aux --sort=-%cpu | head -20
# Check if it's a StarRocks-adjacent tool
ls /proc/$(pgrep <process_name>)/exe -l # What binary is it?
# Check cgroup CPU usage if StarRocks is in a cgroup
cat /sys/fs/cgroup/cpu/starrocks/cpu.stat 2>/dev/null
Common non-StarRocks culprits:
| Process | Cause | Fix |
|---|---|---|
rsync / tar / backup agent | Scheduled backup competing with queries | Schedule backups during off-peak; add ionice / cpulimit |
python3 / spark / other analytics | Co-located workload on same host | Isolate StarRocks nodes; use dedicated hosts |
kcompactd / kswapd | Memory pressure causing kernel to reclaim pages | Reduce memory overcommit; add more RAM |
irqbalance / high si (soft IRQ) | Network interrupt handling at high packet rate | Enable RSS; tune NIC queues; spread IRQ affinity |
dockerd / container overhead | Container overhead or overlay fs writes | Use dedicated bare-metal or tune cgroup limits |
Mitigation: Use cgroup or systemd resource limits to cap non-StarRocks CPU:
# Cap a process to 50% of 2 cores
systemctl set-property <service> CPUQuota=100%
# Or use cpulimit for an ad-hoc process
cpulimit -p $(pgrep rsync) -l 20
| Parameter | Default | Description | Dynamic |
|---|---|---|---|
cumulative_compaction_num_threads_per_disk | 1 | Cumulative compaction threads per disk | No (restart), but curl update_config works at runtime |
base_compaction_num_threads_per_disk | 1 | Base compaction threads per disk | No (restart) |
max_compaction_concurrency | -1 (unlimited) | Global compaction thread cap | Yes (v2.5+ curl or SQL) |
update_compaction_num_threads_per_disk | 1 | PK update compaction threads per disk | Yes (increase only) |
flush_thread_num_per_store | 2 | Memtable flush threads per storage path | No (restart) |
brpc_num_threads | -1 (auto) | BRPC worker thread count | No (restart) |
pipeline_dop | 0 (auto=vCPUs/2) | Query pipeline parallelism; set to 1 for high-QPS OLTP | Yes (SET GLOBAL) |
statistic_collect_concurrency | 3 | Concurrent statistics collection jobs (FE) | Yes (FE ADMIN SET) |
enable_statistic_collect | true | Enable background statistics collection | Yes (FE ADMIN SET) |
clone_worker_count | 3 | Tablet clone (rebalance) thread count | No (restart) |
# Per BE (v2.5+): update compaction threads dynamically
curl -XPOST "http://<be_ip>:<be_http_port>/api/update_config?cumulative_compaction_num_threads_per_disk=1"
curl -XPOST "http://<be_ip>:<be_http_port>/api/update_config?max_compaction_concurrency=4"
# Verify
curl "http://<be_ip>:<be_http_port>/varz" | grep compaction_num_threads
-- v3.2+: apply to all BEs at once
UPDATE information_schema.be_configs SET value = 4
WHERE name = 'max_compaction_concurrency';
-- Verify
SELECT * FROM information_schema.be_configs WHERE name = 'max_compaction_concurrency';
Static parameters (flush_thread_num_per_store, clone_worker_count, brpc_num_threads) require be.conf change and BE restart.
Large full-table-scan or multi-join query submitted without resource group
↓ observable: SHOW PROC '/current_queries' shows query with ScanRows >> expected; no LIMIT
Pipeline executor threads saturate available CPU cores
↓ observable: top -Hp <be_pid> shows many threads at 100%; perf top shows vectorized:: hot
Other queries and imports compete for CPU; scheduling latency rises
↓ observable: concurrent simple queries start taking seconds; import write latency rises
New queries slow → users retry → more load submitted → CPU stays saturated (feedback loop)
↓ observable: SHOW PROC '/current_queries' count growing; no queries completing
Cluster appears frozen; all operations slow; restarting query does not help until root cause is killed.
Trigger conditions: No resource group limits; ad-hoc analytics on production cluster; missing WHERE filter causes full-table scan.
Break point: KILL QUERY <query_id> for the offending query; set query_mem_limit and resource groups to prevent recurrence.
High-throughput import with zstd compression on memtable flush
↓ observable: perf top shows snappy:: or zstd:: hot; starrocks_be_cpu_task_load elevated
Flush thread pool saturates CPU; query pipeline gets fewer CPU slices
↓ observable: query latency increases; SHOW PROC '/current_queries' shows elevated QueryTime
Queries appear slow despite no change in query patterns
↓ observable: profile shows similar execution plan but higher wall time; CPU wa low (not IO)
Queries degrade without any query-side change; root cause is import workload added to the same BEs.
Trigger conditions: Large-batch import with high compression ratio (zstd/snappy) while serving online queries; flush_thread_num_per_store too high relative to available cores.
Break point: Reduce flush_thread_num_per_store; switch to lz4 compression for memtable (compression_type = "LZ4" in table properties); add resource group CPU cap for import tasks.
Compaction thread count set high to clear a historical backlog
↓ observable: starrocks_be_cpu_task_compaction elevated; Grafana shows compaction bytes/sec high
Compaction threads consume a sustained fraction of CPU cores
↓ observable: top shows BE near 100%; perf top shows Compaction:: or SegmentWriter:: functions
Query pipeline gets fewer CPU time slices; scan and join throughput drops
↓ observable: query latency rises proportionally to compaction CPU share; profile shows longer operator times
Compaction-query CPU contention sustained until backlog cleared
↓ observable: query latency normalizes once compaction score drops below threshold
Queries slow for hours after a compaction tuning change; operators suspect query regression but root cause is compaction.
Trigger conditions: cumulative_compaction_num_threads_per_disk increased to catch up on a backlog without considering query workload; no CPU cap on compaction tasks.
Break point: curl -XPOST http://<BE>/api/update_config?cumulative_compaction_num_threads_per_disk=1 to reduce compaction pressure dynamically without restart.
Scheduled backup / monitoring agent launches and consumes CPU (no resource limit set)
↓ observable: top shows non-StarRocks process in top 3 by CPU; ps aux --sort=-%cpu confirms
StarRocks BE and FE receive fewer CPU cycles from OS scheduler
↓ observable: CPU steal (%st in top) > 0 if in VM; or BE CPU user% drops while load average stays high
Queries slow; imports slow; compaction falls behind
↓ observable: ALL metrics degrade simultaneously — if only queries slow, suspect query cause
Non-StarRocks process finishes or is killed → all StarRocks metrics recover
↓ observable: latency returns to baseline without any StarRocks config change
Simultaneous degradation of all metrics (query + import + compaction) at the same time is the key signal for an OS-level cause rather than a StarRocks-level cause.
Trigger conditions: Co-located workloads on StarRocks nodes; no cgroup or systemd resource limits for non-StarRocks processes; scheduled jobs (backup, log rotation, monitoring scan) running during peak hours.
Break point: kill or cpulimit the offending process; long-term: enforce cgroup CPU quotas for non-StarRocks processes, or move StarRocks to dedicated hosts.
| Observation | Most Likely Cause | First Check |
|---|---|---|
| CPU high + only query latency up | Query-caused | SHOW PROC '/current_queries'; perf top shows vectorized:: |
| CPU high + import and query both slow | Import-caused (compression CPU) | starrocks_be_cpu_task_load metric; perf top shows MemTable:: / snappy:: |
| CPU high + query slow + compaction score dropping | Compaction over-provisioned | starrocks_be_cpu_task_compaction; reduce compaction threads |
| CPU high + ALL metrics degrade simultaneously | Non-StarRocks process | ps aux --sort=-%cpu; top %st or %si |
| CPU high on FE (java process) | Plan storm or Full GC | jstack + top -Hp; fe.gc.log |
High iowait (wa in top), not us | IO bottleneck, not CPU | iostat -x 1; see compaction/SKILL.md or import/SKILL.md |
case-014-scan-skew — query-caused CPU spike from scan skew and MERGE-heavy tabletscase-008-be-oom — BE OOM triggered by CPU-saturating compaction during large query burst