Terminal
Terminal/Skill
Helps StarRocks cluster operators fix tablet scheduler problems, such as missing replicas, failing clone tasks, skewed disk usage, or a stuck BE decommission.
By StarRocks
Works with
npx skills add starrocks/starrocks-debug-skills --skill balanceInvestigation guide for tablet scheduling, cluster balance, replica repair, and decommission issues.
Five root causes account for the vast majority of cases:
Before using any metrics, understand the two-layer structure:
| Metric name | Meaning |
|---|---|
starrocks_fe_scheduled_tablet_num | Total tablets currently being scheduled (pending + running) |
starrocks_fe_scheduled_pending_tablet_num{type="BALANCE"} | Tablets waiting for a balance clone slot |
starrocks_fe_scheduled_pending_tablet_num{type="REPAIR"} | Tablets waiting for a repair clone slot |
starrocks_fe_scheduled_running_tablet_num{type="BALANCE"} | Balance clone tasks actively running |
starrocks_fe_scheduled_running_tablet_num{type="REPAIR"} | Repair clone tasks actively running |
starrocks_fe_clone_task_total | Cumulative clone tasks submitted |
starrocks_fe_clone_task_success | Cumulative clone tasks completed successfully |
starrocks_fe_clone_task_copy_duration_ms{type="INTER_NODE"} | Clone task copy duration for inter-node balance |
starrocks_fe_clone_task_copy_bytes{type="INTER_NODE"} | Bytes copied per clone task |
| Metric name | Meaning |
|---|---|
starrocks_be_clone_task_copy_bytes{type="INTER_NODE"} | Bytes transferred per clone on this BE |
starrocks_be_clone_task_copy_duration_ms{type="INTER_NODE"} | Clone duration on this BE |
starrocks_be_engine_requests_total{status="failed",type="clone"} | Failed clone requests on this BE |
Key SHOW PROC "/statistic" fields:
| Field | Meaning |
|---|---|
UnhealthyTabletNum | Tablets with missing replicas, incomplete versions, colocate mismatch, or location mismatch |
InconsistentTabletNum | Tablets with data inconsistency across replicas |
CloningTabletNum | Tablets currently being cloned |
ErrorStateTabletNum | Tablets in error state (PK tables only) |
Key SHOW PROC "/cluster_balance" sub-paths:
| Sub-path | Meaning |
|---|---|
cluster_load_stat | Disk usage and tablet count per BE by medium type |
working_slots | Available/total clone slots per BE path |
sched_stat | Scheduler statistics snapshot (updated every 20s) |
priority_repair | Tablets needing priority repair |
pending_tablets | Tablets waiting for scheduling |
running_tablets | Tablets currently being cloned |
history_tablets | Recently completed tablets (max 1000, includes ErrMsg) |
balance_stat | Balance overview by balance type (3.5+) |
Option 1 — curl the FE HTTP port (immediate, no Prometheus needed)
# Pull all balance-related metrics from FE
curl -s "http://<fe_ip>:8030/metrics" | grep -E "scheduled_tablet|clone_task"
# Pending/running breakdown by type
curl -s "http://<fe_ip>:8030/metrics" | grep "scheduled_pending_tablet_num\|scheduled_running_tablet_num"
Option 2 — curl the BE HTTP port (per-node clone health)
# Clone failures on a specific BE
curl -s "http://<be_ip>:8040/metrics" | grep -E "clone_task|engine_requests.*clone"
# Loop across all BEs
for be in 10.0.0.1 10.0.0.2 10.0.0.3; do
echo "=== $be ==="
curl -s "http://$be:8040/metrics" | grep "engine_requests_total.*clone"
done
Option 3 — SQL (primary method for balance diagnosis)
-- Cluster-wide unhealthy tablet summary
SHOW PROC "/statistic";
-- Unhealthy tablets in a specific database
SHOW PROC "/statistic/<db_id>";
-- Balance overview
SHOW PROC "/cluster_balance";
SHOW PROC "/cluster_balance/sched_stat";
SHOW PROC "/cluster_balance/working_slots";
SHOW PROC "/cluster_balance/pending_tablets";
SHOW PROC "/cluster_balance/running_tablets";
SHOW PROC "/cluster_balance/history_tablets";
-- Disk usage per BE (v3.5+)
SHOW PROC "/cluster_balance/balance_stat";
SHOW PROC "/cluster_balance/cluster_load_stat/HDD";
SHOW PROC "/cluster_balance/cluster_load_stat/SSD";
-- Clone task history for a specific table
SELECT * FROM information_schema.fe_tablet_schedules
WHERE table_id = <table_id>
ORDER BY create_time DESC LIMIT 50;
-- Find partitions with imbalanced tablet distribution
SELECT partition_id, MAX(cnt) AS max_cnt, MIN(cnt) AS min_cnt,
MAX(cnt) - MIN(cnt) AS diff
FROM (
SELECT partition_id, be_id, COUNT(*) AS cnt
FROM information_schema.be_tablets
GROUP BY partition_id, be_id
) t
GROUP BY partition_id
HAVING diff > 1
ORDER BY diff DESC LIMIT 20;
-- Tables with replication_num > 2 (relevant for decommission)
SELECT db_name, table_name, partition_name, replication_num
FROM information_schema.partitions_meta
WHERE replication_num > 2;
Default HTTP ports: BE = 8040, FE = 8030. Verify with SHOW BACKENDS or be.conf / fe.conf.
Key rules for interpretation:
UnhealthyTabletNum > 0 is the primary alert signal. Drill into SHOW PROC "/statistic/<db_id>" to identify which databases are affected.CloningTabletNum > 0 with UnhealthyTabletNum decreasing means repair is in progress — expected behavior. Only investigate if CloningTabletNum is stuck.starrocks_fe_clone_task_success should be rising. If clone_task_total rises but clone_task_success does not, clone tasks are failing — check history_tablets.ErrMsg.SHOW BACKENDS: DataUsedCapacity) triggers disk balance clones automatically.working_slots all zero or all full: zero means no work; all full means scheduler is at capacity — check if tablets are actually moving by watching CloningTabletNum.| Metric / Phenomenon | Healthy | Needs Attention | Act Immediately |
|---|---|---|---|
UnhealthyTabletNum | 0 | 1–100 (and decreasing) | > 100 sustained or not decreasing |
CloningTabletNum | 0 (no activity) or rising then falling | Stuck at same value > 10 min | Stuck with ErrMsg in history_tablets |
| Disk usage difference between BEs | < 10% | 10–20% | > 20% (balance triggered automatically) |
| Tablet count difference between BEs (same partition) | 0–1 | 2–5 | > 5 (imbalanced scans) |
clone_task_total rate vs clone_task_success rate | Equal | Success rate < 90% | Success rate < 50% |
working_slots fill rate | 20–80% | > 95% (saturated) | 100% with tasks still pending > 30 min |
-- Run on Leader FE
SHOW PROC "/statistic";
Key interpretation:
UnhealthyTabletNum = 0 across all DBs → cluster is healthy, no action neededUnhealthyTabletNum > 0 → proceed to Step 1.2CloningTabletNum > 0 → repair/balance is in progress; check if it's making progressSHOW PROC "/cluster_balance/sched_stat";
Check the Increase column to see if counters are advancing:
| Observation | Points toward |
|---|---|
num of balance scheduled rising, UnhealthyTabletNum shrinking | Balance is working; just slow — Cause A |
num of clone task failed rising | Clone tasks failing — Cause D |
num of replica missing error high but no clone tasks starting | Scheduler paused or slots exhausted |
num of colocate replica mismatch non-zero and not decreasing | Cause E — colocate group unstable |
| Large pending_tablets queue, few running_tablets | Cause A — throughput too low |
running_tablets saturated, queries/writes slow | Cause B — balance too aggressive |
-- Drill into pending and running queues
SHOW PROC "/cluster_balance/pending_tablets";
SHOW PROC "/cluster_balance/running_tablets";
-- Check slot utilization per path
SHOW PROC "/cluster_balance/working_slots";
-- Clone task history with error messages
SHOW PROC "/cluster_balance/history_tablets";
-- Or query system table for specific table
SELECT tablet_id, type, state, schedule_reason, priority,
failed_schedule_count, failed_running_count, msg
FROM information_schema.fe_tablet_schedules
WHERE table_id = <table_id>
ORDER BY create_time DESC LIMIT 20;
Confirm all match:
pending_tablets queue large (many tablets waiting)working_slots shows low utilization (slots available but few tasks running)UnhealthyTabletNum not decreasing over 10+ minuteshistory_tabletsMechanism: The tablet scheduler submits clone tasks into slots — tablet_sched_slot_num_per_path per BE path. If slots are configured conservatively (default 8) and disks are fast, the scheduler underutilizes available IO, leaving balance tasks in the pending queue far longer than necessary. Each clone copies a full tablet replica from source to destination.
→ Go to Phase 3, Cause A
Confirm all match:
working_slots fill rate near 100%running_tablets at or near tablet_sched_max_scheduling_tablets limitiostat on affected BEs shows disk IO > 80%Mechanism: Clone tasks perform sequential read on source and sequential write on destination. When many clones run concurrently, they compete with query scan IO and load write IO for disk bandwidth. The scheduler has no IO-awareness — it will fill all slots regardless of disk pressure.
→ Go to Phase 3, Cause B
Confirm all match:
backend <id> lefts N replicas to decommissionSHOW BACKENDS shows BE in DECOMMISSIONING state for > 30 minuteshistory_tablets shows REPLICA_RELOCATING tasks failingMechanism: Decommission moves all replicas off the BE. It can block when: (1) remaining BEs cannot satisfy replication factor (e.g., 3 BEs remaining with 3-replica tables); (2) tablets exist in FE recycle bin holding replica references; (3) statistics system tables have fixed 3-replica config.
→ Go to Phase 3, Cause C
Confirm all match:
history_tablets.ErrMsg contains unable to find dest path or similarfailed_running_count > 3 for specific tablets in fe_tablet_schedulesstarrocks_be_engine_requests_total{status="failed",type="clone"} rising on specific BEMechanism: Clone tasks fail when the destination BE has no path with sufficient free space, the destination BE is down or unreachable, or the source tablet is in error state. Repeated failures cause the tablet to cycle in the pending queue indefinitely, keeping UnhealthyTabletNum elevated.
→ Go to Phase 3, Cause D
Confirm all match:
SHOW PROC "/colocation_group" shows group in Unstable statusnum of colocate replica mismatch in sched_stat is non-zero and not clearingMechanism: Colocate groups require tablets to be co-located on specific BEs as defined by the bucket sequence. When a BE fails or is added, the group is marked Unstable and the scheduler starts relocating tablets to restore the required co-location pattern. Until the group is Stable again, colocate joins degrade to regular joins.
→ Go to Phase 3, Cause E
Step A-1: Increase scheduler throughput (dynamic, no restart)
-- FE: more slots per path (default 8 → 16)
ADMIN SET FRONTEND CONFIG ("tablet_sched_slot_num_per_path" = "16");
-- FE: allow more total concurrent scheduling tablets (default 1000 → 2000)
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "2000");
-- FE: allow more balance tasks (default 500 → 1000)
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "1000");
Step A-2: Increase BE clone parallelism (requires be.conf edit + restart per BE)
# be.conf
parallel_clone_task_per_path = 16 # default 8
Step A-3: Trigger priority repair for specific tables
-- Set high-priority repair (faster scheduling)
ADMIN REPAIR TABLE <table_name>;
ADMIN REPAIR TABLE <table_name> PARTITION (p1, p2);
Step A-4: Monitor progress
-- Watch CloningTabletNum and UnhealthyTabletNum
SHOW PROC "/statistic";
-- Clone task success rate
SELECT * FROM information_schema.fe_tablet_schedules
WHERE state = 'FINISHED'
ORDER BY finish_time DESC LIMIT 20;
Step B-1: Reduce concurrency immediately
-- Reduce slots per path (default 8 → 4)
ADMIN SET FRONTEND CONFIG ("tablet_sched_slot_num_per_path" = "4");
-- Cap balance tasks (default 500 → 200)
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "200");
Step B-2: Clear the queue if needed
-- Remove all pending and running tasks
ALTER SYSTEM CLEAN TABLET SCHEDULER QUEUE;
Step B-3: Optionally disable balance temporarily
-- Disable non-colocate balance (repair still runs)
ADMIN SET FRONTEND CONFIG ("tablet_sched_disable_balance" = "true");
-- Re-enable after import window
ADMIN SET FRONTEND CONFIG ("tablet_sched_disable_balance" = "false");
Step B-4: Throttle clone bandwidth (if BE supports it)
-- Limit clone speed per task in MB/s
ADMIN SET FRONTEND CONFIG ("max_clone_task_speed_mbps" = "50");
Step C-1: Diagnose the blocker
# Check FE leader log for remaining replicas
grep "lefts.*replicas to decommission" fe.log | tail -20
-- Find tables with replication_num > 2 (may block decommission if only 3 BEs remain)
SELECT db_name, table_name, partition_name, replication_num
FROM information_schema.partitions_meta
WHERE replication_num > 2;
-- Check history_tablets for REPLICA_RELOCATING failures
SHOW PROC "/cluster_balance/history_tablets";
Step C-2: Handle recycle bin tablets
-- If tablets show null db/table/partition — they are in the recycle bin
-- Shorten trash expiry to accelerate cleanup
ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second" = "60");
Step C-3: Handle 3-replica tables with minimum BEs
-- Option 1: Temporarily reduce replication factor
ALTER TABLE <table_name> SET ("replication_num" = "2");
-- After decommission completes:
ALTER TABLE <table_name> SET ("replication_num" = "3");
-- Option 2: Add another BE to the cluster first
Step C-4: Verify decommission is progressing
-- Remaining tablets should decrease
SHOW BACKENDS;
-- Check StorageUsedCapacity on decommissioning BE — should approach 0
Step D-1: Identify the error
-- Check error messages
SELECT tablet_id, type, failed_running_count, msg
FROM information_schema.fe_tablet_schedules
WHERE failed_running_count > 2
ORDER BY failed_running_count DESC LIMIT 20;
-- Check history_tablets for recent failures
SHOW PROC "/cluster_balance/history_tablets";
Step D-2: Fix based on error type
Error: "unable to find dest path for new replica"
→ Check disk space on all BEs: SHOW BACKENDS (DataUsedCapacity vs TotalCapacity)
→ If disk full: free space or add BE
→ Check that destination BE is alive and heartbeating
Error: "clone task timeout"
→ Increase timeout (dynamic):
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_clone_task_timeout_sec" = "7200");
Error: source replica in error state
→ Mark replica bad to trigger fresh clone from healthy replica:
ADMIN SET REPLICA STATUS PROPERTIES(
"tablet_id" = "<id>",
"backend_id" = "<bad_be_id>",
"status" = "bad"
);
Step D-3: Restore replica if over-marked
ADMIN SET REPLICA STATUS PROPERTIES(
"tablet_id" = "<id>",
"backend_id" = "<be_id>",
"status" = "ok"
);
Step E-1: Check group state
-- List all colocate groups and their status
SHOW PROC "/colocation_group";
-- Check specific group balance
SHOW PROC "/colocation_group/<group_id>";
Step E-2: Check tolerate time configuration
-- Default 12hr: colocate repair does not start until BE is down this long
-- Reduce if you want faster recovery (but risks unnecessary moves on short outages)
ADMIN SET FRONTEND CONFIG ("tablet_sched_colocate_be_down_tolerate_time_s" = "3600");
-- Wait time before starting balance after system stabilizes (default 15min)
ADMIN SET FRONTEND CONFIG ("tablet_sched_colocate_balance_wait_system_stable_time_s" = "300");
Step E-3: Wait or force
-- If all BEs are healthy, the group should auto-stabilize
-- Monitor:
SHOW PROC "/cluster_balance/sched_stat";
-- Watch "num of colocate replica mismatch" decreasing
-- Primary signal: UnhealthyTabletNum should reach 0
SHOW PROC "/statistic";
-- Clone tasks completing, none stuck
SHOW PROC "/cluster_balance/sched_stat";
-- Confirm: "num of clone task succeeded" rising, "num of clone task failed" stable
-- History confirms clean completions
SHOW PROC "/cluster_balance/history_tablets";
-- Status column should show FINISHED, not FAILED
-- Verify specific table recovery
SELECT tablet_id, type, state, msg
FROM information_schema.fe_tablet_schedules
WHERE table_id = <table_id>
AND state != 'FINISHED'
ORDER BY create_time DESC;
# Grafana: starrocks_fe_scheduled_pending_tablet_num should trend to 0
# Clone success rate: starrocks_fe_clone_task_success / starrocks_fe_clone_task_total → 1.0
Target state:
UnhealthyTabletNum = 0 across all databasesCloningTabletNum = 0REPLICA_MISSING or VERSION_INCOMPLETE entries in pending_tablets| Parameter | Default | Description | Dynamic |
|---|---|---|---|
tablet_sched_slot_num_per_path | 8 | Clone slots per BE path | Yes |
tablet_sched_max_scheduling_tablets | 1000 | Max concurrent scheduling tablets (pending+running) | Yes |
tablet_sched_max_balancing_tablets | 500 | Max tablets in balance queue | Yes |
tablet_sched_max_not_be_scheduled_interval_ms | 15min | Time before tablet priority is upgraded | Yes |
tablet_sched_min_clone_task_timeout_sec | 180 (3min) | Minimum clone timeout | Yes |
tablet_sched_max_clone_task_timeout_sec | 7200 (2hr) | Maximum clone timeout | Yes |
tablet_sched_disable_balance | false | Disable non-colocate balance | Yes |
tablet_sched_disable_colocate_balance | false | Disable colocate balance | Yes |
tablet_sched_balance_load_score_threshold | 0.1 | Disk usage difference threshold to trigger balance | Yes |
tablet_sched_balance_load_disk_safe_threshold | 0.5 | Max disk usage allowed for clone destination | Yes |
tablet_sched_checker_interval_seconds | 20 | Tablet health check interval | Yes |
tablet_sched_be_down_tolerate_time_s | 900 (15min) | BE down time before repair clone starts | Yes |
tablet_sched_colocate_be_down_tolerate_time_s | 43200 (12hr) | BE down tolerance for colocate repair | Yes |
tablet_sched_repair_delay_factor_second | 60 | Repair delay multiplier (HIGH=1x, NORMAL=2x, LOW=3x) | Yes |
tablet_sched_colocate_balance_wait_system_stable_time_s | 900 (15min) | Wait after BE event before colocate balance starts | Yes |
catalog_trash_expire_second | 86400 | Recycle bin TTL; reduce to unblock decommission | Yes |
max_clone_task_speed_mbps | 0 (unlimited) | Per-clone bandwidth cap in MB/s | Yes |
| Parameter | Default | Description | Dynamic |
|---|---|---|---|
parallel_clone_task_per_path | 8 | Concurrent clone threads per disk path | No (restart) |
One or more BE nodes have significantly higher disk utilization than peers
↓ observable: SHOW PROC '/cluster_balance' shows imbalanced disk usage; Grafana per-node disk util divergence
Tablet scheduler initiates migration (clone) from high- to low-utilization nodes
↓ observable: starrocks_be_clone_task_count rising on source and destination; BE log clone tasks started
Clone tasks generate heavy sequential read/write on involved disks
↓ observable: iostat shows disk util > 80% on involved nodes; network throughput between nodes elevated
Normal read and write operations experience latency increase
↓ observable: Profile OLAP_SCAN_NODE Active time variance between instances; load job write latency rises
Intermittent slow queries and elevated load latency on specific nodes.
Trigger conditions: Uneven initial tablet distribution; new node added to cluster; previous node failure leaving survivors with extra tablets.
Break point: Throttle clone bandwidth via max_clone_task_speed_mbps; reduce concurrent clones via tablet_sched_max_scheduling_tablets; schedule rebalancing during off-peak.
Tablet scheduler triggers clone of PK table tablet during active import
↓ observable: starrocks_be_clone_task_count > 0; BE log clone task for PK tablet started
Clone destination BE preloads PK index for the newly cloned tablet
↓ observable: BE log PK index preload started; memory usage on destination BE rising; disk read IO for index files
PK index preload competes with import write path for IO and CPU
↓ observable: BE log write latency increasing; Grafana import bytes/sec drops on destination node
Import transaction write timeout triggered
↓ observable: FE log transaction timeout; load job transitions to CANCELLED
Load job failure coincides with rebalance activity.
Trigger conditions: Active PK table imports while tablet scheduler runs clone tasks; large PK index relative to available IO bandwidth.
Break point: Set skip_pk_preload = true to defer index load; throttle clone concurrency; schedule migrations outside import windows.
Node decommission triggered; all tablets on that node need replicas elsewhere
↓ observable: SHOW PROC '/cluster_balance' shows pending decommission tasks; clone count begins rising
Scheduler creates clone tasks for all tablets on decommissioning node simultaneously
↓ observable: starrocks_be_clone_task_count spikes across multiple destination nodes
Concurrent clone tasks saturate inter-node network bandwidth
↓ observable: Network throughput near capacity; iostat shows high disk read on decommissioning node and write on destinations
All nodes experience increased network latency; read and write latency rises cluster-wide
↓ observable: Grafana p99 query latency rises across all nodes; BE log RPC timeout errors
Cluster-wide performance degradation during decommission.
Trigger conditions: Large node with many tablets decommissioned without throttling; replication factor at minimum; insufficient spare network capacity.
Break point: Limit clone tasks via tablet_sched_max_scheduling_tablets; decommission nodes one at a time; increase replication factor before decommission.
case-004-disk-balancing — IO saturation from migration retry loop