Terminal
Terminal/Skill
Diagnoses PAI-Rec recommendation engine interface errors from service logs and checks engine configuration files for problems before deployment.
By Alibaba Cloud
Works with
npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-pai-rec-diagnosisThis skill provides comprehensive diagnostic and validation capabilities for Alibaba Cloud PAI-Rec (Programmable Recommendation System) engines, including interface troubleshooting and configuration analysis.
PAI-Rec is Alibaba Cloud's programmable recommendation system that provides intelligent recommendation capabilities. This skill helps users:
Diagnose PAI-Rec Engine Interface Issues: When engine API returns errors or unexpected results, trace the request through EAS service logs and engine configurations to identify root causes.
Validate Engine Configurations: Analyze engine configuration files for potential issues, inconsistencies, or misconfigurations before deployment.
Architecture: PAI-EAS Service + PAI-Rec Engine + Engine Configuration Management
Pre-check: Aliyun CLI >= 3.3.3 required
Run
aliyun versionto verify >= 3.3.3. If not installed or version too low, runcurl -fsSL https://aliyuncli.alicdn.com/setup.sh | bashto install/update, or seereferences/cli-installation-guide.mdfor installation instructions.
Pre-check: Aliyun CLI plugin update required
[MUST] run
aliyun configure set --auto-plugin-install trueto enable automatic plugin installation. [MUST] runaliyun plugin updateto ensure that any existing plugins are always up-to-date.
Pre-check: Alibaba Cloud Credentials Required
Security Rules:
- NEVER read, echo, or print AK/SK values (e.g.,
echo $ALIBABA_CLOUD_ACCESS_KEY_IDis FORBIDDEN)- NEVER ask the user to input AK/SK directly in the conversation or command line
- NEVER use
aliyun configure setwith literal credential values- ONLY use
aliyun configure listto check credential statusaliyun configure listCheck the output for a valid profile (AK, STS, or OAuth identity).
If no valid profile exists, STOP here.
- Obtain credentials from Alibaba Cloud Console
- Configure credentials outside of this session (via
aliyun configurein terminal or environment variables in shell profile)- Return and re-run after
aliyun configure listshows a valid profile
[MUST] Permission Failure Handling: When any command or API call fails due to permission errors at any point during execution, follow this process:
- Read
references/ram-policies.mdto get the full list of permissions required by this SKILL- Use
ram-permission-diagnoseskill to guide the user through requesting the necessary permissions- Pause and wait until the user confirms that the required permissions have been granted
For detailed RAM policies required by this skill, see references/ram-policies.md.
IMPORTANT: Parameter Confirmation — Before executing any command or API call, ALL user-customizable parameters (e.g., RegionId, service names, instance IDs, environment names, config names, etc.) MUST be confirmed with the user. Do NOT assume or use default values without explicit user approval.
| Parameter Name | Required/Optional | Description | Default Value |
|---|---|---|---|
service-name | Required | PAI-EAS service name | N/A |
cluster-id | Required | EAS cluster/region ID | N/A |
instance-id | Required | PAI-Rec instance ID | N/A |
request-id | Required (for diagnosis) | Request ID from engine API response | N/A |
config-name | Optional | Engine configuration name | N/A |
environment | Optional | Engine environment (Prod/Pre) | N/A |
engine-config-id | Optional | Specific engine config version ID | N/A |
region | Optional | Alibaba Cloud region | N/A |
Upon loading this skill, generate a random session ID (32-char lowercase hex string) once for the entire session. Use it as {session-id} below.
Rule: Every aliyun CLI command that calls a cloud API MUST include the --user-agent flag.
Local utility commands (e.g. configure, plugin, version) do not support this flag and should be excluded.
--user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-rec-diagnosis/{session-id}
Example (assuming session-id is a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6):
aliyun eas describe-service --cluster-id cn-hangzhou --service-name my_service --user-agent AlibabaCloud-Agent-Skills/alibabacloud-pai-rec-diagnosis/a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6
Do not skip, alter the format, or omit --user-agent on any aliyun API command invocation.
[MUST] Allocate a per-session work directory — All transient artifacts (raw config dumps, log dumps, sanitized output) MUST be written under a unique per-session directory to avoid concurrent overwrite between parallel skill invocations. Run the following at the start of the workflow, before any artifact-producing CLI invocation, and reuse
$WORKDIRfor the whole session. Pass the full template as a positional arg (works on both BSD/macOS and GNU/Linuxmktemp); do NOT use-t prefix, which falls back to$TMPDIR(e.g./var/folderson macOS) and may even fail under sandboxed shells.export WORKDIR=$(mktemp -d /tmp/pairec-diag-XXXXXX)All file paths shown below (
$WORKDIR/engine_configs_list.json, etc.) live inside this directory and MUST NOT be replaced with hard-coded/tmp/...paths.
[MUST] Use ONLY the workflows defined below. Do NOT invent additional steps (e.g. instance resource checks, network probes) or substitute manual analysis for a defined workflow step.
This workflow helps diagnose issues when a PAI-Rec engine API returns errors or unexpected results.
Input Example:
Service Name: embedding_recall
API Response:
{
"code": 299,
"msg": "items size not enough",
"request_id": "941b4e14-d1c5-489f-a184-b2b17f8b4fdb",
"size": 0,
"experiment_id": "",
"items": []
}
Get the service details to find the EAS service ID and configuration:
aliyun eas describe-service \
--cluster-id <cluster-id> \
--service-name <service-name>
What to extract:
Resource: EAS service resource ID (e.g., eas-r-1v4qb1yan3qmnjwxqe)ServiceConfig.envs: Environment variables containing:
REGION: The regionINSTANCE_ID: PAI-Rec instance IDCONFIG_NAME: Engine configuration namePAIREC_ENVIRONMENT: Environment (product/prepub)Parse the API response JSON to get the request_id field. This will be used to search service logs.
Use the request ID as the sole filter to search service logs. Do NOT pass --start-time / --end-time when searching PAI-Rec business logs:
aliyun eas describe-service-log \
--cluster-id <cluster-id> \
--service-name <service-name> \
--keyword <request-id> \
--page-size 500
[CRITICAL] --keyword <request-id> is MANDATORY — local post-processing is FORBIDDEN:
--keyword <request-id> (server-side filter, exact case-sensitive match on the full request_id). The API returns only log lines matching the keyword.--keyword and then filter locally (e.g., piping through head, grep, python3, jq, or any script).describe-service-log calls without --keyword hoping to find relevant lines by scanning the full log stream.--keyword returns empty results, report that no matching logs were found — do NOT fall back to fetching unfiltered logs.--page-size 500 captures the entire trace in a single page; matched entries for one request are usually < 30.❌ WRONG (fetches ALL logs, filters locally — FORBIDDEN):
aliyun eas describe-service-log --cluster-id cn-beijing --service-name embedding_recall | head -300 aliyun eas describe-service-log --cluster-id cn-beijing --service-name embedding_recall | grep "request_id"✅ RIGHT (server-side keyword filter — REQUIRED):
aliyun eas describe-service-log --cluster-id cn-beijing --service-name embedding_recall --keyword 0c6cbd91-5618-4705-8e08-9126bf4600f7 --page-size 500
[CRITICAL] Time range silently drops business logs:
--keyword only (no time range), the CLI returns the full PAI-Rec application trace (controller.go / feed.go / recall.go / rank_service.go etc.) matching the request_id.--start-time / --end-time — even when the window covers the real timestamp — silently drops business logs and returns only infrastructure noise (/bin/sh heartbeats, 502 Bad Gateway retries, postgres.go dbstat).--keyword, in yyyy-MM-dd HH:mm:ss UTC format (no T / no Z); ISO-8601 forms like 2025-04-28T00:00:00Z are rejected with InvalidParameter.Map the environment and list matching configurations:
Environment Mapping:
product → Prodprepub → Prealiyun pairecservice list-engine-configs \
--instance-id <instance-id> \
--environment <Prod|Pre> \
--status Released \
--name <config-name> > "$WORKDIR/engine_configs_list.json" 2>&1
[MUST] Always pass --name <config-name> for server-side filtering:
<config-name> is already known from Step 1 (ServiceConfig.envs.CONFIG_NAME); forward it as --name. Omitting it returns the entire instance's config inventory (often hundreds of unrelated entries), forces client-side filtering, wastes tokens, and risks hitting CLI default pagination so the target row is silently dropped. --name is an exact-match filter on the server; do NOT substitute with grep / jq select post-processing. The same rule applies to every list-engine-configs invocation in this skill (Workflow 2 Step 1 included).
What to extract:
Status: ReleasedEngineConfigId and Versionaliyun pairecservice get-engine-config \
--instance-id <instance-id> \
--engine-config-id <engine-config-id> > "$WORKDIR/raw_engine_config.json" 2>&1
[MUST] Sanitize before display — Config may contain plaintext passwords or
access keys. Always pipe through the sanitizer before printing to terminal;
only sanitized output (with credentials replaced by ***REDACTED***) should
appear there. The raw file at $WORKDIR/raw_engine_config.json can be passed
directly to scripts/validate.py (which does not print credential values).
python3 scripts/sanitize_config.py "$WORKDIR/raw_engine_config.json"
What to extract:
ConfigValue: The actual engine configuration (JSON/YAML)Run scripts/validate.py against the retrieved ConfigValue to rule out structural /
reference errors. See references/config-validation.md.
printf '%s' "$CONFIG_VALUE" | python3 scripts/validate.py --stdin
When to run: when logs point at a config element or when diagnosing the config for the first time.
When to skip: when logs show a non-config root cause (missing scene_id, upstream 5xx).
[MUST NOT] Do not replace or duplicate validate.py (same restriction as Workflow 2 § Step 3).
[MUST] Scoping rule: findings enter the final diagnosis ONLY when tied to log evidence for the current request_id.
Condition: experiment_id in the API response is non-empty (e.g., "ER14_L21_L26#EG21_L38#EG38#E44_GL36_GL37").
Parse: Extract EG{id} (experiment group) and E{id} (experiment) numeric IDs from the string. Ignore ER, L, GL prefixes — they carry no config.
# For each EG{id}:
aliyun pairecservice get-experiment-group \
--instance-id <instance-id> \
--experiment-group-id <id> > "$WORKDIR/experiment_group_<id>.json" 2>&1
# For each E{id}:
aliyun pairecservice get-experiment \
--instance-id <instance-id> \
--experiment-id <id> > "$WORKDIR/experiment_<id>.json" 2>&1
What to extract: The Config field — contains override parameters (e.g., default.RecallNames, rankconf, filterNames, default.SortNames) that supersede the base engine config.
Override priority (low → high): Base Engine Config < ExperimentGroup.Config < Experiment.Config. Apply in Step 6 to understand actual runtime behavior.
Validate experiment configs against the base config (reference existence check):
python3 scripts/validate.py "$WORKDIR/raw_engine_config.json" \
--experiment-config "$WORKDIR/experiment_group_<id>.json" \
--experiment-config "$WORKDIR/experiment_<id>.json"
Analyze the following components together:
experiment_id non-empty): Effective config = base config with experiment parameters applied on topCommon Issues to Check:
RecallNames / rankconf from base config)[MUST] Evidence-only reporting rule:
The final diagnosis delivered to the user MUST be grounded strictly in what the EAS service logs and the engine configuration directly show. Apply the following constraints:
This workflow validates engine configurations for potential issues.
Input: Configuration name and environment (Prod/Pre)
If user doesn't provide engine-config-id, list available versions:
aliyun pairecservice list-engine-configs \
--instance-id <instance-id> \
--environment <Prod|Pre> \
--name <config-name>
Display to user:
Version: Version numberStatus: Configuration status (Released/Draft/Archived)GmtCreateTime: Creation timestampEngineConfigId: Version IDAsk user to select a version or provide the engine-config-id.
aliyun pairecservice get-engine-config \
--instance-id <instance-id> \
--engine-config-id <engine-config-id> > "$WORKDIR/raw_engine_config.json" 2>&1
[MUST] Sanitize before display — Always sanitize before printing to terminal:
python3 scripts/sanitize_config.py "$WORKDIR/raw_engine_config.json"
[MUST] Feed the extracted ConfigValue JSON into scripts/validate.py. The script
enforces JSON Schema (references/schema.json) + reference-consistency rules and exits
with status 0 on pass, 1 on failure.
# From a saved JSON file (recommended)
python3 scripts/validate.py "$WORKDIR/raw_engine_config.json"
# Or pipe ConfigValue directly via stdin
printf '%s' "$CONFIG_VALUE" | python3 scripts/validate.py --stdin
Requires jsonschema (pip install jsonschema); if missing the script falls back to
rule-only validation without Schema checks.
[MUST NOT] Do not replace or duplicate validate.py:
0 error(s), 0 warning(s) run.What the script checks (summary):
RecallConfs,
FilterConfs, SortConfs, AlgoConfs, SceneConfs, RankConf,
FeatureConfs, UserFeatureConfs, DebugConfs, FeatureLogConfs,
CallBackConfs, PipelineConfs, etc.)RecallType / FilterType / SortType /
DebugConfs.OutputType / GeneralRankConfs.ActionConfs[].ActionTypeSceneConfs.RecallNames → RecallConfs;
FilterNames → FilterConfs; SortNames → SortConfs;
RankConf.RankAlgoList → AlgoConfs; any DaoConf.AdapterType +
*Name → the corresponding *Confs (Hologres / Redis / MySQL / TableStore /
FeatureStore / …)User2ItemExposureFilter with WriteLog=true + FeatureStore adapter: must set
TimeInterval > 0PriorityAdjustCountFilter in accumulator mode: Count must be strictly
increasing (use Type="fix" for independent per-recall caps)PipelineConfs.*.Name must be globally uniqueDebugConfs.Rate must be an integer in [0, 100]RecallConfs, FilterConfs, SortConfs,
AlgoConfsDetailed usage, exit codes, example outputs and the full rule list live in references/config-validation.md.
[MUST] Required first line of the report: Quote validate.py's stdout
verbatim — either Validation passed: configuration is well-formed or
Validation finished: N error(s), M warning(s). A report missing this
exact line is INVALID; restart from Step 3.
Manual inspection is allowed only for concerns out of scope of
validate.py: env / region / model-signature mismatches, cross-version diffs,
naming collisions between RankScore variables and model output fields, and
root-cause reading of any [WARNING] the script itself asks a human to judge.
Do NOT add findings the script did not report unless you can tie them to one
of these out-of-scope concerns.
Report structure:
validate.py's 0 error(s), 0 warning(s) line[WARNING] <path>: <message> from the script,
plus any out-of-scope inconsistency from manual inspection[ERROR] <path>: <message> from the scriptDo not add speculative fixes or best-practice tangents; suggestions are provided only when the user explicitly asks for them.
For detailed verification steps, see references/verification-method.md.
Quick Verification:
For Diagnosis Workflow:
For Validation Workflow:
This skill performs read-only Alibaba Cloud API calls (no remote resources are
created). Transient artifacts go to a per-session local $WORKDIR under /tmp
(see Core Workflow preamble). The skill does NOT delete $WORKDIR automatically
— the OS-level temp policy reclaims it (macOS reaps /tmp periodically; most
Linux distros reap on reboot or via systemd-tmpfiles). To free disk space
sooner, manually run rm -rf /tmp/pairec-diag-* outside the workflow.
--keyword <request_id> to aliyun eas describe-service-log and leave --start-time / --end-time unset. NEVER omit --keyword and post-process locally (e.g., | head, | grep, | python3) — this defeats server-side filtering, wastes tokens, and may miss logs beyond the first page. Combining keyword with a time range filters out business logs due to a CLI quirk (see Workflow 1, Step 3). Only use time ranges for broad non-request scans, and only with the yyyy-MM-dd HH:mm:ss UTC format (no T / no Z).validate.py: For Workflow 2, treat scripts/validate.py as the single source of truth for the rules in its catalogue. Do NOT skip it and hand-write checks, and do NOT re-validate its rules manually after a clean run. Manual inspection is reserved for concerns out of its scope (env / region / model signature, cross-version diffs, RankScore vs model output naming).| Reference Document | Description |
|---|---|
| RAM Policies | Required RAM permissions for PAI-Rec and EAS APIs |
| Related Commands | Complete CLI command reference |
| Verification Method | Detailed verification procedures |
| CLI Installation Guide | Alibaba Cloud CLI installation instructions |
| Configuration Examples | Sample engine configurations and common patterns |
| Config Validation | scripts/validate.py usage, exit codes, rule catalogue |
| Troubleshooting Guide | Common issues and solutions |
| Config Sanitization | Credential redaction before LLM analysis |