Terminal
Terminal/Skill
Creates code-based evaluators that score LangSmith-traced agent outputs and runs experiments against LangSmith datasets.
By LangChain
Works with
npx skills add langchain-ai/lca-skills --skill langsmith-code-evalCreates evaluators for LangSmith experiments through structured inspection and implementation.
langsmith Python package installedLANGSMITH_API_KEY environment variable set (check project's .env file)Copy this checklist and track progress:
Evaluator Creation Progress:
- [ ] Step 1: Gather info from user
- [ ] Step 2: Inspect trace and dataset structure
- [ ] Step 3: Read agent code
- [ ] Step 4: Write evaluator
- [ ] Step 5: Write experiment runner
- [ ] Step 6: Run and iterate
IMPORTANT: Do NOT search or explore the codebase. Ask the user all of these questions upfront using AskUserQuestion before doing anything else.
Ask the user the following in a single AskUserQuestion call:
python, python3, uv run python, poetry run python)Using the info from Step 1, run the inspection scripts located in this skill's directory:
{python_cmd} {skill_dir}/scripts/inspect_trace.py PROJECT_NAME [RUN_ID]
{python_cmd} {skill_dir}/scripts/inspect_dataset.py DATASET_NAME
Replace {python_cmd} with the command from Step 1, and {skill_dir} with this skill's directory path.
Verify the trace matches the agent:
From the dataset inspection, note:
expected_tool, difficulty, labels)The dataset metadata often contains ground truth for evaluation (e.g., which tool should be called, expected classification).
Read the agent file provided in Step 1 to identify:
@traceable decorator)Create evaluator functions based on trace and dataset structure. See EVALUATOR_REFERENCE.md for function signatures and return formats.
Create a script that:
evaluate() or aevaluate() against the datasetSee EVALUATOR_REFERENCE.md for evaluate() usage.
Execute the experiment, review results in LangSmith, refine evaluators as needed.