Terminal
Terminal/Skill
Lets code call the Confident AI API to manage organizations, projects, members, roles, and API keys, and read or write project data.
By Confident AI (DeepEval)
Works with
npx skills add confident-ai/confident-client --skill confident-clientUse this skill to call the Confident AI API from code with the
confident-ai package, available for both Python and TypeScript. It covers
275 operations across 35 resources: the account (organizations, projects,
members, RBAC, governance, API keys) and the resources a project holds
(prompts, datasets, traces, spans, threads, metrics, test runs, evaluations,
dashboards, annotation queues, red teaming).
This skill does not instrument a running application or write eval suites. For
those, see the deepeval, deepeval-tracing, and deepeval-otel skills.
Which key a call needs is decided by the resource, not by the operation:
| Key | Environment variable | Reaches |
|---|---|---|
| Organization | CONFIDENT_ORG_API_KEY | client.organization, client.projects, client.project(id) |
| Project | CONFIDENT_PROJ_API_KEY | every other resource — prompts, datasets, traces, metrics, test runs, … |
One client can hold both, and picks the right one per call:
client = ConfidentAI() # reads both environment variables
client.organization.get() # organization key
client.datasets.list() # project key
Neither is CONFIDENT_API_KEY, which deepeval uses — the distinct names let
both be configured at once.
confident-client) — call the API: administer the account,
and create, read, update or delete the resources a project holds.deepeval skill — build pytest eval suites: metrics, traced evals,
deepeval test run, iteration.deepeval-tracing skill — instrument an app (@observe, integrations).deepeval-otel skill — export raw OpenTelemetry / OTLP traces.Use this skill for API calls: pulling a prompt, pushing goldens to a dataset, or reading a test run's results. Use the other three for work inside a running program: emitting a span from a live request, or asserting a metric inside a test suite.
pip install confident-ai (Python) or
npm install confident-ai (TypeScript).confident_us_org_...) for account work, a
Project API Key (confident_us_proj_...) for everything else, or both.
Read by default from CONFIDENT_ORG_API_KEY and CONFIDENT_PROJ_API_KEY.Each reference file contains both Python and TypeScript examples. Before writing
code, determine which language the user is working in, then use the matching
code block (```python or ```typescript) within the
reference.
*.py, pyproject.toml, requirements.txt, setup.py, Pipfile, a .venv/venv*.ts, *.tsx, package.json, tsconfig.json, node_modules (JavaScript-only counts as TypeScript — same confident-ai package)ts/ next to a bare .venv is still ambiguous). Fall back to
Python (never TypeScript) only after asking and getting no answer, and say
you're defaulting.Within each reference, the code block for the target language is authoritative for exact method names, argument shapes (keyword arguments vs positional), and casing (snake_case vs camelCase). Do not guess a binding from the other language's examples.
references/quickstart.md.references/introduction.md maps the whole surface.permissions, bundle them into policies,
bundle policies into roles, then assign roles to members. Read
references/roles-policies-permissions.md.value immediately — it is shown only once. Read
references/api-keys.md.client.organization.list_api_keys().client.organization.get(),
client.datasets.list().list_members() returns an
OrganizationMemberList: .members holds the rows, .page and
.page_size describe the window. Read the rows off the named field.client.project(id), client.prompt(id) and client.dataset(id) return
a stateful handle that holds the record's id, so its methods take only
what is left. get() fills the handle and returns it; update() sends what
the handle holds.from confident_ai.datasets import SingleTurnGolden (Python), import { SingleTurnGolden } from "confident-ai/datasets" (TypeScript).value is returned only at creation (creating a
project also mints its first project key). Store it securely then; later
reads are masked. Prefer rotate_api_key over delete-and-recreate when a key
is in use — it can issue the replacement with a grace period.valid to false) over
deleting when you only need to revoke temporarily.a_:
await client.organization.a_get(). TypeScript is already promise-based.email), and which role (role_id) to
grant when inviting members or adding them to a project. Ask first, then act.Each topic has one reference file containing both Python and TypeScript examples; use the code block matching the target language.
| Topic | File |
|---|---|
| Install, configure both keys, first call | references/quickstart.md |
| The whole surface, the two scopes, stateful handles | references/introduction.md |
| Read and rename the organization | references/organization.md |
| Create, read, update, delete projects | references/projects.md |
| Members, invitations, and role assignment | references/members-and-invitations.md |
| RBAC: permissions, policies, and roles | references/roles-policies-permissions.md |
| Provision, rotate, enable/disable, and delete API keys | references/api-keys.md |
| Governance policies: CRUD and assign/unassign to projects | references/governance.md |
| Prompts, datasets, traces, metrics, test runs and evaluations | references/resources.md |
| Background: why observability matters for AI products | references/ai-pm-observability.md |
Every method carries the route's own summary and description as its docstring (Python) or JSDoc (TypeScript), so a resource with no worked example here is still readable from the method itself.