Terminal
Terminal/Skill
Sets up contract tests, custom Arazzo workflow tests, and integration tests for a Speakeasy-generated SDK, and debugs test failures.
By Speakeasy
Works with
npx skills add speakeasy-api/sdk-generation-skills --skill setup-sdk-testingSet up and run tests for Speakeasy-generated SDKs using contract testing, custom Arazzo workflows, or integration tests against live APIs.
| Topic | Guide |
|---|---|
| Arazzo Reference | content/arazzo-reference.md |
The Arazzo reference provides complete syntax for workflows, steps, success criteria, environment variables, and chaining operations.
gen.yamlResponseValidationError or test failures| Input | Required | Description |
|---|---|---|
gen.yaml | Yes | Generation config; contract testing is enabled here |
| OpenAPI spec | Yes | The API spec the SDK is generated from |
| Target language | Yes | typescript, python, go, or java (contract test support) |
.speakeasy/tests.arazzo.yaml | No | Custom Arazzo test definitions |
| Live API credentials | No | Required only for integration testing |
| Output | Location |
|---|---|
| Generated contract tests | tests/ directory in SDK output |
| Mock server config | Auto-generated alongside contract tests |
| Arazzo test results | Terminal output from speakeasy test |
| CI workflow | .github/workflows/ (if integration testing configured) |
speakeasy auth login)gen.yaml + OpenAPI spec)Pick the right testing approach based on what you need:
| Need | Approach | Effort |
|---|---|---|
| Verify SDK types and methods match the API contract | Contract testing | Low (auto-generated) |
| Test multi-step API workflows (create then verify) | Custom Arazzo tests | Medium |
| Validate against a live API with real data | Integration testing | High |
| Catch regressions on every SDK regeneration | Contract testing + CI | Low |
| Test authentication flows end-to-end | Integration testing | High |
| Verify chained operations with data dependencies | Custom Arazzo tests | Medium |
Start with contract testing. It is auto-generated and catches the most common issues. Add custom Arazzo tests for workflow coverage, and integration tests only when live API validation is required.
Add to gen.yaml:
generation:
tests:
generateTests: true
Then regenerate the SDK:
speakeasy run --output console
# Run all Arazzo-defined tests (contract + custom)
speakeasy test
# Run tests for a specific target
speakeasy test --target my-typescript-sdk
# Run with verbose output for debugging
speakeasy test --verbose
Contract tests run automatically in the Speakeasy GitHub Actions workflow when test generation is enabled. No additional CI configuration is needed for contract tests.
Enable test generation in gen.yaml:
generation:
tests:
generateTests: true
Regenerate the SDK:
speakeasy run --output console
Run the generated tests:
speakeasy test
The CLI generates tests from your OpenAPI spec, creates a mock server that returns spec-compliant responses, and validates that the SDK correctly handles requests and responses.
Create or edit .speakeasy/tests.arazzo.yaml:
arazzo: 1.0.0
info:
title: Custom SDK Tests
version: 1.0.0
sourceDescriptions:
- name: my-api
type: openapi
url: ./openapi.yaml
workflows:
- workflowId: create-and-verify-resource
steps:
- stepId: create-resource
operationId: createResource
requestBody:
payload:
name: "test-resource"
type: "example"
successCriteria:
- condition: $statusCode == 201
outputs:
resourceId: $response.body#/id
- stepId: get-resource
operationId: getResource
parameters:
- name: id
in: path
value: $steps.create-resource.outputs.resourceId
successCriteria:
- condition: $statusCode == 200
- condition: $response.body#/name == "test-resource"
- workflowId: list-and-filter
steps:
- stepId: list-resources
operationId: listResources
parameters:
- name: limit
in: query
value: 10
successCriteria:
- condition: $statusCode == 200
Run the custom tests:
speakeasy test
Reference environment variables for sensitive values:
steps:
- stepId: authenticated-request
operationId: getProtectedResource
parameters:
- name: Authorization
in: header
value: Bearer $env.API_TOKEN
successCriteria:
- condition: $statusCode == 200
Use an overlay to disable a generated test without deleting it:
overlay: 1.0.0
info:
title: Disable flaky test
actions:
- target: $["workflows"][?(@.workflowId=="flaky-test")]
update:
x-speakeasy-test:
disabled: true
For live API testing, use the SDK factory pattern:
TypeScript example:
import { SDK } from "./src";
function createTestClient(): SDK {
return new SDK({
apiKey: process.env.TEST_API_KEY,
serverURL: process.env.TEST_API_URL ?? "https://api.example.com",
});
}
describe("Integration Tests", () => {
const client = createTestClient();
it("should list resources", async () => {
const result = await client.resources.list({ limit: 5 });
expect(result.statusCode).toBe(200);
expect(result.data).toBeDefined();
});
it("should create and delete resource", async () => {
// Create
const created = await client.resources.create({ name: "integration-test" });
expect(created.statusCode).toBe(201);
// Cleanup
const deleted = await client.resources.delete({ id: created.data.id });
expect(deleted.statusCode).toBe(204);
});
});
Python example:
import os
import pytest
from my_sdk import SDK
@pytest.fixture
def client():
return SDK(
api_key=os.environ["TEST_API_KEY"],
server_url=os.environ.get("TEST_API_URL", "https://api.example.com"),
)
def test_list_resources(client):
result = client.resources.list(limit=5)
assert result.status_code == 200
assert result.data is not None
@pytest.mark.cleanup
def test_create_and_delete(client):
created = client.resources.create(name="integration-test")
assert created.status_code == 201
try:
fetched = client.resources.get(id=created.data.id)
assert fetched.data.name == "integration-test"
finally:
client.resources.delete(id=created.data.id)
GitHub Actions CI for integration tests:
name: Integration Tests
on:
schedule:
- cron: "0 6 * * 1-5"
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: speakeasy-api/sdk-generation-action@v15
with:
speakeasy_version: latest
- run: speakeasy test
env:
SPEAKEASY_API_KEY: ${{ secrets.SPEAKEASY_API_KEY }}
- name: Run integration tests
run: npm test -- --grep "integration"
env:
TEST_API_KEY: ${{ secrets.TEST_API_KEY }}
TEST_API_URL: ${{ secrets.TEST_API_URL }}
$env.VARIABLE_NAME syntaxResponseValidationError usually means the spec and API are out of syncResponseValidationError in contract testsThe SDK response does not match the OpenAPI spec. Common causes:
required arrays match actual API responsestype and format fields in schema definitions# Regenerate with latest spec and re-run tests
speakeasy run --output console && speakeasy test --verbose
speakeasy test command not foundUpdate the Speakeasy CLI:
speakeasy update
speakeasy lint openapi -s spec.yamlgenerateTests: true is set in gen.yaml and the SDK has been regenerated.speakeasy/tests.arazzo.yamloperationId values match those in your OpenAPI spec exactlyfinally or afterEach)