Terminal
Terminal/Skill
Generates synthetic PDF documents with paired test questions and uploads them to Unity Catalog volumes for retrieval evaluation.
By Databricks
Works with
npx skills add databricks/databricks-agent-skills --skill databricks-unstructured-pdf-generationWorkflow for producing synthetic PDF documents + paired test questions as a Unity Catalog-resident dataset for Demos and RAG / unstructured-document retrieval evaluation on Databricks. The PDF-generation step uses standard local HTML → PDF tooling; the Databricks-specific value is the workflow shape — UC volume layout, paired question files, and integration with downstream Databricks retrieval / ai_extract / ai_parse_document evaluation.
./raw_data/html/ (write multiple files in parallel for speed) — domain-shaped to match the documents your retrieval pipeline will see in production.<SKILL_ROOT>/scripts/pdf_generator.py (parallel conversion, wraps plutoprint).databricks fs cp — same volume shape your production pipeline will read from../raw_data/pdf/pdf_eval_questions.json pairing each document with retrieval-eval questions; this becomes the gold dataset for mlflow.genai.evaluate() or comparable retrieval-quality scorers.If you only need ad-hoc PDFs (no Databricks workflow), any HTML → PDF tool (
weasyprint,wkhtmltopdf,playwright pdf,plutoprint) works directly — this skill exists for the synthetic-dataset-on-UC end-to-end shape, not as a general PDF generator.
Path convention:
<SKILL_ROOT>below = the directory containing this SKILL.md. Resolve to the absolute install path (e.g.~/.claude/skills/databricks-unstructured-pdf-generation)../raw_data/...paths are relative to your own project cwd.
uv pip install plutoprint
mkdir -p ./raw_data/html
Write HTML documents to ./raw_data/html/filename.html. Use subdirectories to organize (structure is preserved).
# Convert entire folder (parallel, 4 workers)
python <SKILL_ROOT>/scripts/pdf_generator.py convert --input ./raw_data/html --output ./raw_data/pdf
Skips files where PDF exists and is newer than HTML. Use --force to reconvert all.
databricks fs requires the dbfs: scheme prefix even for UC Volume paths. -r copies the contents of the source directory into the target (the source directory name is not preserved), so name the target raw_data/pdf explicitly to keep the PDFs in their own folder on the volume. They land under raw_data/pdf/ — i.e. dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf/report.pdf — so a Knowledge Assistant or ingest pipeline can point at that single folder.
databricks fs cp -r --overwrite ./raw_data/pdf dbfs:/Volumes/my_catalog/my_schema/raw_data/pdf
Create ./raw_data/pdf/pdf_eval_questions.json with questions for Knowledge Assistant (KA) or Multi-Agent Supervisor (MAS) evaluation. It's fine for this file to be uploaded to the volume alongside the PDFs — downstream agents can use it:
{
"api_errors_guide.pdf": {
"question": "What is the solution for error ERR-4521?",
"expected_fact": "Call /api/v2/auth/refresh with refresh_token before the 3600s TTL expires"
},
"installation_manual.pdf": {
"question": "What port does the service use by default?",
"expected_fact": "Port 8443 for HTTPS, configurable via CONFIG_PORT environment variable"
}
}
This JSON can be used to build KA test cases and validate retrieval accuracy.
When generating documents for Knowledge Assistant testing or demos:
Good document types:
Example content: Instead of generic "Connection failed" errors, write:
/api/v2/auth/refresh with your refresh_token before expiration. See Section 4.2 for token lifecycle management."python <SKILL_ROOT>/scripts/pdf_generator.py convert [OPTIONS]
--input, -i Input HTML file or folder (required)
--output, -o Output folder for PDFs (required)
--force, -f Force reconvert (ignore timestamps)
--workers, -w Parallel workers (default: 4)
Subfolder structure is preserved:
./raw_data/html/ ./raw_data/pdf/
├── report.html → ├── report.pdf
├── quarterly/ ├── quarterly/
│ └── q1.html → │ └── q1.pdf
└── legal/ └── legal/
└── terms.html → └── terms.pdf
This skill ships one helper script:
| File | Description |
|---|---|
| scripts/pdf_generator.py | HTML → PDF converter (wraps plutoprint); parallel folder conversion with timestamp-skip. Referenced by Step 2 and the CLI Reference. |
The script ships at <SKILL_ROOT>/scripts/pdf_generator.py. If it is absent, recreate it from the CLI Reference above (a convert subcommand taking --input/--output/--force/--workers, wrapping plutoprint for HTML → PDF).
| Issue | Solution |
|---|---|
| "plutoprint not installed" | uv pip install plutoprint |
| PDF looks wrong | Check HTML/CSS syntax |
| "Volume does not exist" | databricks volumes create CATALOG SCHEMA VOLUME_NAME MANAGED (four separate positional args, not catalog.schema.volume) |