Terminal
Terminal/Skill
Shows how to build, pack, and place a Go SDK bundle on Airflow workers so the executable coordinator can run it.
By Astronomer
Works with
npx skills add astronomer/agents --skill deploying-go-sdk-bundlesA Go SDK deployment has one artifact: a bundle, a single self-contained native executable that also carries its embedded source and a manifest (the AFBNDL01 format, "the executable is the bundle"). You build and pack it with go, place it where Airflow's ExecutableCoordinator scans, and the Python task runner forks it once per task instance. This skill is platform-neutral: it shows the build, the coordinator wiring, then how to get the bundle onto a worker.
Experimental. The Go SDK is under active development and not production-ready. Everything resolves against the single module
github.com/apache/airflow/go-sdk(Go 1.24+).
Order of operations: write the tasks (authoring-go-sdk-tasks) -> build and pack the bundle (this skill) -> place it under
executables_rootand configure the coordinator -> deploy the matching Python stub DAG.
The coordinator only recognizes a packed bundle: it scans for the AFBNDL01 trailer and silently skips any file that lacks it, so a plain go build binary is not deployable on its own. Use the packer, shipped as a Go 1.24 tool directive in go.mod (no global install, version pinned per project):
go tool airflow-go-pack ./example/bundle # build + pack in one step
go tool airflow-go-pack --goos linux --goarch amd64 ./example/bundle -- -trimpath # cross-compile; flags after -- pass to `go build`
go tool airflow-go-pack --executable ./bin/sample-dag-bundle --source main.go --airflow-metadata <airflow-metadata.yaml> # pack an existing binary
go tool airflow-go-pack inspect ./bin/sample-dag-bundle # inspect a packed bundle
The packer builds the binary, execs it with --airflow-metadata to capture the manifest from RegisterDags, then appends source + manifest + a 64-byte trailer. The result is one runnable file.
--goos/--goarch. A mismatched binary fails on the worker with exec format error.binary_sha256, and the bundle is then rejected.Python's ExecutableCoordinator scans executables_root, matches the incoming dag_id against each bundle's embedded manifest, verifies its integrity hash, then forks the bundle. No Go process runs on the host.
Place the packed executable under a scanned directory:
cp ./bundle /opt/airflow/executable-bundles/ # identified by the AFBNDL01 trailer, not by filename
Register ExecutableCoordinator and route the queue to it (see configuring-airflow-language-sdks):
[sdk]
coordinators = {"go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator", "kwargs": {"executables_root": ["/opt/airflow/executable-bundles"]}}}
queue_to_coordinator = {"golang": "go"}
Deploy the matching Python stub DAG; its queue= must equal the queue_to_coordinator key (golang here), and its dag_id/task_ids must match what the bundle registered.
The SDK runs on any Airflow with the Task SDK; Astronomer tooling is not required.
Cross-compile the bundle for the image's platform and bake it in. No Go runtime or worker process is needed in the image; the Python task runner forks the bundle.
FROM apache/airflow:3.3.0 # the language SDKs target Airflow 3.3+
COPY ./executable-bundles/ /opt/airflow/executable-bundles/
# set AIRFLOW__SDK__COORDINATORS and AIRFLOW__SDK__QUEUE_TO_COORDINATOR as env vars
On the Helm chart, bake the bundle into a custom image as above or mount it via a shared volume, and set the [sdk] config through environment variables on the worker/scheduler. See deploying-airflow for the broader Docker Compose and Helm workflow.
The
apache/airflow:3.3.0tag above is illustrative: the language SDKs need Airflow 3.3 or newer. Pin whatever current 3.x you actually run rather than copying this tag from memory; read the base image's current tags or docs.
mkdir -p include/executable-bundles && cp ../go-bundle/<packed-bundle> include/executable-bundles/.Dockerfile, copy the bundle to the coordinator's directory: COPY include/executable-bundles/ /opt/airflow/executable-bundles/..env (loaded automatically): the AIRFLOW__SDK__* JSON values (see configuring-airflow-language-sdks).astro dev start (or astro dev restart after changes); deploy with astro deploy.Don't pin Astro Runtime / Airflow versions from memory; read the generated
Dockerfileor current docs. While the Go SDK is in preview, a beta/dev image may be required.
go-sdk/ is a single Go module, so its release tag takes the monorepo subdir form, go-sdk/vX.Y.Z (do not create per-cmd tags). Your bundle module depends on github.com/apache/airflow/go-sdk; pinning that version also pins airflow-go-pack, which is a package in the same module referenced through the tool directive. Pin against the release tag:
go get github.com/apache/airflow/go-sdk@v1.0.0
To build against an unreleased commit or branch (for example, to try a fix ahead of the next tag), depend on it directly and Go fabricates a pseudo-version:
go get github.com/apache/airflow/go-sdk@<commit-or-branch>
go tool airflow-go-pack); registered dag_id/task_id match the Python stubs.--goos linux --goarch amd64).executables_root.ExecutableCoordinator + queue_to_coordinator configured (configuring-airflow-language-sdks).queue= routed to the Go coordinator.binary_sha256).ExecutableCoordinator and route the queue.