Python API: run DuckLess jobs from code

Everything duckless run, status, logs and cancel do is available from Python, without the terminal. It’s how an orchestrator, a script or a notebook runs DuckLess jobs.

import duckless

client = duckless.Client.from_env()  # the DUCKLESS_* settings, as the CLI reads them
job = client.run("checks.py", machine="n2-highmem-8", spot=True, env={"DAY": "2026-10-20"})
report = client.wait(job.job_id, check=True)

print(report.status.state, report.metrics["seconds"])

The client

duckless.Client.from_env() reads the same DUCKLESS_* settings as the CLI. Keyword arguments override them, and a mapping replaces the environment:

client = duckless.Client.from_env(project="my-project", region="europe-west1")
client = duckless.Client.from_env(
    {"DUCKLESS_PROJECT": "my-project", "DUCKLESS_BUCKET": "…", "DUCKLESS_SA": "…", "DUCKLESS_IMAGE": "…"}
)

Nothing is called on Google Cloud until the first job: building the client is free. It uses the caller’s credentials (Application Default Credentials), which need the job submitter roles.

Method	Returns
run(source, *, machine, spot, local_ssd_count, env, on, name, max_run_seconds)	JobStatus	submits a .sql or .py file and returns at once; on is auto, batch or cloudrun
run(image=…, command=[…], …)	JobStatus	runs any image with a command, like duckless exec
status(job_id)	JobReport	state, timeline, and the runner’s metrics once the job is over
wait(job_id, *, poll_seconds, timeout_seconds, check)	JobReport	blocks until the job ends
cancel(job_id)	None	cancels a queued or running job
logs(job_id, since)	tuple[LogLine, …]	the runner’s log lines, oldest first

Waiting and failures

wait returns the final report, whatever the outcome. With check=True, it raises duckless.JobFailedError unless the job succeeded; the message carries the runner’s error and the exception keeps the report:

try:
    client.wait(job.job_id, check=True)
except duckless.JobFailedError as failed:
    print(failed.report.metrics["error"])

timeout_seconds bounds the wait, not the job: duckless.JobTimeoutError is raised and the job keeps running. Cancel it if that’s what you want; the job’s own limit is max_run_seconds (3 hours by default).

try:
    client.wait(job.job_id, timeout_seconds=1800)
except duckless.JobTimeoutError:
    client.cancel(job.job_id)
    raise

report.metrics is the job’s metrics.json: duration and memory, each SQL statement with its duration and last rows, and what ran (image, versions, source sha256, BigQuery writes).

From Airflow (Cloud Composer)

Use the DuckLess operator: it waits for the job in the triggerer instead of a worker, fails the task with the runner’s error and pushes a summary of the job to XCom.