API reference¶
Lifecycle classes and task models¶
All names below are exported from airflow_supervisor.
Execution |
Lifecycle class |
Configuration model |
|
|---|---|---|---|
Local |
|
|
|
SSH |
|
|
|
Supervisor(dag, cfg, **kwargs) adds lifecycle tasks to an existing Airflow DAG.
cfg accepts a configuration model or a dictionary validated as that model.
Construction requires an active DAG context for internally created tasks.
SupervisorSSH(dag, cfg, host=None, port=None, **kwargs) adds the same lifecycle
with SSH operators for remote configuration and daemon commands. Program
operations and checks use XML-RPC. A supplied Host overrides the SSH target
and XML-RPC host; a supplied Port overrides the supervisord listen port. A
host’s pool is used when cfg.pool is unset. command_prefix can also be passed
as a constructor keyword.
SupervisorTaskArgs contains cfg and inherited Airflow task arguments.
SupervisorSSHTaskArgs adds host and port, accepting Host/Port models,
queries of kind select, or callables and their import paths. SupervisorTask
and SupervisorSSHTask resolve their default operator to the corresponding
lifecycle class.
The SupervisorOperator, SupervisorOperatorArgs, SupervisorSSHOperator, and
SupervisorSSHOperatorArgs names are aliases for the corresponding task models.
DAG constraints and boundaries¶
Both lifecycle classes set catchup=False, concurrency=1,
max_active_tasks=1, and max_active_runs=1 on their DAG.
Generated task IDs use <dag_id>-<step>, independent of a task model’s task_id.
A DAG supports one supervisor lifecycle; multiple programs belong in its
cfg.program mapping.
The exposed boundaries are configure_supervisor, start_supervisor,
start_programs, check_programs, restart_programs, stop_programs,
stop_supervisor, and unconfigure_supervisor. Dependency operators connect
upstream tasks to configuration and downstream tasks to unconfiguration.
supervisor_client returns an XML-RPC client for the lifecycle’s configuration.
Stop and cleanup steps follow the successful monitoring branch. They are not a
general failure finalizer. With cleanup=False, unconfiguration is a skipped
task. With stop_on_exit=False, both program stop and unconfiguration are
skipped, and the default downstream trigger rules skip daemon shutdown.
Configuration models¶
SupervisorAirflowConfiguration extends the re-exported
SupervisorConvenienceConfiguration. SupervisorSSHAirflowConfiguration
extends it with SSH settings.
Supervisor fields¶
Field |
Default |
Meaning |
|---|---|---|
|
Required |
Mapping of program names to |
|
Derived temporary directory |
Instance directory on the worker or SSH host. |
|
|
Configuration path resolved under |
|
|
Host used by the XML-RPC client. |
|
|
Supervisord HTTP/XML-RPC listen address. Integer ports normalize to |
|
|
Supervisord HTTP/XML-RPC credentials. |
|
|
XML-RPC protocol; client ports 80 and 443 select HTTP and HTTPS respectively. |
|
|
XML-RPC request path. |
|
|
Required startup duration, applied to all programs. |
|
|
Startup retries; when set, overrides each program’s value. |
|
|
Accepted program exit statuses, applied to all programs. |
|
|
Stop signal, applied to all programs. |
|
|
Wait before forced termination. |
|
|
Stop and kill the process group. |
|
|
Convenience-command timeout in seconds. |
The convenience model forces program autostart=False and autorestart=False.
Program stdout and stderr files are <working_dir>/<program>/output.log and
error.log. The daemon log and PID file default to supervisord.log and
supervisord.pid in the working directory. Serialized configuration is stored
as pydantic.json; cleanup removes the working directory, including logs.
Monitoring and lifecycle fields¶
Field |
Default |
Meaning |
|---|---|---|
|
5 seconds |
Polling interval for the |
|
8 hours |
Sensor timeout. |
|
|
Forwards workload stdout and stderr through the Airflow task logger. |
|
|
Byte budget per stream per poll; |
|
|
Monitoring end condition relative to |
|
|
Time-of-day end condition passed to |
|
|
Retrigger limit passed to |
|
|
Reference for end conditions; also accepts |
|
|
Airflow pool name or |
|
|
Enables program stop on the success path. |
|
|
Enables directory removal when |
|
|
Restarts existing programs on an initial Airflow run. |
|
|
Restarts programs in the startup step on retriggered runs; takes precedence over |
Duration fields accept Python timedelta values or numeric seconds and duration
strings in YAML. These fields are passed to HighAvailabilityOperator as
poke_interval, timeout, runtime, endtime, maxretrigger, and
reference_date. The sensor uses mode="poke".
SSH fields¶
Field |
Default |
Meaning |
|---|---|---|
|
Empty string |
Shell commands prepended to remote lifecycle commands. |
|
|
|
|
|
Overrides the local model’s |
ssh_operator_args.ssh_conn_id selects an Airflow SSH connection. Its port is
the SSH port; cfg.port remains the supervisord listen port. cfg.host sets the
XML-RPC target and does not by itself select an SSH connection.
load_airflow_config aliases SupervisorAirflowConfiguration.load.
load_airflow_ssh_config aliases SupervisorSSHAirflowConfiguration.load.
These load supervisor configuration models. The airflow_config.load_config
function loads a collection of declarative DAGs; the
how-to guides show that workflow.
Health checks and diagnostics¶
check_supervisor_health(cfg, require_running=True, supervisor_client=None, **context)
checks an existing instance without configuration, startup, stop, or cleanup.
cfg accepts SupervisorAirflowConfiguration or a dictionary validated as that
model. supervisor_client optionally supplies the XML-RPC client.
With require_running=True, every workload must be STARTING or RUNNING.
With False, the configured exit codes and successful completion states are
accepted. Unhealthy, empty, and unreachable instances raise AirflowException.
A successful result contains host, healthy, and a processes summary.
Event listeners are excluded from workload health and completion checks.
supervisor_process_status XCom contains process name, group, state,
exitstatus, spawnerr, and pid. supervisor_failure stores the last unhealthy
process snapshot. supervisor_log_offsets stores byte cursors and the run ID;
log contents are written to task logs rather than XCom. Final drains read at
most 16 chunks per stream. The observability guide provides
Python and YAML configurations.
Generated API¶
|
|
|
|
Settings that MUST be set when running in airflow |
|
|
|
|
|
|
Check an existing instance without starting, stopping, or reconfiguring it. |
Re-exported supervisor models¶
Convenience layer, settings that MUST be set when running via convenience API |
|
A light wrapper over the supervisor xmlrpc api: http://supervisord.org/api.html |
|