API reference¶
Supervisor viewer¶
SupervisorViewerConfiguration defines scheduled collection:
Field |
Default |
Meaning |
|---|---|---|
|
Required, nonempty |
Stable instance IDs mapped to |
|
|
Airflow Variable storing the latest snapshot. |
|
|
Positive socket timeout, at most 30 seconds, for each XML-RPC request. |
|
|
Positive maximum age of the last probe before the UI marks an instance stale. |
Each SupervisorViewerInstance contains cfg (a local or SSH Supervisor
configuration), optional HTTP(S) run_url, and optional HTTP(S) log_url.
program={} is valid when only endpoint settings are supplied. Collection reads
all processes, including event listeners, regardless of configured program names.
collect_supervisor_status(cfg) accepts a model or its serialized dictionary.
It collects up to four instances concurrently, writes one JSON snapshot, and
returns {instance_id: reachable}. Unreachable instances do not fail collection;
storage failures fail the task. XML-RPC clients can retry a disconnected request,
so timeout is not a deadline for the entire collection task.
The snapshot contains collection time, maximum probe age, instance ID, host, port, probe time, last successful observation time, reachability, sanitized request-error type, process details, and configured links. Failed probes retain previous process details for the same instance ID and endpoint. Removed instances disappear on the next collection. Changed host/port settings discard the previous endpoint’s process details. RPC usernames, passwords, and complete Supervisor configurations are not stored in the snapshot.
The viewer reads the Variable selected by AIRFLOW_SUPERVISOR_SNAPSHOT_KEY
(default airflow-supervisor:instances). It refreshes every 30 seconds and never
initiates XML-RPC calls. Its only application route is GET
/airflow-supervisor/; it provides no process-control endpoints.
Display state |
Meaning |
|---|---|
|
The latest probe succeeded and is within |
|
The latest probe failed and is within |
|
The latest probe is older than |
Reachability is separate from individual process state and application readiness. A reachable instance can contain stopped or failed processes. The viewer does not evaluate replica availability or restart programs.
Typed subtask overrides¶
SupervisorTask and SupervisorSSHTask accept subtasks, a mapping of names to
ordinary airflow-pydantic task models. Inline Supervisor and SupervisorSSH
constructors accept the same keyword. A nonempty mapping enables per-program task
generation without requiring program_lifecycle to be set explicitly.
An entry matching a generated task overrides only explicitly supplied fields.
Use a model compatible with its operator: normally PythonTask for program
start/stop operations and SSHTask for SSH daemon/configuration steps. Omitted
callables, commands, and hooks remain generated. Operator type and task identity
cannot be changed through an override.
A complete model under a new name adds a task to the lifecycle. New Python tasks
require a callable; new SSH tasks require a command. An incomplete entry under an
unknown name is rejected. Custom task fields survive model serialization,
airflow-config instantiation, and Python source generation.
Names are local to the lifecycle: start-my-program resolves to the existing
<dag_id>-start-my-program task. Full generated IDs are also accepted. Added tasks
receive the same DAG prefix. Dependencies resolve local names first, then existing
outer DAG task IDs. Attribute references such as check-programs.stop_pass use
the normal task dependency syntax.
Omitting dependencies retains default upstream edges. Supplying it replaces
those edges; [] clears them. Startup and shutdown tasks can therefore be ordered
independently. Unknown dependencies, duplicate aliases, incompatible operator
types, and cycles fail during DAG construction. Whole-lifecycle upstream and
downstream dependencies also cover added roots and leaves.
Shutdown operations still follow the monitoring outcome: changing dependencies does not stop programs or remove configuration during a recovery retrigger.
Per-program lifecycle¶
program_lifecycle=None retains the grouped lifecycle. A mapping, including {},
enables discrete program tasks. Omitted policies receive defaults. Policy keys are
Supervisor process-group names: normally a [program:name] name, or an explicit
[group:name] name when programs are grouped. A numprocs group is controlled as
one unit; event listeners are excluded.
ProgramLifecycleConfiguration accepts:
Field |
Default |
Meaning |
|---|---|---|
|
|
Predecessor program start tasks. Startup ordering is reversed for shutdown. |
|
zero |
Delay from the latest predecessor process start, or run start without predecessors. |
|
|
Additional shutdown ordering constraints. |
|
|
Airflow retries for generated program operation and delay tasks. |
|
30 seconds |
Delay between Airflow task retries. |
|
|
Whether a program failure selects the failed HA outcome. |
|
|
Disable an optional program for the chain after this many failures; must be positive. |
Unknown dependencies and startup/shutdown cycles are rejected. A failure limit
requires required=False. Duration fields accept timedelta or numeric seconds.
stop_on_exit and cleanup still control terminal shutdown and file removal.
The grouped restart_programs property is replaced by individual control tasks
and generated restart control DAGs in this mode.
Supervisor.program_tasks[name] exposes start, delay, and stop operators.
Supervisor.program_operator(name, action, **kwargs) creates a separately composable
control task; action is start, stop, or restart. Task kwargs include retries,
task ID, and DAG. Control tasks update the desired state as well as issuing the
XML-RPC operation.
create_supervisor_control_dags(supervisor_or_dag) returns native DAGs keyed by
their IDs. Pass a managed lifecycle or its native DAG, then register the returned
mapping in the DAG module’s globals. All controls are unscheduled. Program controls
require program_lifecycle to be enabled.
supervisor_program_state contains per-program failure counts, disabled flags,
and manual-control revisions in monitoring XCom and retrigger configuration.
airflow-supervisor:<dag_id>:<program>:control Variables retain explicit start/stop
intent. Airflow 3 tasks use the Task SDK Variable interface.
Lifecycle classes and task models¶
All names below are exported from airflow_supervisor.
Execution |
Lifecycle class |
Configuration model |
|
|---|---|---|---|
Local |
|
|
|
SSH |
|
|
|
Supervisor(dag, cfg, **kwargs) adds lifecycle tasks to an existing Airflow DAG.
cfg accepts a configuration model or a dictionary validated as that model.
Construction requires an active DAG context for internally created tasks.
SupervisorSSH(dag, cfg, host=None, port=None, **kwargs) adds the same lifecycle
with SSH operators for remote configuration and daemon commands. Program
operations and checks use XML-RPC. A supplied Host overrides the SSH target
and XML-RPC host; a supplied Port overrides the supervisord listen port. A
host’s pool is used when cfg.pool is unset. command_prefix can also be passed
as a constructor keyword.
SupervisorTaskArgs contains cfg and inherited Airflow task arguments.
SupervisorSSHTaskArgs adds host and port, accepting Host/Port models,
queries of kind select, or callables and their import paths. SupervisorTask
and SupervisorSSHTask resolve their default operator to the corresponding
lifecycle class.
Resolved SSH host and port values are applied to a copy of cfg before rendering;
the nested HTTP server port matches the selected port. Shared input configurations
are unchanged. Callables that cannot resolve during loading retain runtime resolution.
The SupervisorOperator, SupervisorOperatorArgs, SupervisorSSHOperator, and
SupervisorSSHOperatorArgs names are aliases for the corresponding task models.
DAG constraints and boundaries¶
Both lifecycle classes set catchup=False, concurrency=1,
max_active_tasks=1, and max_active_runs=1 on their DAG.
Generated task IDs use <dag_id>-<step>, independent of a task model’s task_id.
A DAG supports one supervisor lifecycle; multiple programs belong in its
cfg.program mapping.
The exposed boundaries are configure_supervisor, start_supervisor,
start_programs, check_programs, restart_programs, stop_programs,
stop_supervisor, and unconfigure_supervisor. Dependency operators connect
upstream tasks to configuration and downstream tasks to unconfiguration.
supervisor_client returns an XML-RPC client for the lifecycle’s configuration.
Stop and cleanup steps follow the successful monitoring branch. They are not a
general failure finalizer. With cleanup=False, unconfiguration is a skipped
task. With stop_on_exit=False, both program stop and unconfiguration are
skipped, and the default downstream trigger rules skip daemon shutdown.
Configuration models¶
SupervisorAirflowConfiguration extends the re-exported
SupervisorConvenienceConfiguration. SupervisorSSHAirflowConfiguration
extends it with SSH settings.
Supervisor fields¶
Field |
Default |
Meaning |
|---|---|---|
|
Required |
Mapping of program names to |
|
Derived temporary directory |
Instance directory on the worker or SSH host. |
|
|
Configuration path resolved under |
|
|
Host used by the XML-RPC client. |
|
|
Supervisord HTTP/XML-RPC listen address. Integer ports normalize to |
|
|
Supervisord HTTP/XML-RPC credentials. |
|
|
XML-RPC protocol; client ports 80 and 443 select HTTP and HTTPS respectively. |
|
|
XML-RPC request path. |
|
|
Required startup duration, applied to all programs. |
|
|
Startup retries; when set, overrides each program’s value. |
|
|
Accepted program exit statuses, applied to all programs. |
|
|
Stop signal, applied to all programs. |
|
|
Wait before forced termination. |
|
|
Stop and kill the process group. |
|
|
Convenience-command timeout in seconds. |
The convenience model forces program autostart=False and autorestart=False.
Program stdout and stderr files are <working_dir>/<program>/output.log and
error.log. The daemon log and PID file default to supervisord.log and
supervisord.pid in the working directory. Serialized configuration is stored
as pydantic.json; cleanup removes the working directory, including logs.
Monitoring and lifecycle fields¶
preset selects defaults before explicit input values are validated. It accepts
always_on, daily, half_day, or None (the default). Both local and SSH
configuration models support it. Explicit values, including False and None,
override preset values. Omitting preset preserves the ordinary field defaults.
Setting |
|
|
|
|---|---|---|---|
|
7 days |
1 day |
12 hours |
|
|
|
|
|
|
|
|
|
30 seconds |
30 seconds |
30 seconds |
|
8 hours |
8 hours |
8 hours |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Presets do not set the DAG schedule. Runtime uses reference_date; the half-day
cutoff uses the DAG timezone. Stop and cleanup follow the successful lifecycle
branch, as described above.
Field |
Default |
Meaning |
|---|---|---|
|
5 seconds |
Polling interval for the |
|
8 hours |
Sensor timeout. |
|
|
Forwards workload stdout and stderr through the Airflow task logger. |
|
|
Byte budget per stream per poll; |
|
|
Monitoring end condition relative to |
|
|
Time-of-day end condition passed to |
|
|
Retrigger limit passed to |
|
|
Reference for end conditions; also accepts |
|
|
Airflow pool name or |
|
|
Enables program stop on the success path. |
|
|
Enables directory removal when |
|
|
Restarts existing programs on an initial Airflow run. |
|
|
Restarts programs in the startup step on retriggered runs; takes precedence over |
Duration fields accept Python timedelta values or numeric seconds and duration
strings in YAML. These fields are passed to HighAvailabilityOperator as
poke_interval, timeout, runtime, endtime, maxretrigger, and
reference_date. The sensor uses mode="poke".
SSH fields¶
Field |
Default |
Meaning |
|---|---|---|
|
Empty string |
Shell commands prepended to remote lifecycle commands. |
|
|
|
|
|
Overrides the local model’s |
ssh_operator_args.ssh_conn_id selects an Airflow SSH connection. Its port is
the SSH port; cfg.port remains the supervisord listen port. cfg.host sets the
XML-RPC target and does not by itself select an SSH connection.
load_airflow_config aliases SupervisorAirflowConfiguration.load.
load_airflow_ssh_config aliases SupervisorSSHAirflowConfiguration.load.
These load supervisor configuration models. The airflow_config.load_config
function loads a collection of declarative DAGs; the
how-to guides show that workflow.
Health checks and diagnostics¶
check_supervisor_health(cfg, require_running=True, supervisor_client=None, **context)
checks an existing instance without configuration, startup, stop, or cleanup.
cfg accepts SupervisorAirflowConfiguration or a dictionary validated as that
model. supervisor_client optionally supplies the XML-RPC client.
With require_running=True, every workload must be STARTING or RUNNING.
With False, the configured exit codes and successful completion states are
accepted. Unhealthy, empty, and unreachable instances raise AirflowException.
A successful result contains host, healthy, and a processes summary.
Event listeners are excluded from workload health and completion checks.
supervisor_process_status XCom contains process name, group, state,
exitstatus, spawnerr, and pid. supervisor_failure stores the last unhealthy
process snapshot. supervisor_log_offsets stores byte cursors and the run ID;
log contents are written to task logs rather than XCom. Final drains read at
most 16 chunks per stream. The observability guide provides
Python and YAML configurations.
Generated API¶
|
|
|
|
Settings that MUST be set when running in airflow |
|
|
|
|
|
|
Check an existing instance without starting, stopping, or reconfiguring it. |
Re-exported supervisor models¶
Convenience layer, settings that MUST be set when running via convenience API |
|
A light wrapper over the supervisor xmlrpc api: http://supervisord.org/api.html |
|