Christmas is near, so let's talk about Airflow's web server.

What i want to say

** It takes a long time to load ** If there is a DAG
The DAG may not be read
You may not be able to access the Airflow management screen
It's okay if there are only a lot of tasks
Can I get around a thousand with the default settings of Componser?
The future is bright

Modules that make up Airflow

The leading web server this time is one of the Airflow modules. For other modules,

scheduler
DAG / Task scheduling and status management
worker
Actually execute the task instance
database
Holding TaskInstance, DagRun, xcom, Variable, etc.
executor
Allocate scheduled tasks to workers

And so on. For more information, Astronomer's article is easy to understand.

What is a web server

webserver is the management screen (shown below), CLI command (part), [API](https://airflow.apache. org / docs / stable / api.html) etc.

Understanding the status of DAG / task instance, etc.
Re-execute DAG / task instance and change status

We accept processing such as.

Internally, it has a Flask + Gunicorn configuration, and the endpoint from the screen is [here](https://github.com/apache/airflow/blob/1.10.2/airflow/www/views. Defined in py).

Airflow UI (The figure is from Airflow official page)

DAG problem that takes a long time to load

The webserver not only accepts requests, but also ** reads DAG files on a regular basis **.

As a result, if the DAG file ** reads ** takes a long time,

The corresponding DAG cannot be read
Heavy access to webserver
I can't even access (error page display)

Sometimes

Is it a warning? It has been.

Does it take a long time to load?

It may be confusing that ** it takes a long time to load the DAG ** and ** it takes a long time to execute DAGRun **, but it is a different story, and the former is the problem this time.

To give an example, this is a ** DAG that takes a long time to load **

    sleep(10000000)
    start = DummyOperator(task_id='start')

This is a ** DAG that takes a long time to run ** DAGRun.

        def hoge():
            sleep(1000000)
        slow_task = PythonOperator(
            task_id='query_' + str(i),
            python_callable=hoge,
        )

Loading can be slow if there are a large number of tasks or if you are accessing the outside ** outside the task.

Flow of webserver parsing DAG

For those who are worried about the detailed flow:

Periodically (*) [Restart child process (gunicorn worker)] when starting webserver Set to (https://github.com/apache/airflow/blob/4a344f13d26ecbb627bb9968895b290bfd86e4da/airflow/cli/commands/webserver_command.py#L146)
[DagBag object is created] when loading the endpoint file (https://github.com/apache/airflow/blob/cb8b2a1dc64c3ea6ba445893c65c6c953dfb476a/airflow/www/views.py#L92)
While the DagBag object is created, DAG file is parsed

Specifically, worker_refresh_interval seconds

Relationship with the number of tasks

Cloud Composer (Airflow 1.10.2) -I tried it with a DAG only for BigQuery Operator:

About 1000 tasks per DAG can be displayed even with the default settings
When there are about 3000 tasks per DAG, it cannot be displayed with the default settings.
Scheduler and Worker will work even if the webserver is heavy (check with Stackdriver)
You can extend the timeout (worker_refresh_interval) or make the load asynchronous (async_dagbag_loader).

If only Graph View or Tree View is heavy, default_dag_run_display_number should be changed.

A bright future story

Some improvements have been proposed for this "loading DAG".

Cloud Composer implements an option to make DAG loading asynchronous on webserver (https://cloud.google.com/composer/docs/how-to/accessing/airflow-web-interface#asynchronous-load) It has also been [ported] to Airflow 1.10.4 (https://issues.apache.org/jira/browse/AIRFLOW-4924).

It's still a draft, but [AIP-24 DAG Persistence in DB using JSON for Airflow Webserver and (optional) Scheduler](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-24+DAG+Persistence+in The proposal (+ DB + using + JSON + for + Airflow + Webserver + and +% 28optional% 29 + Scheduler? FocusedCommentId = 123898950) is a more significant change.

Stop parsing DAGs on webserver
scheduler parses DAG and puts serialization result in DB
webserver uses the result

We are proposing options. (It seems that it is not good that the webserver has a state in the first place It seems that there is a story))

Cloud Composer webserver

A note about the Cloud Composer webserver:

Tenant project (project not managed by customer), running on GAE Flexible with CPU2 core
Specs cannot be changed ...
You can [can] move the webserver to the GKE cluster (https://cloud.google.com/composer/docs/how-to/managing/deploy-webserver?hl=ja)
If changing the Airflow settings doesn't work, it might be worth trying

By the way, Astronomer.io can change the size of vCPU / memory.

[PYTHON] The story of Airflow's webserver and DAG, which takes a long time to load