установка Apache Airflow Windows 10
Julie Followell
This procedure can be useful for learning and exploration. However, adapting it for use in real-world situations can be complicated. Making changes to this procedure will require specialized expertise in Docker & Docker Compose, and the Airflow community may not be able to help you.
Install Docker Community Edition (CE) on your workstation. Depending on your OS, you may need to configure Docker to use at least 4.00 GB of memory for the Airflow containers to run properly. Please refer to the Resources section in the Docker for Windows or Docker for Mac documentation for more information.
The default amount of memory available for Docker on macOS is often not enough to get Airflow up and running.If enough memory is not allocated, it might lead to the webserver continuously restarting.You should allocate at least 4GB memory for the Docker Engine (ideally 8GB).
Some operating systems (Fedora, ArchLinux, RHEL, Rocky) have recently introduced Kernel changes that result inAirflow in Docker Compose consuming 100% memory when run inside the community Docker implementation maintainedby the OS teams.
On Linux, the quick-start needs to know your host user id and needs to have group id set to 0.Otherwise the files created in dags, logs and plugins will be created with root user ownership.You have to make sure to configure them for the docker-compose:
For other operating systems, you may get a warning that AIRFLOW_UID is not set, but you cansafely ignore it. You can also manually create an .env file in the same folder asdocker-compose.yaml with this content to get rid of the warning:
When you want to run Airflow locally, you might want to use an extended image, containing some additional dependencies - forexample you might add new python packages, or upgrade airflow providers to a later version. This can be done very easilyby specifying build: . in your docker-compose.yaml and placing a custom Dockerfile alongside yourdocker-compose.yaml. Then you can use docker compose build commandto build your image (you need to do it only once). You can also add the --build flag to your docker compose commandsto rebuild the images on-the-fly when you run other docker compose commands.
Creating custom images means that you need to maintain also a level of automation as you need to re-create the imageswhen either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts.Also keep in mind, that in cases when you run pure Python tasks, you can use thePython Virtualenv functions which willdynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached.
Usual case for custom images, is when you want to add a set of requirements to it - usually stored inrequirements.txt file. For development, you might be tempted to add it dynamically when you arestarting the original airflow image, but this has a number of side effects (for example your containerswill start much slower - each additional dependency will further delay your containers start up time).Also it is completely unnecessary, because docker compose has the development workflow built-in.You can - following the previous chapter, automatically build and use your custom image when youiterate with docker compose locally. Specifically when you want to add your own requirement file,you should do those steps:
Comment out the image: ... line and remove comment from the build: . line in thedocker-compose.yaml file. The relevant part of the docker-compose file of yours should look similarto (use correct image tag):
It is the best practice to install apache-airflow in the same version as the one that comes from theoriginal image. This way you can be sure that pip will not try to downgrade or upgrade apacheairflow while installing other requirements, which might happen in case you try to add a dependencythat conflicts with the version of apache-airflow that you are using.
In general, if you want to use Airflow locally, your DAGs may try to connect to servers which are running on the host. In order to achieve that, an extra configuration must be added in docker-compose.yaml. For example, on Linux the configuration must be in the section services: airflow-worker adding extra_hosts: - "host.docker.internal:host-gateway"; and use host.docker.internal instead of localhost. This configuration vary in different platforms. Please check the Docker documentation for Windows and Mac for further information.
UID of the user to run Airflow containers as.Override if you want to use non-default AirflowUID (for example when you map folders from host,it should be set to result of id -u call.When it is changed, a user with the UID iscreated with default name inside the containerand home of the use is set to /airflow/home/in order to share Python libraries installed there.This is in order to achieve the OpenShiftcompatibility. See more in theArbitrary Docker User
Those additional variables are useful in case you are trying out/testing Airflow installation via Docker Compose.They are not intended to be used in production, but they make the environment faster to bootstrap for first timeusers with the most common customizations.
Username for the administrator UI account.If this value is specified, admin UI user getscreated automatically. This is only useful whenyou want to run Airflow for a test-drive andwant to start a container with embedded developmentdatabase.
For many (most?) Python codebases, running on Windows is reasonable enough. For data, Anaconda even makes it easy - create an environment, install your library and go. Unfortunately, Airbnb handed us a pathologically non-portable codebase. I was flabbergasted to find that casually trying to run Airflow on Windows resulted in a bad shim script, a really chintzy pathing bug, a symlinking issue* and an attempt to use the Unix-only passwords database.
So running Airflow in Windows natively is dead in the water, unless you want to spend a bunch of months rewriting a bunch of the logic and arguing with the maintainers**. Luckily, there are two fairly sensible alternate approaches to consider which will let you run Airflow on a Windows machine: WSL and Docker.
Docker is a tool for managing Linux containers, which are a little like virtual machines without the virtualization, making them act like self-contained machines but much more lightweight than a full VM. Surprisingly it works on Windows - casually, even.
Brief sidebar: Docker isn't a silver bullet, and honestly it's kind of a pain in the butt. I personally find it tough to debug and its aggressive caching makes both cache busting and resource clearing difficult. Even so, the alternatives - such as Vagrant - are generally worse. Docker is also a pseudo-standard and Kubernetes - the heinously confusing thing your DevOps team makes you deploy to - works with Docker images, so it's overall a useful tool to reach for especially for problems like this one.
Docker containers can be ran in two ways: either in a bespoke capacity via the command line, or using a tool called Docker Compose that takes a yaml file which specifies which containers to run and how, and then does what's needed. For a single container the command line is often the thing you want - and we use it later on - but for a collection of services that need to talk to each other, Docker Compose is what we need.
First of all, we create three services: a metadb, a scheduler and a webserver. Architecturally, Airflow stores its state in a database (the metadb), the scheduler process connects to that database to figure out what to run when, and the webserver process puts a web UI in front of the whole thing. Individual jobs can connect to other databases, such as RedShift, to do actual ETL.
Docker containers are created based on Docker images, which hold the starting state for a container. We use two images here: apache/airflow, the official Airflow image, and postgres, the official PostgreSQL image.
The metadb implementation is pluggable and supports most SQL databases via SQLAlchemy. Airflow uses SQLite by default, but in practice most people either use MySQL or PostgreSQL. I'm partial to the latter, so I chose to set it up here.
On the PostgreSQL side: you need to configure it to have a user and database that Airflow can connect to. The Docker image supports this via environment variables. There are many variables that are supported, but the ones I used are POSTGRES_USER, POSTGRES_PASSWORD and POSTGRES_DB. By setting all of these to airflow, I ensured that there was a superuser named airflow, with a password of airflow and a default database of airflow.
Note that you'll definitely want to think about this harder before you go to production. Database security is out of scope of this post, but you'll probably want to create a regular user for Airflow, set up secrets management with your deploy system, and possibly change the authentication backend. Your DevOps team, if you have one, can probably help you here.
Docker has containers connect over virtual networks. Practically speaking, this means that you have to make sure that any containers that need to talk to each other are all connected to the same network (named "airflow" in this example), and that any containers that you need to talk to from outside have their ports explicitly exposed. You'll definitely want to expose port 8080 of the webserver to your host so that you can visit the UI in your browser. You may want to expose PostgreSQL as well, though I haven't done that here.
Finally, by default Docker Compose won't bother to restart a container if it crashes. This may be desired behavior, but in my case I wanted them to restart unless I told them to stop, and so set it to unless-stopped.
When Airflow starts it looks for a file called airflow.cfg inside of the AIRFLOW_HOME directory, which is ini-formatted and which is used to configure Airflow. This file supports a number of options, but the only one we need for now is core.sql_alchemy_conn. This field contains a SQLAlchemy connection string for connecting to PostgreSQL.
d3342ee215