![]() ![]() => load metadata for docker.io/library/amazonlinux:2 1.6s ![]() ![]() => load build definition from Dockerfile 0.0s Hopefully, you should see something like: Building 543.8s (27/27) FINISHED Go grab your favourite drink and then come back. This will now take around 10 minutes to complete the build process. You do this using the following command (and this works against any version of Apache Airflow that mwaa supports) mwaa-local-env build-image You can now save this file, and you are ready to build your mwaa-local-runner image. Line 757 is changed #dag_dir_list_interval = 300 The next change, still in the airflow.cfg, to update how quickly it picks up DAGs in the DAGs folder. The first is to update the airflow.cfg (docker/config/airflow.cfg) so that we can update the base_url parameter (line 406) with the public IP of your Cloud9 instance. We now need to change a couple of things. You can now check out the mwaa-local-runner project into your Cloud9 environment git clone Run the following command to find yours curl Ĭonfiguring and building mwaa-local-runner (If you do not do this, you will get 400 Gateway errors when accessing Airflow via the UI). We need this as we are going to update the URL configuration parameter in mwaa-local-runner. However, we do need one piece of information before we proceed, and that is the public IP address for your Cloud9 instance. This should be everything you need to get started. Sudo mv docker-compose-$(uname -s)-$(uname -m) /usr/bin/docker-composeĪnd if this is successful, you should now have it available via the command line ops:~/environment $ docker-compose version This is the script I use wget $(uname -s)-$(uname -m) The next thing I need to do is install “docker-compose” as whilst Docker is installed, docker-compose is not. pip3 install -user -upgrade boto3įrom botocore.exceptions import ClientError This is fine for typical use cases, but we are going to be building container images, so need to set this higher. The first thing I needed to do was to increase the size of my local disk as Cloud9 only provides 10gb of storage. What I will cover here is how to deploy mwaa-local-runner onto a standard Cloud9 IDE, deployed in a default VPC. This might not be the most optimised way, so I am very happy to received suggestions on how to improve this. Here is a quick recipe if you are looking to get mwaa-local-runner up and running on your Cloud9 developer setup. ![]()
0 Comments
Leave a Reply. |