avhost/mesos-airflow-autoscaler-aws
This AWS autoscaler will start EC2 instances if Airflow does not get matched offers from mesos.
Image
Integration & delivery
Machine learning & AI
Data science
0
6.8K
avhost/mesos-airflow-autoscaler-aws repository overview
AWS Autoscaler for Apache Airflow Mesos Provider
This AWS autoscaler will start EC2 instances if Airflow does not get matched offers from mesos.
Funding
Issues
To open an issue, please use this place: https://github.com/m3scluster/mesos-airflow-autoscaler-aws/issues
Requirements
- Airflow min 2.1.x
- Apache Mesos min 1.6.x
- Airflow Mesos Provider min 2.0
- AWS
Configuration
| ENV | Default | Description |
|---|---|---|
| AIRFLOW_MESOS_SCHEDULER | 127.0.0.1:11000 | IP Address and port of the Apache Airflow Mesos provider |
| LOGLEVEL | debug | Loglevel (info, warn, debug) |
| WAIT_TIME | 2m | The time in minutes the autoscaler have to wait until it will create a mesos instance in AWS |
| WAIT_TIME_OVERWRITE_INSTANCE | 15m | If the DAG is still after 15m in the queue, the autoscaler will ignore the custom instance type to start a new ec2 instance. |
| REDIS_SERVER | 127.0.0.1:6480 | Redis server and port |
| REDIS_PASSWORD | Redis DB password | |
| REDIS_DB | 2 | Redis DB Number |
| REDIS_PREFIX | asg | Prefix for every Redis key |
| API_USERNAME | user | Username to authenticate against the Apache Airflow Mesos provider |
| API_PASSWORD | password | Password to authenticate against the Apache Airflow Mesos provider |
| AWS_SECRET | AWS Secret | |
| AWS_REGION | eu-central-1 | AWS Region |
| AWS_WAIT_TIME | 10m | The time the autoscaler have to wait until it check if the EC2 intance can be terminated. |
| AWS_LAUNCH_TEMPLATE_ID | The AWS Launche Template ID | |
| AWS_INSTANCE_FALLBACK | t3a.2xlarge | Fallback instance type will be used if there are no more ec2 resources in AWS. |
| AWS_INSTANCE_DEFAULT_ARCHITECTURE | x86_64 | Default architecture of ec2 instance. |
| AWS_INSTANCE_ALLOW | "t3.,r5.,c5." | Only these instances are allowed. |
| AWS_INSTANCE_MAX_AGE | 6h | Max age on an instance. That should prevent instances to run endless. |
| MESOS_AGENT_SSL | false | Enable SSL for the communication to the Mesos agent |
| MESOS_AGENT_USERNAME | Username of the Mesos Agent | |
| MESOS_AGENT_PASSWORD | Password of the Mesos Agent | |
| MESOS_AGENT_PORT | 5051 | Port of the Mesos Agent |
| MESOS_AGENT_TIMEOUT | 10m | Mesos agent timeout |
| SSL | false | Enable SSL for the communication to the Airflow Scheduler API |
| DAG_TTL | 6h | Set the TTL for DAG'keys in Redis. |
Airflow Executor Config
These autoscaler support the following executor config parameters:
executor_config={
"mem_limit": 2048,
"instance_type": "t2.xlarge"
}
- With the help if the
mem_limitparameter, the autoscaler will determin which AWS instance is the right one. instance_typewill overwrite the mem_limit and just crate a instance of that type.
Documentation
AWS - https://docs.aws.amazon.com/code-samples/latest/catalog/go-ec2-create_instance.go.html
Tag summary
Content type
Image
Digest
sha256:b4df056bd…
Size
75.7 MB
Last updated
6 months ago
docker pull avhost/mesos-airflow-autoscaler-aws