Initial Commit for Airflow Prometheus Exporter

.dockerignore

@@ -0,0 +1,30 @@
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/

.gitignore

@@ -0,0 +1,27 @@
+.DS_Store
+*.pyc
+*$py.class
+*~
+dist/
+*.egg-info
+*.egg
+*.eggs/
+*.egg/
+*.cache/
+build/
+.build/
+_build/
+pip-log.txt
+.directory
+erl_crash.dump
+*.db
+db.sqlite3
+Documentation/
+.tox/
+.idea/
+.coverage
+.ve*
+cover/
+coverage.*
+htmlcov/
+.vagrant/

.travis.yml

@@ -0,0 +1,9 @@
+# Config file for automatic testing at travis-ci.org
+dist: xenial
+sudo: false
+language: python
+python:
+  - "3.6"
+  - "3.7"
+install: pip install tox-travis
+script: tox

LICENSE

@@ -0,0 +1,53 @@
+Copyright (c) 2017-2019, Robinhood Markets, Inc.
+All rights reserved.
+
+Airflow Prometheus Exporter is licensed under The BSD License (3 Clause, also known as
+the new BSD license).  The license is an OSI approved Open Source
+license and is GPL-compatible(1).
+
+The license text can also be found here:
+http://www.opensource.org/licenses/BSD-3-Clause
+
+License
+=======
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+    * Neither the name of Robinhood Markets, Inc. nor the
+      names of its contributors may be used to endorse or promote products
+      derived from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL Robinhood Markets, Inc. OR CONTRIBUTORS
+BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.
+
+Documentation License
+=====================
+
+The documentation portion of Airflow Prometheus Exporter (the rendered contents of the
+"docs" directory of a software distribution or checkout) is supplied
+under the "Creative Commons Attribution-ShareAlike 4.0
+International" (CC BY-SA 4.0) License as described by
+http://creativecommons.org/licenses/by-sa/4.0/
+
+Footnotes
+=========
+(1) A GPL-compatible license makes it possible to
+    combine Airflow Prometheus Exporter with other software that is released
+    under the GPL, it does not mean that we're distributing
+    Faust under the GPL license.  The BSD license, unlike the GPL,
+    let you distribute a modified version without making your
+    changes open source.

README.md

@@ -0,0 +1,71 @@
+# Airflow Prometheus Exporter
+
+The Airflow Prometheus Exporter exposes various metrics about the Scheduler, DAGs and Tasks which helps improve the observability of an Airflow cluster.
+
+The exporter is based on this [prometheus exporter for Airflow](https://github.com/epoch8/airflow-exporter).
+
+## Requirements
+
+The plugin has been tested with:
+
+- Airflow >= 1.10.4
+- Python 3.6+
+
+The scheduler metrics assume that there is a DAG named `canary_dag`. In our setup, the `canary_dag` is a DAG which has a tasks which perform very simple actions such as establishing database connections. This DAG is used to test the uptime of the Airflow scheduler itself.
+
+## Installation
+
+The exporter can be installed as an Airflow Plugin using:
+
+```pip install airflow-prometheus-exporter```
+
+This should ideally be installed in your Airflow virtualenv.
+
+## Metrics
+
+Metrics will be available at
+
+`http://<your_airflow_host_and_port>/admin/metrics/`
+
+### Task Specific Metrics
+
+#### `airflow_task_status`
+
+Number of tasks with a specific status.
+
+All the possible states are listed [here](https://github.com/apache/airflow/blob/master/airflow/utils/state.py#L46).
+
+#### `airflow_task_duration`
+
+Duration of successful tasks in seconds.
+
+#### `airflow_task_fail_count`
+
+Number of times a particular task has failed.
+
+### Dag Specific Metrics
+
+#### `airflow_dag_status`
+
+Number of DAGs with a specific status.
+
+All the possible states are listed [here](https://github.com/apache/airflow/blob/master/airflow/utils/state.py#L59)
+
+#### `airflow_dag_run_duration`
+Duration of successful DagRun in seconds.
+
+### Scheduler Metrics
+
+#### `airflow_dag_scheduler_delay`
+
+Scheduling delay for a DAG Run in seconds. This metric assumes there is a `canary_dag`.
+
+The scheduling delay is measured as the delay between when a DAG is marked as `SCHEDULED` and when it actually starts `RUNNING`.
+
+#### `airflow_task_scheduler_delay`
+
+Scheduling delay for a Task in seconds. This metric assumes there is a `canary_dag`.
+
+#### `airflow_num_queued_tasks`
+
+Number of tasks in the `QUEUED` state at any given instance.

airflow_prometheus_exporter/__init__.py

@@ -0,0 +1,7 @@
+# -*- coding: utf-8 -*-
+
+"""Top-level package for Airflow Prometheus Exporter."""
+
+__author__ = """Robinhood Markets, Inc."""
+__email__ = '[email protected]'
+__version__ = '1.0.1'