Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: wbertelsen/purpleair-to-prometheus
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.0.2
Choose a base ref
...
head repository: wbertelsen/purpleair-to-prometheus
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
  • 13 commits
  • 6 files changed
  • 4 contributors

Commits on Aug 30, 2020

  1. Update example-kube.yaml

    wbertelsen authored Aug 30, 2020
    Copy the full SHA
    504e866 View commit details
  2. string changes

    wbertelsen authored Aug 30, 2020
    Copy the full SHA
    615e903 View commit details

Commits on Sep 3, 2020

  1. Add LRAPA and AQandU conversions

    This adds the LRAPA and AQandU conversions
    
    As described on PurpleAir:
    
    Conversions help accomodate different types of pollution with different particle densities.
    For the same reason that wood floats and rocks sink in water, different particles have different densities - for example wild fire smoke vs road dust in the air. This is why a conversion may be needed when calculating the mass of any combination of particulates derived from particle counts.
    
    AQandU: Courtesy of the University of Utah, conversion factors from their study of the PA sensors during winter in Salt Lake City. Visit their web site.
    PM2.5 (µg/m³) = 0.778 x PA + 2.65
    
    LRAPA: Courtesy of the Lane Regional Air Protection Agency, conversion factors from their study of the PA sensors. Visit their web site.
    0 - 65 µg/m³ range:
    LRAPA PM2.5 (µg/m³) = 0.5 x PA (PM2.5 CF=ATM) – 0.66
    wbertelsen committed Sep 3, 2020
    Copy the full SHA
    5f50722 View commit details

Commits on Oct 9, 2020

  1. Copy the full SHA
    0eefb2f View commit details
  2. Copy the full SHA
    5689589 View commit details
  3. Don't let AQI go below 0

    This is a speculative fix for #3
    wbertelsen committed Oct 9, 2020
    Copy the full SHA
    44bc235 View commit details
  4. Revert "Don't let AQI go below 0"

    This reverts commit 44bc235.
    wbertelsen committed Oct 9, 2020
    Copy the full SHA
    b44054a View commit details
  5. Don't let AQI go below 0 (#7)

    This is a speculative fix for #3
    wbertelsen authored Oct 9, 2020
    Copy the full SHA
    8d39315 View commit details
  6. Stop exporting when there's an exception (#8)

    * Return early to reduce indented lines.
    
    * Remove metrics when there's an error.
    
    Fixes #4
    
    * Catch exception from Gauge.remove.
    
    * Fix error from git merge.
    
    * Fix another merge error.
    
    * Additional fix.
    yegle authored Oct 9, 2020
    Copy the full SHA
    00013dd View commit details

Commits on Dec 12, 2020

  1. Remove unused variable

    wbertelsen committed Dec 12, 2020
    Copy the full SHA
    ce23703 View commit details
  2. Add flake8 github action (#11)

    * Add flake8 github action
    
    * dont test
    wbertelsen authored Dec 12, 2020
    Copy the full SHA
    3c55864 View commit details
  3. Add codeql action (#12)

    * Add codeql action
    
    * dont run on PRs
    wbertelsen authored Dec 12, 2020
    Copy the full SHA
    34efb27 View commit details

Commits on Dec 19, 2020

  1. Fix the threshold colors in grafana (#13)

    This fixes #10
    wbertelsen authored Dec 19, 2020
    Copy the full SHA
    b8a83b5 View commit details
Showing with 283 additions and 280 deletions.
  1. +64 −0 .github/workflows/codeql-analysis.yml
  2. +36 −0 .github/workflows/python-package.yml
  3. +1 −1 example-kube.yaml
  4. +89 −240 grafana-dashboard.json
  5. BIN grafana-screenshot.png
  6. +93 −39 purple_to_prom.py
64 changes: 64 additions & 0 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL"

on:
push:
branches: [ main ]
schedule:
- cron: '42 2 * * 0'

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
language: [ 'python' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]
# Learn more:
# https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed

steps:
- name: Checkout repository
uses: actions/checkout@v2

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v1
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main

# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v1

# ℹ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl

# ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
# and modify them (or add more) to build your code if your project
# uses a compiled language

#- run: |
# make bootstrap
# make release

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
36 changes: 36 additions & 0 deletions .github/workflows/python-package.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Python package

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
build:

runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.8']

steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
2 changes: 1 addition & 1 deletion example-kube.yaml
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ spec:
spec:
containers:
- name: purpleair-to-prometheus
image: wbertelsen/purpleair-to-prometheus:v0.0.0-4
image: wbertelsen/purpleair-to-prometheus:latest
command:
- "./purple_to_prom.py"
- "--sensor-ids"
329 changes: 89 additions & 240 deletions grafana-dashboard.json

Large diffs are not rendered by default.

Binary file modified grafana-screenshot.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
132 changes: 93 additions & 39 deletions purple_to_prom.py
Original file line number Diff line number Diff line change
@@ -22,6 +22,7 @@
# SOFTWARE.

import time
import traceback
from typing import List


@@ -33,82 +34,135 @@


aqi_g = prometheus_client.Gauge(
'purpleair_pm_25_10m_iaqi', 'iAQI (10 min average)', ['parent_sensor_id', 'sensor_id', 'sensor_name']
'purpleair_pm_25_10m_iaqi', 'iAQI (10 min average)',
['parent_sensor_id', 'sensor_id', 'sensor_name']
)
aqi_AQandU_g = prometheus_client.Gauge(
'purpleair_pm_25_10m_iaqi_AQandU', 'iAQI (10 min average) w/ AQandU correction',
['parent_sensor_id', 'sensor_id', 'sensor_name']
)
aqi_LRAPA_g = prometheus_client.Gauge(
'purpleair_pm_25_10m_iaqi_LRAPA', 'iAQI (10 min average) w/ LRAPA correction',
['parent_sensor_id', 'sensor_id', 'sensor_name']
)
temp_g = prometheus_client.Gauge(
'purpleair_temp_f', 'Sensor temp reading (degrees Fahrenheit)', ['parent_sensor_id', 'sensor_id', 'sensor_name']
'purpleair_temp_f', 'Sensor temp reading (degrees Fahrenheit)',
['parent_sensor_id', 'sensor_id', 'sensor_name']
)
humidity_g = prometheus_client.Gauge(
'purpleair_humidity_pct', 'Sensor humidity reading (percent)', ['parent_sensor_id', 'sensor_id', 'sensor_name']
'purpleair_humidity_pct', 'Sensor humidity reading (percent)',
['parent_sensor_id', 'sensor_id', 'sensor_name']
)
pressure_g = prometheus_client.Gauge(
'purpleair_pressure_mb', 'Sensor pressure reading (Millibars)', ['parent_sensor_id', 'sensor_id', 'sensor_name']
'purpleair_pressure_mb', 'Sensor pressure reading (millibars)',
['parent_sensor_id', 'sensor_id', 'sensor_name']
)

def clear_metrics():
# NOTE: there's no official way to support it unless we convert this script
# to a "custom collector".
# See https://github.com/prometheus/client_python/issues/277
for g in [aqi_g, aqi_AQandU_g, aqi_LRAPA_g, temp_g, pressure_g,
humidity_g]:
with g._lock():
g._metrics.clear()


def check_sensor(parent_sensor_id: str) -> None:
resp = requests.get("https://www.purpleair.com/json?show={}".format(parent_sensor_id))
if resp.status_code == 200:
if resp.status_code != 200:
clear_metrics()
raise Exception("got {} responde code from purpleair".format(resp.status_code))

try:
resp_json = resp.json()
except ValueError:
clear_metrics()
raise
for sensor in resp_json.get("results"):
sensor_id = sensor.get("ID")
name = sensor.get("Label")
stats = sensor.get("Stats")
temp_f = sensor.get("temp_f")
humidity = sensor.get("humidity")
pressure = sensor.get("pressure")
try:
resp_json = resp.json()
except ValueError:
return
for sensor in resp_json.get("results"):
sensor_id = sensor.get("ID")
name = sensor.get("Label")
stats = sensor.get("Stats")
temp_f = sensor.get("temp_f")
humidity = sensor.get("humidity")
pressure = sensor.get("pressure")
if stats:
stats = json.loads(stats)
pm25_10min = stats.get("v1")
if pm25_10min:
i_aqi = aqi.to_iaqi(aqi.POLLUTANT_PM25, pm25_10min, algo=aqi.ALGO_EPA)
pm25_10min_raw = stats.get("v1")
if pm25_10min_raw:
pm25_10min = max(float(pm25_10min_raw), 0)
i_aqi = aqi.to_iaqi(aqi.POLLUTANT_PM25, str(pm25_10min), algo=aqi.ALGO_EPA)
aqi_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(i_aqi)
if temp_f:
temp_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(float(temp_f))
if pressure:
pressure_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(float(pressure))
if humidity:
humidity_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(float(humidity))
else:
raise Exception("got {} from purpleair".format(resp.status_code))

# https://www.aqandu.org/airu_sensor#calibrationSection
pm25_10min_AQandU = 0.778 * float(pm25_10min) + 2.65
i_aqi_AQandU = aqi.to_iaqi(aqi.POLLUTANT_PM25, str(pm25_10min_AQandU), algo=aqi.ALGO_EPA)
aqi_AQandU_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(i_aqi_AQandU)

# https://www.lrapa.org/DocumentCenter/View/4147/PurpleAir-Correction-Summary
pm25_10min_LRAPA = max(0.5 * float(pm25_10min) - 0.66, 0)
i_aqi_LRAPA = aqi.to_iaqi(aqi.POLLUTANT_PM25, str(pm25_10min_LRAPA), algo=aqi.ALGO_EPA)
aqi_LRAPA_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(i_aqi_LRAPA)

if temp_f:
temp_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(float(temp_f))
if pressure:
pressure_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(float(pressure))
if humidity:
humidity_g.labels(
parent_sensor_id=parent_sensor_id, sensor_id=sensor_id, sensor_name=name
).set(float(humidity))
except Exception:
try:
# Stop exporting metrics, instead of showing as a flat line.
aqi_g.remove(parent_sensor_id, sensor_id, name)
aqi_AQandU_g.remove(parent_sensor_id, sensor_id, name)
aqi_LRAPA_g.remove(parent_sensor_id, sensor_id, name)
temp_g.remove(parent_sensor_id, sensor_id, name)
pressure_g.remove(parent_sensor_id, sensor_id, name)
humidity_g.remove(parent_sensor_id, sensor_id, name)
except KeyError:
# No data produced yet. Silently ignore it.
pass
raise


def poll(sensor_ids: List[str], refresh_seconds: int) -> None:
while True:
print("refreshing...", flush=True)
print("refreshing sensors...", flush=True)
for sensor_id in sensor_ids:
try:
check_sensor(sensor_id)
except Exception as e:
print(e)
print("Got error, skipping rest of poll")
except Exception:
traceback.print_exc()
print("Error fetching sensor data, sleeping till next poll")
break
time.sleep(refresh_seconds)


def main():
parser = argparse.ArgumentParser(
description="Get's sensor data from purple air, converts it to AQI, and exports it to prometheus"
description="Gets sensor data from purple air, converts it to AQI, and exports it to prometheus"
)
parser.add_argument('--sensor-ids', nargs="+", help="Sensors to collect from", required=True)
parser.add_argument("--port", type=int, help="What port to serve prometheus on", default=9760)
parser.add_argument("--port", type=int, help="What port to serve prometheus metrics on", default=9760)
parser.add_argument("--refresh-seconds", type=int, help="How often to refresh", default=60)
args = parser.parse_args()

prometheus_client.start_http_server(args.port)

print("Serving prometheus on {}".format(args.port))
print("Serving prometheus metrics on {}/metrics".format(args.port))
poll(args.sensor_ids, args.refresh_seconds)