Skip to content

Commit

Permalink
Merge pull request #136 from powerapi-ng/fix/correct-docker-deploy-co…
Browse files Browse the repository at this point in the history 
…nfigurations-issues

refactor(deploy): Cleanup deployment configurations
  • Loading branch information
@gfieni
gfieni authored Jul 23, 2024
2 parents 032ecd8 + 2a37068 commit e6ca260
Show file tree
Hide file tree
Showing 11 changed files with 69 additions and 219 deletions.
29 changes: 12 additions & 17 deletions deploy/docker/.env
View file
Original file line number Diff line number Diff line change
@@ -1,29 +1,24 @@
# Sensor selection
# Available options: hwpc
SENSOR_TYPE=hwpc
SENSOR_IMAGE=powerapi/${SENSOR_TYPE}-sensor:1.3.0
# Sensor image
SENSOR_IMAGE=powerapi/hwpc-sensor:${HWPC_SENSOR_VERSION:-latest}

# Formula selection
# Available options: smartwatts
FORMULA_TYPE=smartwatts
FORMULA_IMAGE=powerapi/${FORMULA_TYPE}-formula:2.2.0
# Formula image
FORMULA_IMAGE=powerapi/smartwatts-formula:${SMARTWATTS_VERSION:-latest}

# Source selection
# Available options: mongodb, socket
POWERAPI_SOURCE=socket
POWERAPI_SOURCE=mongodb

# Destination selection
# Available options: influxdb2, prometheus, mongodb, csv, filedb
POWERAPI_DESTINATION=prometheus
# Available options: influxdb2, prometheus, mongodb, csv
POWERAPI_DESTINATION=mongodb

# Third party images
MONGO_IMAGE=mongo:7.0.8
MONGOEXPRESS_IMAGE=mongo-express:1.0.2-20
INFLUXDB_IMAGE=influxdb:2.7.5
GRAFANA_IMAGE=grafana/grafana:10.1.9
PROMETHEUS_IMAGE=prom/prometheus:v2.51.2
MONGO_IMAGE=mongo:latest
MONGOEXPRESS_IMAGE=mongo-express:latest
INFLUXDB_IMAGE=influxdb:latest
GRAFANA_IMAGE=grafana/grafana:latest
PROMETHEUS_IMAGE=prom/prometheus:latest
CSV_IMAGE=busybox:stable-glibc
FILEDB_IMAGE=busybox:stable-glibc
SOCKET_IMAGE=busybox:stable-glibc

# Docker compose profiles
Expand Down
152 changes: 35 additions & 117 deletions deploy/docker/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
# PowerAPI Docker Toolchain

This folder contains several resources to test and configure PowerAPI on Docker based hosting platform using docker-compose.
This folder contains several resources to test and configure PowerAPI on Docker based hosting platform by using `docker compose`.


**NOTE:**
* This toolchain is meant for running quick tests
**NOTE:**
* This toolchain is meant for running quick tests
of the PowerAPI energy metering solution
and is not suitable for detailed analysis.
* For simplicity sake all components are deployed on a single host.
* For simplicity sake all components are deployed on a single host.
While it makes it much easier to quickly experiment with PowerAPI monitoring,
it should not be used when running actual tests. You should only run the sensor
on your target systems and all other components on an independent system
Expand All @@ -17,41 +17,41 @@ This folder contains several resources to test and configure PowerAPI on Docker

## Prerequisites

* a linux system (tested on ubuntu 20.04 and 22.04)
* docker
* docker-compose
* internet access for docker registry
* a Linux system (tested on Ubuntu 20.04, Ubuntu 22.04, Debian 12)
* `docker`
* `docker compose`
* Internet access for `docker` registry

## Quick Start

No build step is needed, the default settings based on [mongodb/prometheus](#mongodb_prometheus) configuration can be launched "Out of the box" with command below:
No build step is needed, the default settings based on [mongodb/mongodb](#mongodb_mongodb) configuration can be launched "Out of the box" with command below:

```
docker-compose up
docker compose up
```

Always stop properly components to unmount associated resources:

```
docker-compose down
docker compose down
```

For more accurate measurements, you should configure the formula according to the cpu on you host. This can be done by configuring CPU profile in _./formula_ folder configuration files.
Look at [the offical powerapi documentation](https://powerapi-ng.github.io/howto_monitor_process/deploy_formula.html) to see which value to use for `cpu-ratio-base`,`cpu-ratio-min` and `cpu-ratio-max`.
For more accurate measurements, you should configure the formula according to the CPU on your host. This can be done by configuring CPU profile in _./formula_ folder configuration files.
Look at [the offical powerapi documentation](https://powerapi.org/reference/formulas/smartwatts/#parameters) to check CPU related values to define. In particular, you need to indicate `cpu-base-freq`.

Please notice that you also need to adapt the sensor configuration file in _./sensor_ to indicate suitable `core events` according to your [CPU family](https://powerapi.org/reference/sensors/hwpc-sensor/#events).

## Configuration

Based on docker-compose **profiles** and **.env** file, several configurations are available based on [Sources/Destinations](https://powerapi.org/reference/database/sources_destinations/) matrix:
Based on `docker compose` **profiles** and **.env** file, several configurations are available based on [Sources/Destinations](https://powerapi.org/reference/database/sources_destinations/) matrix:

| Source | Destination | |
|---------|-------------|-----------------------------|
| socket | csv | [view](#socket_csv) |
| socket | filedb | [view](#socket_filedb) |
| socket | influxdb2 | [view](#socket_influxdb2) |
| socket | mongodb | [view](#socket_mongodb) |
| socket | prometheus | [view](#socket_prometheus) |
| mongodb | csv | [view](#mongodb_csv) |
| mongodb | filedb | [view](#mongodb_filedb) |
| mongodb | influxdb2 | [view](#mongodb_influxdb2) |
| mongodb | mongodb | [view](#mongodb_mongodb) |
| mongodb | prometheus | [view](#mongodb_prometheus) |
Expand All @@ -63,24 +63,18 @@ Source and Destination selection can be done editing **.env** file as follow:
POWERAPI_SOURCE=mongodb

# Destination selection
# Available options: influxdb2, prometheus, mongodb, csv, filedb
POWERAPI_DESTINATION=prometheus
# Available options: influxdb2, prometheus, mongodb, csv
POWERAPI_DESTINATION=mongodb
```

Currently, only [HWPC Sensor](https://powerapi.org/reference/sensors/hwpc-sensor/) and [SmartWatts Formula](https://powerapi.org/reference/formulas/smartwatts/) are supported and can be modified with properties below:
```properties
# Sensor selection
# Available options: hwpc
SENSOR_TYPE=hwpc
SENSOR_IMAGE=powerapi/${SENSOR_TYPE}-sensor:1.3.0

# Formula selection
# Available options: smartwatts
FORMULA_TYPE=smartwatts
FORMULA_IMAGE=powerapi/${FORMULA_TYPE}-formula:2.2.0
You can also select versions for [HWPC Sensor](https://powerapi.org/reference/sensors/hwpc-sensor/) and [SmartWatts Formula](https://powerapi.org/reference/formulas/smartwatts/) via `HWPC_SENSOR_VERSION` and `SMARTWATTS_VERSION` environment variables (for both, default version is `latest`) with the command below:

```
HWPC_SENSOR_VERSION=1.3.0 SMARTWATTS_VERSION=2.2.0 docker compose up
```


At last, it's also possible to use specific images for third party components:
At last, it's also possible to use specific images for third party components (default version is `latest`) by editing **.env** file:
```properties
# Third party images
MONGO_IMAGE=mongo:7.0.8
Expand All @@ -89,12 +83,11 @@ INFLUXDB_IMAGE=influxdb:2.7.5
GRAFANA_IMAGE=grafana/grafana:10.1.9
PROMETHEUS_IMAGE=prom/prometheus:v2.51.2
CSV_IMAGE=busybox:stable-glibc
FILEDB_IMAGE=busybox:stable-glibc
SOCKET_IMAGE=busybox:stable-glibc
```

**NOTE:**
* .env mechanism is native to docker-compose, this way, environnement is loaded by docker-compose
**NOTE:**
* .env mechanism is native to `docker compose`, this way, environment is loaded by `docker compose`
without manual operations,
* profiles mechanism allows to deploy only required components without editing docker-compose.yml,
* each configuration is detailed below for further information on usage and settings.
Expand Down Expand Up @@ -136,42 +129,6 @@ An additional service **socket** is created and started to allow custom treatmen

An additional service **csv** is created and started to allow custom treatments on ./csv folder and files, it can be edited using specific image **CSV_IMAGE** (default set to busybox) and using custom command attribute.

#### <a name="socket_filedb"></a> Source: Socket - Destination: FileDB

<!--
@startuml plantuml/socket_filedb
actor user
package host {
cloud {
component sensor
component formula
component filedb
component socket
}
folder "./filedb" as filedb_folder
file "./sensor/hwpc-socket.json" as sensor_conf
file "./formula/smartwatts-socket-filedb.json" as formula_conf
}
sensor <-[#black,dashed]- sensor_conf
formula <-[#black,dashed]- formula_conf
formula <-[#red]- sensor
formula <-[#black]- socket
filedb_folder <-[#blue]- formula
filedb_folder <-[#black]- filedb
filedb<-[#black]- filedb_folder
user <-[#black]- filedb_folder
@enduml
-->

![](./plantuml/socket_filedb.svg)

An additional service **socket** is created and started to allow custom treatments on sockets, it can be edited using specific image **SOCKET_IMAGE** (default set to busybox) and using custom command attribute.

An additional service **filedb** is created and started to allow custom treatments on ./filedb folder and files, it can be edited using specific image **FILEDB_IMAGE** (default set to busybox) and using custom command attribute.

#### <a name="socket_influxdb2"></a> Source: Socket - Destination: InfluxDB2

Expand Down Expand Up @@ -200,15 +157,15 @@ formula <-[#black]- socket
influxdb2 <-[#blue]- formula
grafana <-[#black]- influxdb2
grafana <-[#black]- influxdb2
user <-[#black]- grafana : http://localhost:3000
user <-[#black]- influxdb2 : http://localhost:8086
@enduml
-->

![](./plantuml/socket_influxdb2.svg)

Report can be viewed in Grafana dashboard with two panels:
* the top panel represents the energy consumption fo each constainer running on your host.
* the bottom panel represents the energy consumption for the whole CPU as reported directly by the CPU (in green) and as estimated by powerAPI (in yellow).
Expand Down Expand Up @@ -237,7 +194,7 @@ formula <-[#black,dashed]- formula_conf
formula <-[#red]- sensor
formula <-[#black]- socket
mongodb <-[#blue]- formula
mongodb <-[#blue]- formula
mongo_express <-[#black]- mongodb
Expand All @@ -255,7 +212,7 @@ An additional service **socket** is created and started to allow custom treatmen

<!--
@startuml plantuml/socket_prometheus
actor user
actor user
package host {
cloud {
component sensor
Expand Down Expand Up @@ -335,45 +292,6 @@ user <-[#black]- mongo_express : http://localhost:8081

An additional service **csv** is created and started to allow custom treatments on ./csv folder and files, it can be edited using specific image **CSV_IMAGE** (default set to busybox) and using custom command attribute.

#### <a name="mongodb_filedb"></a> Source: MongoDB - Destination: FileDB

<!--
@startuml plantuml/mongodb_filedb
actor user
package host {
cloud {
component sensor
component formula
component filedb
component mongodb
component mongo_express
}
folder "./filedb" as filedb_folder
file "./sensor/hwpc-mongodb.json" as sensor_conf
file "./formula/smartwatts-mongodb-filedb.json" as formula_conf
}
sensor <-[#black,dashed]- sensor_conf
formula <-[#black,dashed]- formula_conf
mongodb <-[#red]- sensor
formula <-[#blue]- mongodb
filedb_folder <-[#blue]- formula
filedb_folder <-[#black]- filedb
filedb<-[#black]- filedb_folder
mongo_express <-[#black]- mongodb
user <-[#black]- filedb_folder
user <-[#black]- mongo_express : http://localhost:8081
@enduml
-->

![](./plantuml/mongodb_filedb.svg)

An additional service **filedb** is created and started to allow custom treatments on ./filedb folder and files, it can be edited using specific image **FILEDB_IMAGE** (default set to busybox) and using custom command attribute.

#### <a name="mongodb_influxdb2"></a> Source: MongoDB - Destination: InfluxDB2

<!--
Expand Down Expand Up @@ -412,11 +330,11 @@ user <-[#black]- influxdb2 : http://localhost:8086
-->

![](./plantuml/mongodb_influxdb2.svg)

Report can be viewed in Grafana dashboard with two panels:
* the top panel represents the energy consumption fo each constainer running on your host.
* the bottom panel represents the energy consumption for the whole CPU as reported directly by the CPU (in green) and as estimated by powerAPI (in yellow).

#### <a name="mongodb_mongodb"></a> Source: MongoDB - Destination: MongoDB

<!--
Expand All @@ -438,7 +356,7 @@ formula <-[#black,dashed]- formula_conf
mongodb <-[#red]- sensor
formula <-[#blue]- mongodb
mongodb <-[#blue]- formula
mongodb <-[#blue]- formula
mongo_express <-[#black]- mongodb
Expand All @@ -454,7 +372,7 @@ Mongo Express is used as GUI, MongoDB datasource for Grafana is part of Enterpri

<!--
@startuml plantuml/mongodb_prometheus
actor user
actor user
package host {
cloud {
component sensor
Expand Down Expand Up @@ -495,4 +413,4 @@ Formula acts as Prometheus exporter in this configuration, Prometheus component

Report can be viewed in Grafana dashboard with two panels:
* the top panel represents the energy consumption fo each constainer running on your host.
* the bottom panel represents the energy consumption for the whole CPU as reported directly by the CPU (in green) and as estimated by powerAPI (in yellow).
* the bottom panel represents the energy consumption for the whole CPU as reported directly by the CPU (in green) and as estimated by powerAPI (in yellow).
27 changes: 5 additions & 22 deletions deploy/docker/docker-compose.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
version: '3.8'

services:
############################################
# SOURCES & DESTINATIONS #
Expand Down Expand Up @@ -56,15 +54,6 @@ services:
profiles:
- csv

# FileDB (Default is a dummy image, can be used to add treatments on data files)
filedb:
container_name: filedb
image: ${FILEDB_IMAGE}
volumes:
- ${PWD}/filedb:/tmp/filedb
profiles:
- filedb

# Socket (Default is a dummy image, can be used to add treatments on formula socket)
socket:
container_name: socket
Expand All @@ -73,7 +62,7 @@ services:
- socket

############################################
# POWER API #
# POWERAPI #
############################################
# PowerAPI Sensor
sensor:
Expand All @@ -84,8 +73,7 @@ services:
- "--config-file"
- "/etc/sensor.json"
volumes:
- ${PWD}/sensor/${SENSOR_TYPE}-${POWERAPI_SOURCE}.json:/etc/sensor.json
- ${PWD}/csv:/tmp/csv
- ${PWD}/sensor/hwpc-${POWERAPI_SOURCE}.json:/etc/sensor.json
- type: bind
source: /proc
target: /proc
Expand All @@ -109,14 +97,13 @@ services:
- "--config-file"
- "/etc/formula.json"
volumes:
- ${PWD}/formula/${FORMULA_TYPE}-${POWERAPI_SOURCE}-${POWERAPI_DESTINATION}.json:/etc/formula.json
- ${PWD}/formula/smartwatts-${POWERAPI_SOURCE}-${POWERAPI_DESTINATION}.json:/etc/formula.json
- ${PWD}/csv:/tmp/csv
- ${PWD}/filedb:/tmp/filedb
depends_on:
- ${POWERAPI_DESTINATION}
networks:
- powerapi-network
restart: unless-stopped



############################################
# TOOLS #
Expand Down Expand Up @@ -165,7 +152,3 @@ services:

networks:
powerapi-network:
driver: bridge
ipam:
config:
- subnet: 192.168.34.0/24
Loading

0 comments on commit e6ca260

Please sign in to comment.