Merge pull request #124 from mag37/prometheus_addon

Prometheus addon

README.md

@@ -10,15 +10,17 @@
   <a href="https://github.com/sponsors/mag37"><img src="https://img.shields.io/badge/-Sponsor-grey?style=flat-square&logo=github" alt="Github Sponsor"></a>
 </p>
 
-<h3 align="center">CLI tool to automate docker image updates. <br>No <b>pre-pull</b>, selective, optional notifications and prune when done.</h3>
+<h2 align="center">CLI tool to automate docker image updates or notifying when updates are available.</h2>
-<h2 align="center">Now with simple notification integrations!</h2>
+<h3 align="center">Features:</h3>
-<h4 align="center">With features like excluding specific containers, custom container labels, auto-prune when done and more.</h4>
+<h3 align="center">selective updates, exclude containers, custom labels, notification plugins, prune when done and more.</h3>
-<h4 align="center">Also see the fresh Podman fork <a href="https://github.com/sudo-kraken/podcheck">sudo-kraken/podcheck</a>!</h4>
+
+<h4 align="center">For Podman - see the fork <a href="https://github.com/sudo-kraken/podcheck">sudo-kraken/podcheck</a>!</h4>
 
 ___
 ## :bell: Changelog
 
-- **v0.5.3**: Local image check changed (use imageId instead of name) and Gotify-template fixed (whale icon removed).
+- **v0.5.4.0**: Added support for a Prometheus+node_exporter metric collection through a file collector.
+- **v0.5.3.0**: Local image check changed (use imageId instead of name) and Gotify-template fixed (whale icon removed).
 - **v0.5.2.1**: Rewrite of dependency downloads, jq can be installed with package manager or static binary.
 - **v0.5.1**: DEPENDENCY WARNING: now requires **jq**. + Upstreaming changes from [sudo-kraken/podcheck](https://github.com/sudo-kraken/podcheck)
 - **v0.5.0**: Rewritten notify logic - all templates are adjusted and should be migrated!
@@ -41,6 +43,7 @@ Example:    dockcheck.sh -y -d 10 -e nextcloud,heimdall
 
 Options:"
 -a|y   Automatic updates, without interaction.
+-c D   Exports metrics as prom file for the prometheus node_exporter. Provide the collector textfile directory.
 -d N   Only update to new images that are N+ days old. Lists too recent with +prefix and age. 2xSlower.
 -e X   Exclude containers, separated by comma.
 -f     Force stack restart after update. Caution: restarts once for every updated container within stack.
@@ -135,6 +138,17 @@ nginx  ->  https://github.com/docker-library/official-images/blob/master/library
 ```
 The `urls.list` file is just an example and I'd gladly see that people contribute back when they add their preferred URLs to their lists.
 
+## :chart_with_upwards_trend: Prometheus and node_exporter
+Dockcheck can be used together with [Prometheus](https://github.com/prometheus/prometheus) and [node_exporter](https://github.com/prometheus/node_exporter) to export metrics via the file collector, scheduled with cron or likely.
+This is done with the `-c` option, like this:
+```
+dockcheck.sh -c /path/to/exporter/directory
+```
+
+See the [README_prom.md](./addons/prometheus/README.md) for more detailed information on how to set it up!
+
+<sub><sup>Contributed by [tdralle](https://github.com/tdralle).</sup></sub>  
+
 ## :bookmark: Labels
 Optionally add labels to compose-files. Currently these are the usable labels:
 ```
@@ -199,4 +213,3 @@ dockcheck is created and released under the [GNU GPL v3.0](https://www.gnu.org/l
 ___
 
 ### :floppy_disk: The [story](https://mag37.org/posts/project_dockcheck/) behind it. 1 year in retrospect.
-

addons/prometheus/README.md

@@ -0,0 +1,61 @@
+## [Prometheus](https://github.com/prometheus/prometheus) and [node_exporter](https://github.com/prometheus/node_exporter)
+Dockcheck is capable to export metrics to prometheus via the text file collector provided by the node_exporter.
+In order to do so the -c flag has to be specified followed by the file path that is configured in the text file collector of the node_exporter.
+A simple cron job can be configured to export these metrics on a regular interval as shown in the sample below:
+
+```
+0 1 * * * /root/dockcheck.sh -n -c /var/lib/node_exporter/textfile_collector
+```
+
+The following metrics are exported to prometheus
+
+```
+# HELP dockcheck_images_analyzed Docker images that have been analyzed
+# TYPE dockcheck_images_analyzed gauge
+dockcheck_images_analyzed 22
+# HELP dockcheck_images_outdated Docker images that are outdated
+# TYPE dockcheck_images_outdated gauge
+dockcheck_images_outdated 7
+# HELP dockcheck_images_latest Docker images that are outdated
+# TYPE dockcheck_images_latest gauge
+dockcheck_images_latest 14
+# HELP dockcheck_images_error Docker images with analysis errors
+# TYPE dockcheck_images_error gauge
+dockcheck_images_error 1
+# HELP dockcheck_images_analyze_timestamp_seconds Last dockercheck run time
+# TYPE dockcheck_images_analyze_timestamp_seconds gauge
+dockcheck_images_analyze_timestamp_seconds 1737924029
+```
+
+Once those metrics are exported they can be used to define alarms as shown below
+
+```
+- alert: dockcheck_images_outdated
+  expr: sum by(instance) (dockcheck_images_outdated) > 0
+  for: 15s
+  labels:
+    severity: warning
+  annotations:
+    summary: "{{ $labels.instance }} has {{ $value }} outdated docker images."
+    description: "{{ $labels.instance }} has {{ $value }} outdated docker images."
+- alert: dockcheck_images_error
+  expr: sum by(instance) (dockcheck_images_error) > 0
+  for: 15s
+  labels:
+    severity: warning
+  annotations:
+    summary: "{{ $labels.instance }} has {{ $value }} docker images having an error."
+    description: "{{ $labels.instance }} has {{ $value }} docker images having an error."
+- alert: dockercheck_image_last_analyze
+  expr: (time() - dockcheck_images_analyze_timestamp_seconds) > (3600 * 24 * 3)
+  for: 15s
+  labels:
+    severity: warning
+  annotations:
+    summary: "{{ $labels.instance }} has not updated the dockcheck statistics for more than  3 days."
+    description: "{{ $labels.instance }} has not updated the dockcheck statistics for more than 3 days."
+```
+
+There is a reference Grafana dashboard in [grafana/grafana_dashboard.json](./grafana/grafana_dashboard.json).
+
+![](./grafana/grafana_dashboard.png)

addons/prometheus/grafana/grafana_dashboard.json

@@ -0,0 +1,382 @@
+{
+  "__inputs": [
+    {
+      "name": "DS_PROMETHEUS",
+      "label": "prometheus",
+      "description": "",
+      "type": "datasource",
+      "pluginId": "prometheus",
+      "pluginName": "Prometheus"
+    }
+  ],
+  "__elements": {},
+  "__requires": [
+    {
+      "type": "grafana",
+      "id": "grafana",
+      "name": "Grafana",
+      "version": "11.4.0"
+    },
+    {
+      "type": "datasource",
+      "id": "prometheus",
+      "name": "Prometheus",
+      "version": "1.0.0"
+    },
+    {
+      "type": "panel",
+      "id": "table",
+      "name": "Table",
+      "version": ""
+    }
+  ],
+  "annotations": {
+    "list": [
+      {
+        "builtIn": 1,
+        "datasource": {
+          "type": "grafana",
+          "uid": "-- Grafana --"
+        },
+        "enable": true,
+        "hide": true,
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "name": "Annotations & Alerts",
+        "type": "dashboard"
+      }
+    ]
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 0,
+  "id": null,
+  "links": [],
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "thresholds"
+          },
+          "custom": {
+            "align": "auto",
+            "cellOptions": {
+              "type": "auto"
+            },
+            "inspect": false
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          }
+        },
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "last_analyze_timestamp"
+            },
+            "properties": [
+              {
+                "id": "unit",
+                "value": "dateTimeAsIso"
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "last_analyze_since"
+            },
+            "properties": [
+              {
+                "id": "unit",
+                "value": "s"
+              },
+              {
+                "id": "custom.cellOptions",
+                "value": {
+                  "mode": "gradient",
+                  "type": "color-background"
+                }
+              },
+              {
+                "id": "thresholds",
+                "value": {
+                  "mode": "absolute",
+                  "steps": [
+                    {
+                      "color": "green",
+                      "value": null
+                    },
+                    {
+                      "color": "red",
+                      "value": 259200
+                    }
+                  ]
+                }
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "images_outdated"
+            },
+            "properties": [
+              {
+                "id": "custom.cellOptions",
+                "value": {
+                  "mode": "gradient",
+                  "type": "color-background"
+                }
+              },
+              {
+                "id": "thresholds",
+                "value": {
+                  "mode": "absolute",
+                  "steps": [
+                    {
+                      "color": "green",
+                      "value": null
+                    },
+                    {
+                      "color": "red",
+                      "value": 1
+                    }
+                  ]
+                }
+              }
+            ]
+          },
+          {
+            "matcher": {
+              "id": "byName",
+              "options": "images_error"
+            },
+            "properties": [
+              {
+                "id": "custom.cellOptions",
+                "value": {
+                  "mode": "gradient",
+                  "type": "color-background"
+                }
+              },
+              {
+                "id": "thresholds",
+                "value": {
+                  "mode": "absolute",
+                  "steps": [
+                    {
+                      "color": "green",
+                      "value": null
+                    },
+                    {
+                      "color": "red",
+                      "value": 1
+                    }
+                  ]
+                }
+              }
+            ]
+          }
+        ]
+      },
+      "gridPos": {
+        "h": 14,
+        "w": 24,
+        "x": 0,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "cellHeight": "sm",
+        "footer": {
+          "countRows": false,
+          "fields": "",
+          "reducer": [
+            "sum"
+          ],
+          "show": false
+        },
+        "frameIndex": 1,
+        "showHeader": true,
+        "sortBy": []
+      },
+      "pluginVersion": "11.4.0",
+      "targets": [
+        {
+          "disableTextWrap": false,
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "sum by(instance) (dockcheck_images_analyzed)",
+          "format": "table",
+          "fullMetaSearch": false,
+          "hide": false,
+          "includeNullMetadata": true,
+          "instant": true,
+          "interval": "",
+          "legendFormat": "{{instance}}",
+          "range": false,
+          "refId": "dockcheck_images_analyzed",
+          "useBackend": false,
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          }
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "disableTextWrap": false,
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "sum by(instance) (dockcheck_images_outdated)",
+          "format": "table",
+          "fullMetaSearch": false,
+          "hide": false,
+          "includeNullMetadata": true,
+          "instant": true,
+          "legendFormat": "{{instance}}",
+          "range": false,
+          "refId": "dockcheck_images_outdated",
+          "useBackend": false
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "disableTextWrap": false,
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "sum by(instance) (dockcheck_images_latest)",
+          "format": "table",
+          "fullMetaSearch": false,
+          "hide": false,
+          "includeNullMetadata": true,
+          "instant": true,
+          "legendFormat": "{{instance}}",
+          "range": false,
+          "refId": "dockcheck_images_latest",
+          "useBackend": false
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "sum by(instance) (dockcheck_images_error)",
+          "format": "table",
+          "hide": false,
+          "instant": true,
+          "legendFormat": "{{instance}}",
+          "range": false,
+          "refId": "dockcheck_images_error"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "dockcheck_images_analyze_timestamp_seconds * 1000",
+          "format": "table",
+          "hide": false,
+          "instant": true,
+          "legendFormat": "{{instance}}",
+          "range": false,
+          "refId": "dockcheck_images_analyze_timestamp_seconds"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "editorMode": "code",
+          "exemplar": false,
+          "expr": "time() - dockcheck_images_analyze_timestamp_seconds",
+          "format": "table",
+          "hide": false,
+          "instant": true,
+          "legendFormat": "{{instance}}",
+          "range": false,
+          "refId": "dockcheck_images_last_analyze"
+        }
+      ],
+      "title": "Dockcheck Status",
+      "transformations": [
+        {
+          "id": "merge",
+          "options": {}
+        },
+        {
+          "id": "organize",
+          "options": {
+            "excludeByName": {
+              "Time": true,
+              "__name__": true,
+              "job": true
+            },
+            "includeByName": {},
+            "indexByName": {
+              "Time": 0,
+              "Value #dockcheck_images_analyze_timestamp_seconds": 2,
+              "Value #dockcheck_images_analyzed": 4,
+              "Value #dockcheck_images_error": 7,
+              "Value #dockcheck_images_last_analyze": 3,
+              "Value #dockcheck_images_latest": 5,
+              "Value #dockcheck_images_outdated": 6,
+              "instance": 1,
+              "job": 8
+            },
+            "renameByName": {
+              "Value #A": "analyze_timestamp",
+              "Value #dockcheck_images_analyze_timestamp_seconds": "last_analyze_timestamp",
+              "Value #dockcheck_images_analyzed": "images_analyzed",
+              "Value #dockcheck_images_error": "images_error",
+              "Value #dockcheck_images_last_analyze": "last_analyze_since",
+              "Value #dockcheck_images_latest": "images_latest",
+              "Value #dockcheck_images_outdated": "images_outdated"
+            }
+          }
+        }
+      ],
+      "type": "table"
+    }
+  ],
+  "schemaVersion": 40,
+  "tags": [],
+  "templating": {
+    "list": []
+  },
+  "time": {
+    "from": "now-6h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "browser",
+  "title": "Dockcheck Status",
+  "uid": "feb4pv3kv1hxca",
+  "version": 17,
+  "weekStart": ""
+}

addons/prometheus/prometheus_collector.sh

@@ -0,0 +1,28 @@
+prometheus_exporter() {
+  checkedImages=$(($1 + $2 + $3))
+  checkTimestamp=$(date +%s)
+  
+  promFileContent=()
+  promFileContent+=("# HELP dockcheck_images_analyzed Docker images that have been analyzed")
+  promFileContent+=("# TYPE dockcheck_images_analyzed gauge")
+  promFileContent+=("dockcheck_images_analyzed $checkedImages")
+  
+  promFileContent+=("# HELP dockcheck_images_outdated Docker images that are outdated")
+  promFileContent+=("# TYPE dockcheck_images_outdated gauge")
+  promFileContent+=("dockcheck_images_outdated ${#GotUpdates[@]}")
+
+  promFileContent+=("# HELP dockcheck_images_latest Docker images that are outdated")
+  promFileContent+=("# TYPE dockcheck_images_latest gauge")
+  promFileContent+=("dockcheck_images_latest ${#NoUpdates[@]}")
+  
+  promFileContent+=("# HELP dockcheck_images_error Docker images with analysis errors")
+  promFileContent+=("# TYPE dockcheck_images_error gauge")
+  promFileContent+=("dockcheck_images_error ${#GotErrors[@]}")
+  
+  promFileContent+=("# HELP dockcheck_images_analyze_timestamp_seconds Last dockercheck run time")
+  promFileContent+=("# TYPE dockcheck_images_analyze_timestamp_seconds gauge")
+  promFileContent+=("dockcheck_images_analyze_timestamp_seconds $checkTimestamp")
+  
+  printf "%s\n" "${promFileContent[@]}" > "$CollectorTextFileDirectory/dockcheck_info.prom\$\$"
+  mv -f "$CollectorTextFileDirectory/dockcheck_info.prom\$\$" "$CollectorTextFileDirectory/dockcheck_info.prom"
+}

dockcheck.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
-VERSION="v0.5.3.0"
+VERSION="v0.5.4.0"
-### ChangeNotes: Bugfixes - local image check changed, Gotify-template fixed
+### ChangeNotes: Added support for a Prometheus+node_exporter metric collection through a file collector.
 Github="https://github.com/mag37/dockcheck"
 RawUrl="https://raw.githubusercontent.com/mag37/dockcheck/main/dockcheck.sh"
 
@@ -20,6 +20,7 @@ Help() {
   echo
   echo "Options:"
   echo "-a|y   Automatic updates, without interaction."
+  echo "-c     Exports metrics as prom file for the prometheus node_exporter. Provide the collector textfile directory."
   echo "-d N   Only update to new images that are N+ days old. Lists too recent with +prefix and age. 2xSlower."
   echo "-e X   Exclude containers, separated by comma."
   echo "-f     Force stack restart after update. Caution: restarts once for every updated container within stack."
@@ -47,9 +48,11 @@ c_reset="\033[0m"
 
 Timeout=10
 Stopped=""
-while getopts "aynpfrhlisvme:d:t:" options; do
+while getopts "aynpfrhlisvmc:e:d:t:" options; do
   case "${options}" in
     a|y) AutoUp="yes" ;;
+    c)   CollectorTextFileDirectory="${OPTARG}"
+         if ! [[ -d  $CollectorTextFileDirectory ]] ; then { printf "The directory (%s) does not exist.\n" "${CollectorTextFileDirectory}"  ; exit 2; } fi ;;
     n)   AutoUp="no" ;;
     r)   DRunUp="yes" ;;
     p)   AutoPrune="yes" ;;
@@ -310,6 +313,11 @@ NoUpdates=($(sort <<<"${NoUpdates[*]}"))
 GotUpdates=($(sort <<<"${GotUpdates[*]}"))
 unset IFS
 
+# Run the prometheus exporter function
+if [ -n "$CollectorTextFileDirectory" ] ; then
+  source "$ScriptWorkDir"/addons/prometheus/prometheus_collector.sh && prometheus_exporter ${#NoUpdates[@]} ${#GotUpdates[@]} ${#GotError[@]}
+fi
+
 # Define how many updates are available
 UpdCount="${#GotUpdates[@]}"