diff --git a/README.md b/README.md index e03bd32..5251e02 100644 --- a/README.md +++ b/README.md @@ -10,21 +10,23 @@ Github Sponsor

-

CLI tool to automate docker image updates.
No pre-pull, selective, optional notifications and prune when done.

-

Now with simple notification integrations!

-

With features like excluding specific containers, custom container labels, auto-prune when done and more.

-

Also see the fresh Podman fork sudo-kraken/podcheck!

+

CLI tool to automate docker image updates or notifying when updates are available.

+

Features:

+

selective updates, exclude containers, custom labels, notification plugins, prune when done and more.

+ +

For Podman - see the fork sudo-kraken/podcheck!

___ ## :bell: Changelog -- **v0.5.3**: Local image check changed (use imageId instead of name) and Gotify-template fixed (whale icon removed). +- **v0.5.4.0**: Added support for a Prometheus+node_exporter metric collection through a file collector. +- **v0.5.3.0**: Local image check changed (use imageId instead of name) and Gotify-template fixed (whale icon removed). - **v0.5.2.1**: Rewrite of dependency downloads, jq can be installed with package manager or static binary. - **v0.5.1**: DEPENDENCY WARNING: now requires **jq**. + Upstreaming changes from [sudo-kraken/podcheck](https://github.com/sudo-kraken/podcheck) - **v0.5.0**: Rewritten notify logic - all templates are adjusted and should be migrated! - Copy the custom settings from your current template to the new version of the same template. - Look into, copy and customize the `urls.list` file if that's of interest. - - Other changes: + - Other changes: - Added Discord notify template. - Verbosity changed of `regctl`. - **v0.4.9**: Added a function to enrich the notify-message with release note URLs. See [Release notes addon](https://github.com/mag37/dockcheck#date-release-notes-addon-to-notifications) @@ -41,6 +43,7 @@ Example: dockcheck.sh -y -d 10 -e nextcloud,heimdall Options:" -a|y Automatic updates, without interaction. +-c D Exports metrics as prom file for the prometheus node_exporter. Provide the collector textfile directory. -d N Only update to new images that are N+ days old. Lists too recent with +prefix and age. 2xSlower. -e X Exclude containers, separated by comma. -f Force stack restart after update. Caution: restarts once for every updated container within stack. @@ -80,7 +83,7 @@ ___ ## :nut_and_bolt: Dependencies - Running docker (duh) and compose, either standalone or plugin. (see [Podman fork](https://github.com/sudo-kraken/podcheck) - Bash shell or compatible shell of at least v4.3 -- [jq](https://github.com/jqlang/jq) +- [jq](https://github.com/jqlang/jq) - User will be prompted to install with package manager or download static binary. - [regclient/regctl](https://github.com/regclient/regclient) (Licensed under [Apache-2.0 License](http://www.apache.org/licenses/LICENSE-2.0)) - User will be prompted to download `regctl` if not in `PATH` or `PWD`. @@ -123,8 +126,8 @@ Further additions are welcome - suggestions or PR! Initiated and first contributed by [yoyoma2](https://github.com/yoyoma2). ### :date: Release notes addon to Notifications -There's a function to use a lookup-file to add release note URL's to the notification message. -Copy the notify_templates/`urls.list` file to the script directory, it will be used automatically if it's there. Modify it as necessary, the names of interest in the left column needs to match your container names. +There's a function to use a lookup-file to add release note URL's to the notification message. +Copy the notify_templates/`urls.list` file to the script directory, it will be used automatically if it's there. Modify it as necessary, the names of interest in the left column needs to match your container names. The output of the notification will look something like this: ``` Containers on hostname with updates available: @@ -135,6 +138,17 @@ nginx -> https://github.com/docker-library/official-images/blob/master/library ``` The `urls.list` file is just an example and I'd gladly see that people contribute back when they add their preferred URLs to their lists. +## :chart_with_upwards_trend: Prometheus and node_exporter +Dockcheck can be used together with [Prometheus](https://github.com/prometheus/prometheus) and [node_exporter](https://github.com/prometheus/node_exporter) to export metrics via the file collector, scheduled with cron or likely. +This is done with the `-c` option, like this: +``` +dockcheck.sh -c /path/to/exporter/directory +``` + +See the [README_prom.md](./addons/prometheus/README.md) for more detailed information on how to set it up! + +Contributed by [tdralle](https://github.com/tdralle). + ## :bookmark: Labels Optionally add labels to compose-files. Currently these are the usable labels: ``` @@ -186,7 +200,7 @@ function dchk { Containers need to be manually stopped, removed and created again to run on the new image. ## :wrench: Debugging -If you hit issues, you could check the output of the `extras/errorCheck.sh` script for clues. +If you hit issues, you could check the output of the `extras/errorCheck.sh` script for clues. Another option is to run the main script with debugging in a subshell `bash -x dockcheck.sh` - if there's a particular container/image that's causing issues you can filter for just that through `bash -x dockcheck.sh nginx`. ## :scroll: License @@ -199,4 +213,3 @@ dockcheck is created and released under the [GNU GPL v3.0](https://www.gnu.org/l ___ ### :floppy_disk: The [story](https://mag37.org/posts/project_dockcheck/) behind it. 1 year in retrospect. - diff --git a/addons/prometheus/README.md b/addons/prometheus/README.md new file mode 100644 index 0000000..0bb84b8 --- /dev/null +++ b/addons/prometheus/README.md @@ -0,0 +1,61 @@ +## [Prometheus](https://github.com/prometheus/prometheus) and [node_exporter](https://github.com/prometheus/node_exporter) +Dockcheck is capable to export metrics to prometheus via the text file collector provided by the node_exporter. +In order to do so the -c flag has to be specified followed by the file path that is configured in the text file collector of the node_exporter. +A simple cron job can be configured to export these metrics on a regular interval as shown in the sample below: + +``` +0 1 * * * /root/dockcheck.sh -n -c /var/lib/node_exporter/textfile_collector +``` + +The following metrics are exported to prometheus + +``` +# HELP dockcheck_images_analyzed Docker images that have been analyzed +# TYPE dockcheck_images_analyzed gauge +dockcheck_images_analyzed 22 +# HELP dockcheck_images_outdated Docker images that are outdated +# TYPE dockcheck_images_outdated gauge +dockcheck_images_outdated 7 +# HELP dockcheck_images_latest Docker images that are outdated +# TYPE dockcheck_images_latest gauge +dockcheck_images_latest 14 +# HELP dockcheck_images_error Docker images with analysis errors +# TYPE dockcheck_images_error gauge +dockcheck_images_error 1 +# HELP dockcheck_images_analyze_timestamp_seconds Last dockercheck run time +# TYPE dockcheck_images_analyze_timestamp_seconds gauge +dockcheck_images_analyze_timestamp_seconds 1737924029 +``` + +Once those metrics are exported they can be used to define alarms as shown below + +``` +- alert: dockcheck_images_outdated + expr: sum by(instance) (dockcheck_images_outdated) > 0 + for: 15s + labels: + severity: warning + annotations: + summary: "{{ $labels.instance }} has {{ $value }} outdated docker images." + description: "{{ $labels.instance }} has {{ $value }} outdated docker images." +- alert: dockcheck_images_error + expr: sum by(instance) (dockcheck_images_error) > 0 + for: 15s + labels: + severity: warning + annotations: + summary: "{{ $labels.instance }} has {{ $value }} docker images having an error." + description: "{{ $labels.instance }} has {{ $value }} docker images having an error." +- alert: dockercheck_image_last_analyze + expr: (time() - dockcheck_images_analyze_timestamp_seconds) > (3600 * 24 * 3) + for: 15s + labels: + severity: warning + annotations: + summary: "{{ $labels.instance }} has not updated the dockcheck statistics for more than 3 days." + description: "{{ $labels.instance }} has not updated the dockcheck statistics for more than 3 days." +``` + +There is a reference Grafana dashboard in [grafana/grafana_dashboard.json](./grafana/grafana_dashboard.json). + +![](./grafana/grafana_dashboard.png) diff --git a/addons/prometheus/grafana/grafana_dashboard.json b/addons/prometheus/grafana/grafana_dashboard.json new file mode 100644 index 0000000..ad308bb --- /dev/null +++ b/addons/prometheus/grafana/grafana_dashboard.json @@ -0,0 +1,382 @@ +{ + "__inputs": [ + { + "name": "DS_PROMETHEUS", + "label": "prometheus", + "description": "", + "type": "datasource", + "pluginId": "prometheus", + "pluginName": "Prometheus" + } + ], + "__elements": {}, + "__requires": [ + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "11.4.0" + }, + { + "type": "datasource", + "id": "prometheus", + "name": "Prometheus", + "version": "1.0.0" + }, + { + "type": "panel", + "id": "table", + "name": "Table", + "version": "" + } + ], + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "last_analyze_timestamp" + }, + "properties": [ + { + "id": "unit", + "value": "dateTimeAsIso" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "last_analyze_since" + }, + "properties": [ + { + "id": "unit", + "value": "s" + }, + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "color-background" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 259200 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "images_outdated" + }, + "properties": [ + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "color-background" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "images_error" + }, + "properties": [ + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "color-background" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + } + ] + } + ] + }, + "gridPos": { + "h": 14, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 2, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "frameIndex": 1, + "showHeader": true, + "sortBy": [] + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(instance) (dockcheck_images_analyzed)", + "format": "table", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "interval": "", + "legendFormat": "{{instance}}", + "range": false, + "refId": "dockcheck_images_analyzed", + "useBackend": false, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + } + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(instance) (dockcheck_images_outdated)", + "format": "table", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "dockcheck_images_outdated", + "useBackend": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(instance) (dockcheck_images_latest)", + "format": "table", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "dockcheck_images_latest", + "useBackend": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(instance) (dockcheck_images_error)", + "format": "table", + "hide": false, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "dockcheck_images_error" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "dockcheck_images_analyze_timestamp_seconds * 1000", + "format": "table", + "hide": false, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "dockcheck_images_analyze_timestamp_seconds" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "time() - dockcheck_images_analyze_timestamp_seconds", + "format": "table", + "hide": false, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "dockcheck_images_last_analyze" + } + ], + "title": "Dockcheck Status", + "transformations": [ + { + "id": "merge", + "options": {} + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "__name__": true, + "job": true + }, + "includeByName": {}, + "indexByName": { + "Time": 0, + "Value #dockcheck_images_analyze_timestamp_seconds": 2, + "Value #dockcheck_images_analyzed": 4, + "Value #dockcheck_images_error": 7, + "Value #dockcheck_images_last_analyze": 3, + "Value #dockcheck_images_latest": 5, + "Value #dockcheck_images_outdated": 6, + "instance": 1, + "job": 8 + }, + "renameByName": { + "Value #A": "analyze_timestamp", + "Value #dockcheck_images_analyze_timestamp_seconds": "last_analyze_timestamp", + "Value #dockcheck_images_analyzed": "images_analyzed", + "Value #dockcheck_images_error": "images_error", + "Value #dockcheck_images_last_analyze": "last_analyze_since", + "Value #dockcheck_images_latest": "images_latest", + "Value #dockcheck_images_outdated": "images_outdated" + } + } + } + ], + "type": "table" + } + ], + "schemaVersion": 40, + "tags": [], + "templating": { + "list": [] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "Dockcheck Status", + "uid": "feb4pv3kv1hxca", + "version": 17, + "weekStart": "" +} \ No newline at end of file diff --git a/addons/prometheus/grafana/grafana_dashboard.png b/addons/prometheus/grafana/grafana_dashboard.png new file mode 100644 index 0000000..c3878df Binary files /dev/null and b/addons/prometheus/grafana/grafana_dashboard.png differ diff --git a/addons/prometheus/prometheus_collector.sh b/addons/prometheus/prometheus_collector.sh new file mode 100644 index 0000000..c98af2b --- /dev/null +++ b/addons/prometheus/prometheus_collector.sh @@ -0,0 +1,28 @@ +prometheus_exporter() { + checkedImages=$(($1 + $2 + $3)) + checkTimestamp=$(date +%s) + + promFileContent=() + promFileContent+=("# HELP dockcheck_images_analyzed Docker images that have been analyzed") + promFileContent+=("# TYPE dockcheck_images_analyzed gauge") + promFileContent+=("dockcheck_images_analyzed $checkedImages") + + promFileContent+=("# HELP dockcheck_images_outdated Docker images that are outdated") + promFileContent+=("# TYPE dockcheck_images_outdated gauge") + promFileContent+=("dockcheck_images_outdated ${#GotUpdates[@]}") + + promFileContent+=("# HELP dockcheck_images_latest Docker images that are outdated") + promFileContent+=("# TYPE dockcheck_images_latest gauge") + promFileContent+=("dockcheck_images_latest ${#NoUpdates[@]}") + + promFileContent+=("# HELP dockcheck_images_error Docker images with analysis errors") + promFileContent+=("# TYPE dockcheck_images_error gauge") + promFileContent+=("dockcheck_images_error ${#GotErrors[@]}") + + promFileContent+=("# HELP dockcheck_images_analyze_timestamp_seconds Last dockercheck run time") + promFileContent+=("# TYPE dockcheck_images_analyze_timestamp_seconds gauge") + promFileContent+=("dockcheck_images_analyze_timestamp_seconds $checkTimestamp") + + printf "%s\n" "${promFileContent[@]}" > "$CollectorTextFileDirectory/dockcheck_info.prom\$\$" + mv -f "$CollectorTextFileDirectory/dockcheck_info.prom\$\$" "$CollectorTextFileDirectory/dockcheck_info.prom" +} diff --git a/dockcheck.sh b/dockcheck.sh index 238db09..f42febf 100755 --- a/dockcheck.sh +++ b/dockcheck.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -VERSION="v0.5.3.0" -### ChangeNotes: Bugfixes - local image check changed, Gotify-template fixed +VERSION="v0.5.4.0" +### ChangeNotes: Added support for a Prometheus+node_exporter metric collection through a file collector. Github="https://github.com/mag37/dockcheck" RawUrl="https://raw.githubusercontent.com/mag37/dockcheck/main/dockcheck.sh" @@ -20,6 +20,7 @@ Help() { echo echo "Options:" echo "-a|y Automatic updates, without interaction." + echo "-c Exports metrics as prom file for the prometheus node_exporter. Provide the collector textfile directory." echo "-d N Only update to new images that are N+ days old. Lists too recent with +prefix and age. 2xSlower." echo "-e X Exclude containers, separated by comma." echo "-f Force stack restart after update. Caution: restarts once for every updated container within stack." @@ -47,9 +48,11 @@ c_reset="\033[0m" Timeout=10 Stopped="" -while getopts "aynpfrhlisvme:d:t:" options; do +while getopts "aynpfrhlisvmc:e:d:t:" options; do case "${options}" in a|y) AutoUp="yes" ;; + c) CollectorTextFileDirectory="${OPTARG}" + if ! [[ -d $CollectorTextFileDirectory ]] ; then { printf "The directory (%s) does not exist.\n" "${CollectorTextFileDirectory}" ; exit 2; } fi ;; n) AutoUp="no" ;; r) DRunUp="yes" ;; p) AutoPrune="yes" ;; @@ -310,6 +313,11 @@ NoUpdates=($(sort <<<"${NoUpdates[*]}")) GotUpdates=($(sort <<<"${GotUpdates[*]}")) unset IFS +# Run the prometheus exporter function +if [ -n "$CollectorTextFileDirectory" ] ; then + source "$ScriptWorkDir"/addons/prometheus/prometheus_collector.sh && prometheus_exporter ${#NoUpdates[@]} ${#GotUpdates[@]} ${#GotError[@]} +fi + # Define how many updates are available UpdCount="${#GotUpdates[@]}"