diff --git a/README.md b/README.md index 4170f35..16ca230 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,18 @@ ___ ## :bell: Changelog +- **v0.6.0**: + - **Grafana & Prometheus Integration:** + - Added a detailed Prometheus metrics exporter that now reports not only the number of containers with updates, no-updates, and errors, but also the total number of containers checked, the duration of the update check, and the epoch timestamp of the last check. + - Enhanced documentation with instructions on integrating these metrics with Grafana for visual monitoring. + - **Improved Error Handling & Code Refactoring:** + - Introduced `set -euo pipefail` and local variable scoping within functions to improve reliability and prevent unexpected behaviour. + - Standardised container name handling and refined the Quadlet detection logic. + - **Self-Update Enhancements:** + - Updated the self-update mechanism to support both Git-based and HTTP-based updates, with an automatic restart that preserves the original arguments. + - **Miscellaneous Improvements:** + - Enhanced dependency installer to support both package manager and static binary installations for `jq` and `regctl`. + - General code refactoring across the project for better readability and maintainability. - **v0.5.7**: Rewrite of dependency downloads, now jq can be installed with package manager or static binary. - **v0.5.6**: Directly checking for systemd units matching container names. - Improved Quadlet detection by checking for systemd units named after the container. @@ -133,6 +145,15 @@ it-tools -> https://github.com/CorentinTh/it-tools/releases ``` The `urls.list` file is just an example and I'd gladly see that people contribute back when they add their preferred URLs to their lists. +## :chart_with_upwards_trend: Prometheus and node_exporter +Dockcheck can be used together with [Prometheus](https://github.com/prometheus/prometheus) and [node_exporter](https://github.com/prometheus/node_exporter) to export metrics via the file collector, scheduled with cron or likely. +This is done with the `-c` option, like this: +``` +dockcheck.sh -c /path/to/exporter/directory +``` +See the [README_prom.md](./addons/prometheus/README.md) for more detailed information on how to set it up! +Contributed by [tdralle](https://github.com/tdralle). + ## :bookmark: Labels Optionally, you can add labels to your containers to control how Podcheck handles them. Currently, these are the usable labels: diff --git a/addons/prometheus/README.md b/addons/prometheus/README.md new file mode 100644 index 0000000..3bc4f09 --- /dev/null +++ b/addons/prometheus/README.md @@ -0,0 +1,61 @@ +## [Prometheus](https://github.com/prometheus/prometheus) and [node_exporter](https://github.com/prometheus/node_exporter) +Podcheck check is capable to export metrics to prometheus via the text file collector provided by the node_exporter. +In order to do so the -c flag has to be specified followed by the file path that is configured in the text file collector of the node_exporter. +A simple cron job can be configured to export these metrics on a regular interval as shown in the sample below: + +``` +0 1 * * * /root/podcheck.sh -n -c /var/lib/node_exporter/textfile_collector +``` + +The following metrics are exported to prometheus + +``` +# HELP podcheck_images_analyzed Podman images that have been analyzed +# TYPE podcheck_images_analyzed gauge +podcheck_images_analyzed 22 +# HELP podcheck_images_outdated Podman images that are outdated +# TYPE podcheck_images_outdated gauge +podcheck_images_outdated 7 +# HELP podcheck_images_latest Podman images that are outdated +# TYPE podcheck_images_latest gauge +podcheck_images_latest 14 +# HELP podcheck_images_error Podman images with analysis errors +# TYPE podcheck_images_error gauge +podcheck_images_error 1 +# HELP podcheck_images_analyze_timestamp_seconds Last podcheck run time +# TYPE podcheck_images_analyze_timestamp_seconds gauge +podcheck_images_analyze_timestamp_seconds 1737924029 +``` + +Once those metrics are exported they can be used to define alarms as shown below + +``` +- alert: podcheck_images_outdated + expr: sum by(instance) (podcheck_images_outdated) > 0 + for: 15s + labels: + severity: warning + annotations: + summary: "{{ $labels.instance }} has {{ $value }} outdated podman images." + description: "{{ $labels.instance }} has {{ $value }} outdated podman images." +- alert: podcheck_images_error + expr: sum by(instance) (podcheck_images_error) > 0 + for: 15s + labels: + severity: warning + annotations: + summary: "{{ $labels.instance }} has {{ $value }} podman images having an error." + description: "{{ $labels.instance }} has {{ $value }} podman images having an error." +- alert: podcheck_image_last_analyze + expr: (time() - podcheck_images_analyze_timestamp_seconds) > (3600 * 24 * 3) + for: 15s + labels: + severity: warning + annotations: + summary: "{{ $labels.instance }} has not updated the podcheck statistics for more than 3 days." + description: "{{ $labels.instance }} has not updated the podcheck statistics for more than 3 days." +``` + +There is a reference Grafana dashboard in [grafana/grafana_dashboard.json](./grafana/grafana_dashboard.json). + +![](./grafana/grafana_dashboard.png) diff --git a/addons/prometheus/grafana/grafana_dashboard.json b/addons/prometheus/grafana/grafana_dashboard.json new file mode 100644 index 0000000..e731337 --- /dev/null +++ b/addons/prometheus/grafana/grafana_dashboard.json @@ -0,0 +1,382 @@ +{ + "__inputs": [ + { + "name": "DS_PROMETHEUS", + "label": "prometheus", + "description": "", + "type": "datasource", + "pluginId": "prometheus", + "pluginName": "Prometheus" + } + ], + "__elements": {}, + "__requires": [ + { + "type": "grafana", + "id": "grafana", + "name": "Grafana", + "version": "11.4.0" + }, + { + "type": "datasource", + "id": "prometheus", + "name": "Prometheus", + "version": "1.0.0" + }, + { + "type": "panel", + "id": "table", + "name": "Table", + "version": "" + } + ], + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 0, + "id": null, + "links": [], + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "custom": { + "align": "auto", + "cellOptions": { + "type": "auto" + }, + "inspect": false + }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + } + }, + "overrides": [ + { + "matcher": { + "id": "byName", + "options": "last_analyze_timestamp" + }, + "properties": [ + { + "id": "unit", + "value": "dateTimeAsIso" + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "last_analyze_since" + }, + "properties": [ + { + "id": "unit", + "value": "s" + }, + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "color-background" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 259200 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "images_outdated" + }, + "properties": [ + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "color-background" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + } + ] + }, + { + "matcher": { + "id": "byName", + "options": "images_error" + }, + "properties": [ + { + "id": "custom.cellOptions", + "value": { + "mode": "gradient", + "type": "color-background" + } + }, + { + "id": "thresholds", + "value": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + } + } + ] + } + ] + }, + "gridPos": { + "h": 14, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 2, + "options": { + "cellHeight": "sm", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "frameIndex": 1, + "showHeader": true, + "sortBy": [] + }, + "pluginVersion": "11.4.0", + "targets": [ + { + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(instance) (podcheck_images_analyzed)", + "format": "table", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "interval": "", + "legendFormat": "{{instance}}", + "range": false, + "refId": "podcheck_images_analyzed", + "useBackend": false, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + } + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(instance) (podcheck_images_outdated)", + "format": "table", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "podcheck_images_outdated", + "useBackend": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "disableTextWrap": false, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(instance) (podcheck_images_latest)", + "format": "table", + "fullMetaSearch": false, + "hide": false, + "includeNullMetadata": true, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "podcheck_images_latest", + "useBackend": false + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "sum by(instance) (podcheck_images_error)", + "format": "table", + "hide": false, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "podcheck_images_error" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "podcheck_images_analyze_timestamp_seconds * 1000", + "format": "table", + "hide": false, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "podcheck_images_analyze_timestamp_seconds" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "editorMode": "code", + "exemplar": false, + "expr": "time() - podcheck_images_analyze_timestamp_seconds", + "format": "table", + "hide": false, + "instant": true, + "legendFormat": "{{instance}}", + "range": false, + "refId": "podcheck_images_last_analyze" + } + ], + "title": "podcheck Status", + "transformations": [ + { + "id": "merge", + "options": {} + }, + { + "id": "organize", + "options": { + "excludeByName": { + "Time": true, + "__name__": true, + "job": true + }, + "includeByName": {}, + "indexByName": { + "Time": 0, + "Value #podcheck_images_analyze_timestamp_seconds": 2, + "Value #podcheck_images_analyzed": 4, + "Value #podcheck_images_error": 7, + "Value #podcheck_images_last_analyze": 3, + "Value #podcheck_images_latest": 5, + "Value #podcheck_images_outdated": 6, + "instance": 1, + "job": 8 + }, + "renameByName": { + "Value #A": "analyze_timestamp", + "Value #podcheck_images_analyze_timestamp_seconds": "last_analyze_timestamp", + "Value #podcheck_images_analyzed": "images_analyzed", + "Value #podcheck_images_error": "images_error", + "Value #podcheck_images_last_analyze": "last_analyze_since", + "Value #podcheck_images_latest": "images_latest", + "Value #podcheck_images_outdated": "images_outdated" + } + } + } + ], + "type": "table" + } + ], + "schemaVersion": 40, + "tags": [], + "templating": { + "list": [] + }, + "time": { + "from": "now-6h", + "to": "now" + }, + "timepicker": {}, + "timezone": "browser", + "title": "podcheck Status", + "uid": "feb4pv3kv1hxca", + "version": 17, + "weekStart": "" +} \ No newline at end of file diff --git a/addons/prometheus/grafana/grafana_dashboard.png b/addons/prometheus/grafana/grafana_dashboard.png new file mode 100644 index 0000000..c3878df Binary files /dev/null and b/addons/prometheus/grafana/grafana_dashboard.png differ diff --git a/addons/prometheus/prometheus_collector.sh b/addons/prometheus/prometheus_collector.sh new file mode 100644 index 0000000..58f755f --- /dev/null +++ b/addons/prometheus/prometheus_collector.sh @@ -0,0 +1,62 @@ +#!/usr/bin/env bash +# prometheus_collector.sh - Exports detailed update metrics for Prometheus node_exporter. +# +# This script generates metrics about the state of Podman container update checks. +# It is designed to be sourced by podcheck.sh and then invoked with: +# +# prometheus_exporter +# +# Metrics: +# podcheck_no_updates: +# Number of containers that are already on the latest image. +# podcheck_updates: +# Number of containers with updates available. +# podcheck_errors: +# Number of containers that encountered errors during the update check. +# podcheck_total: +# Total number of containers checked. +# podcheck_check_duration: +# Duration (in seconds) it took to perform the update check. +# podcheck_last_check_timestamp: +# Epoch timestamp when the update check was performed. +# +# The metrics are written to a file named podcheck.prom in the specified +# CollectorTextFileDirectory, or /tmp if not specified. +# + +prometheus_exporter() { + local no_updates="$1" + local updates="$2" + local errors="$3" + local total="$4" + local check_duration="$5" + local collector_dir="${CollectorTextFileDirectory:-/tmp}" + local last_check_timestamp + last_check_timestamp=$(date +%s) + + { + echo "# HELP podcheck_no_updates Number of containers already on latest image." + echo "# TYPE podcheck_no_updates gauge" + echo "podcheck_no_updates $no_updates" + + echo "# HELP podcheck_updates Number of containers with updates available." + echo "# TYPE podcheck_updates gauge" + echo "podcheck_updates $updates" + + echo "# HELP podcheck_errors Number of containers with errors during update check." + echo "# TYPE podcheck_errors gauge" + echo "podcheck_errors $errors" + + echo "# HELP podcheck_total Total number of containers checked." + echo "# TYPE podcheck_total gauge" + echo "podcheck_total $total" + + echo "# HELP podcheck_check_duration Duration in seconds for the update check." + echo "# TYPE podcheck_check_duration gauge" + echo "podcheck_check_duration $check_duration" + + echo "# HELP podcheck_last_check_timestamp Epoch timestamp of the last update check." + echo "# TYPE podcheck_last_check_timestamp gauge" + echo "podcheck_last_check_timestamp $last_check_timestamp" + } > "$collector_dir/podcheck.prom" +} diff --git a/extras/apprise_quickstart.md b/extras/apprise_quickstart.md index 2d1ddd6..85689f4 100644 --- a/extras/apprise_quickstart.md +++ b/extras/apprise_quickstart.md @@ -1,9 +1,9 @@ # A small guide on getting started with Apprise notifications. -## Standalone docker container: [linuxserver/apprise-api](https://hub.docker.com/r/linuxserver/apprise-api) +## Standalone podman container: [linuxserver/apprise-api](https://hub.docker.com/r/linuxserver/apprise-api) -Set up the docker compose as preferred: +Set up the podman compose as preferred: ```yaml --- version: "2.1" diff --git a/extras/dc_brief.sh b/extras/dc_brief.sh deleted file mode 100755 index 61d56d8..0000000 --- a/extras/dc_brief.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/usr/bin/env bash - -### If not in PATH, set full path. Else just "regctl" -regbin="regctl" -### options to allow exclude: -while getopts "e:" options; do - case "${options}" in - e) Exclude=${OPTARG} ;; - *) exit 0 ;; - esac -done -shift "$((OPTIND-1))" -### Create array of excludes -IFS=',' read -r -a Excludes <<< "$Exclude" ; unset IFS - -SearchName="$1" - -for i in $(podman ps --filter "name=$SearchName" --format '{{.Names}}') ; do - for e in "${Excludes[@]}" ; do [[ "$i" == "$e" ]] && continue 2 ; done - printf ". " - RepoUrl=$(podman inspect "$i" --format='{{.ImageName}}') - LocalHash=$(podman image inspect "$RepoUrl" --format '{{.Digest}}') - ### Checking for errors while setting the variable: - if RegHash=$($regbin image digest --list "$RepoUrl" 2>/dev/null) ; then - if [[ "$LocalHash" == "$RegHash" ]] ; then NoUpdates+=("$i"); else GotUpdates+=("$i"); fi - else - GotErrors+=("$i") - fi -done - -### Sort arrays alphabetically -IFS=$'\n' -NoUpdates=($(sort <<<"${NoUpdates[*]}")) -GotUpdates=($(sort <<<"${GotUpdates[*]}")) -GotErrors=($(sort <<<"${GotErrors[*]}")) -unset IFS - -### List what containers got updates or not -if [[ -n ${NoUpdates[*]} ]] ; then - printf "\n\033[0;32mContainers on latest version:\033[0m\n" - printf "%s\n" "${NoUpdates[@]}" -fi -if [[ -n ${GotErrors[*]} ]] ; then - printf "\n\033[0;31mContainers with errors; won't get updated:\033[0m\n" - printf "%s\n" "${GotErrors[@]}" -fi -if [[ -n ${GotUpdates[*]} ]] ; then - printf "\n\033[0;33mContainers with updates available:\033[0m\n" - printf "%s\n" "${GotUpdates[@]}" -fi -printf "\n\n" diff --git a/extras/errorCheck.sh b/extras/errorCheck.sh index d4e10bf..4e271c8 100755 --- a/extras/errorCheck.sh +++ b/extras/errorCheck.sh @@ -1,37 +1,55 @@ #!/usr/bin/env bash +# Usage: ./script.sh + SearchName="$1" -for i in $(podman ps --filter "name=$SearchName" --format '{{.Names}}') ; do - echo "------------ $i ------------" - ContLabels=$(podman inspect "$i" --format '{{json .Config.Labels}}') - ContImage=$(podman inspect "$i" --format='{{.ImageName}}') + +# Iterate over containers whose name contains the search term. +podman ps --filter "name=$SearchName" --format '{{.Names}}' | while read -r container; do + echo "------------ $container ------------" + + # Retrieve container labels and image name. + ContLabels=$(podman inspect "$container" --format '{{json .Config.Labels}}') + ContImage=$(podman inspect "$container" --format '{{.ImageName}}') + + # Extract values from labels; if not set, default to an empty string. ContPath=$(jq -r '."com.docker.compose.project.working_dir"' <<< "$ContLabels") [ "$ContPath" == "null" ] && ContPath="" + ContConfigFile=$(jq -r '."com.docker.compose.project.config_files"' <<< "$ContLabels") [ "$ContConfigFile" == "null" ] && ContConfigFile="" + ContName=$(jq -r '."com.docker.compose.service"' <<< "$ContLabels") [ "$ContName" == "null" ] && ContName="" + ContEnv=$(jq -r '."com.docker.compose.project.environment_file"' <<< "$ContLabels") [ "$ContEnv" == "null" ] && ContEnv="" + ContUpdateLabel=$(jq -r '."sudo-kraken.podcheck.update"' <<< "$ContLabels") [ "$ContUpdateLabel" == "null" ] && ContUpdateLabel="" + ContRestartStack=$(jq -r '."sudo-kraken.podcheck.restart-stack"' <<< "$ContLabels") [ "$ContRestartStack" == "null" ] && ContRestartStack="" - - if [[ $ContConfigFile = '/'* ]] ; then + + # Determine the compose file location. + if [[ $ContConfigFile = '/'* ]]; then ComposeFile="$ContConfigFile" else ComposeFile="$ContPath/$ContConfigFile" fi - + + # Output the extracted details. echo -e "Service name:\t\t$ContName" echo -e "Project working dir:\t$ContPath" echo -e "Compose files:\t\t$ComposeFile" echo -e "Environment files:\t$ContEnv" echo -e "Container image:\t$ContImage" - echo -e "Update label:\t$ContUpdateLabel" + echo -e "Update label:\t\t$ContUpdateLabel" echo -e "Restart Stack label:\t$ContRestartStack" echo echo "Mounts:" - podman inspect -f '{{ range .Mounts }}{{ .Source }}:{{ .Destination }}{{ printf "\n" }}{{ end }}' "$i" + + # Display container mount points. + podman inspect -f '{{ range .Mounts }}{{ .Source }}:{{ .Destination }}{{ "\n" }}{{ end }}' "$container" echo done + diff --git a/extras/pc_brief.sh b/extras/pc_brief.sh new file mode 100644 index 0000000..77bc577 --- /dev/null +++ b/extras/pc_brief.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# pc_brief.sh - Provides a brief diagnostic summary of Podman Compose containers. +# Usage: pc_brief.sh + +set -euo pipefail + +# Check if a name filter argument was provided +if [ "$#" -eq 0 ]; then + echo "Usage: $0 " + exit 1 +fi + +SearchName="$1" + +# Use a while-read loop to correctly handle container names with spaces +podman ps --filter "name=$SearchName" --format '{{.Names}}' | while IFS= read -r container; do + echo "------------ $container ------------" + + # Retrieve container labels and image name + ContLabels=$(podman inspect "$container" --format '{{json .Config.Labels}}') + ContImage=$(podman inspect "$container" --format '{{.ImageName}}') + + # Extract Docker Compose-related labels via jq; default to empty if null + ContPath=$(jq -r '."com.docker.compose.project.working_dir"' <<< "$ContLabels") + [ "$ContPath" == "null" ] && ContPath="" + + ContConfigFile=$(jq -r '."com.docker.compose.project.config_files"' <<< "$ContLabels") + [ "$ContConfigFile" == "null" ] && ContConfigFile="" + + ContName=$(jq -r '."com.docker.compose.service"' <<< "$ContLabels") + [ "$ContName" == "null" ] && ContName="" + + ContEnv=$(jq -r '."com.docker.compose.project.environment_file"' <<< "$ContLabels") + [ "$ContEnv" == "null" ] && ContEnv="" + + ContUpdateLabel=$(jq -r '."sudo-kraken.podcheck.update"' <<< "$ContLabels") + [ "$ContUpdateLabel" == "null" ] && ContUpdateLabel="" + + ContRestartStack=$(jq -r '."sudo-kraken.podcheck.restart-stack"' <<< "$ContLabels") + [ "$ContRestartStack" == "null" ] && ContRestartStack="" + + # Determine the full path to the compose file(s) + if [[ $ContConfigFile = /* ]]; then + ComposeFile="$ContConfigFile" + else + ComposeFile="$ContPath/$ContConfigFile" + fi + + # Output a concise summary of container configuration + echo -e "Service name:\t\t$ContName" + echo -e "Project working dir:\t$ContPath" + echo -e "Compose files:\t\t$ComposeFile" + echo -e "Environment files:\t$ContEnv" + echo -e "Container image:\t$ContImage" + echo -e "Update label:\t\t$ContUpdateLabel" + echo -e "Restart Stack label:\t$ContRestartStack" + echo + echo "Mounts:" + podman inspect -f '{{ range .Mounts }}{{ .Source }}:{{ .Destination }}{{ "\n" }}{{ end }}' "$container" + echo +done diff --git a/notify_templates/notify_DSM.sh b/notify_templates/notify_DSM.sh index d9a0cd9..e96d5f2 100644 --- a/notify_templates/notify_DSM.sh +++ b/notify_templates/notify_DSM.sh @@ -49,4 +49,7 @@ Content-Transfer-Encoding: 7bit $MessageBody From $SenderName __EOF +# This ensures DSM's container manager will also see the update +/var/packages/ContainerManager/target/tool/image_upgradable_checker + } diff --git a/notify_templates/notify_gotify.sh b/notify_templates/notify_gotify.sh index b88f597..49d07ae 100644 --- a/notify_templates/notify_gotify.sh +++ b/notify_templates/notify_gotify.sh @@ -14,7 +14,7 @@ send_notification() { # Setting the MessageTitle and MessageBody variable here. MessageTitle="${FromHost} - updates available." - printf -v MessageBody "🐋 Containers on $FromHost with updates available:\n$UpdToString" + printf -v MessageBody "Containers on $FromHost with updates available:\n$UpdToString" # Modify to fit your setup: GotifyToken="Your Gotify token here" @@ -24,6 +24,6 @@ send_notification() { -F "title=${MessageTitle}" \ -F "message=${MessageBody}" \ -F "priority=5" \ - -X POST "${GotifyUrl}" &> /dev/null + -X POST "${GotifyUrl}" 1> /dev/null } diff --git a/podcheck.sh b/podcheck.sh index 14dff5e..3120315 100755 --- a/podcheck.sh +++ b/podcheck.sh @@ -1,8 +1,7 @@ #!/usr/bin/env bash -VERSION="v0.5.7" -# ChangeNotes: Rewrite of dependency installer. jq can now be installed via package manager or static binary. +VERSION="v0.6.0" Github="https://github.com/sudo-kraken/podcheck" -RawUrl="https://raw.githubusercontent.com/sudo-kraken/podcheck/main/podcheck.sh" +RawUrl="https://raw.githubusercontent.com/sudo-kraken/podcheck/upstream_patches/podcheck.sh" # Variables for self-updating ScriptArgs=( "$@" ) @@ -10,23 +9,23 @@ ScriptPath="$(readlink -f "$0")" ScriptWorkDir="$(dirname "$ScriptPath")" # Check if there's a new release of the script -LatestRelease="$(curl -s -r 0-100 $RawUrl | sed -n "/VERSION/s/VERSION=//p" | tr -d '"')" -LatestChanges="$(curl -s -r 0-200 $RawUrl | sed -n "/ChangeNotes/s/# ChangeNotes: //p")" +LatestRelease="$(curl -s -r 0-100 "$RawUrl" | sed -n "/VERSION/s/VERSION=//p" | tr -d '"')" +LatestChanges="$(curl -s -r 0-200 "$RawUrl" | sed -n "/ChangeNotes/s/# ChangeNotes: //p")" -# Help Function Help() { echo "Syntax: podcheck.sh [OPTION] [part of name to filter]" echo "Example: podcheck.sh -y -d 10 -e nextcloud,heimdall" echo echo "Options:" echo "-a|y Automatic updates, without interaction." + echo "-c Exports metrics as prom file for the prometheus node_exporter. Provide the collector textfile directory." echo "-d N Only update to new images that are N+ days old. Lists too recent with +prefix and age." echo "-e X Exclude containers, separated by comma." echo "-f Force pod restart after update." echo "-h Print this Help." echo "-i Inform - send a preconfigured notification." echo "-l Only update if label is set. See readme." - echo "-m Monochrome mode, no printf color codes." + echo "-m Monochrome mode, no printf colour codes." echo "-n No updates; only checking availability." echo "-p Auto-prune dangling images after update." echo "-r Allow updating images for podman run; won't update the container." @@ -37,7 +36,7 @@ Help() { echo "Project source: $Github" } -# Colors +# Colours c_red="\033[0;31m" c_green="\033[0;32m" c_yellow="\033[0;33m" @@ -45,76 +44,113 @@ c_blue="\033[0;34m" c_teal="\033[0;36m" c_reset="\033[0m" -Timeout=10 +# Initialise variables first +AutoUp="no" +AutoPrune="" Stopped="" -while getopts "aynpfrhlisvme:d:t:" options; do +Timeout=10 +NoUpdateMode=false +Excludes=() +GotUpdates=() +NoUpdates=() +GotErrors=() +NotifyUpdates=() +SelectedUpdates=() +OnlyLabel=false +ForceRestartPods=false + +# regbin will be set later. +regbin="" + +set -euo pipefail + +while getopts "aynpfrhlisvmc:e:d:t:v" options; do case "${options}" in a|y) AutoUp="yes" ;; - n) AutoUp="no" ;; + c) + CollectorTextFileDirectory="${OPTARG}" + if ! [[ -d $CollectorTextFileDirectory ]]; then + printf "The directory (%s) does not exist.\n" "${CollectorTextFileDirectory}" + exit 2 + fi + ;; + n) NoUpdateMode=true ;; r) DRunUp="yes" ;; p) AutoPrune="yes" ;; l) OnlyLabel=true ;; f) ForceRestartPods=true ;; - i) [ -s "$ScriptWorkDir"/notify.sh ] && { source "$ScriptWorkDir"/notify.sh ; Notify="yes" ; } ;; - e) Exclude=${OPTARG} ;; + i) [ -s "$ScriptWorkDir/notify.sh" ] && { source "$ScriptWorkDir/notify.sh"; Notify="yes"; } ;; + e) Exclude="${OPTARG}" + IFS=',' read -ra Excludes <<< "$Exclude" + ;; m) declare c_{red,green,yellow,blue,teal,reset}="" ;; s) Stopped="-a" ;; t) Timeout="${OPTARG}" ;; - v) printf "%s\n" "$VERSION" ; exit 0 ;; - d) DaysOld=${OPTARG} - if ! [[ $DaysOld =~ ^[0-9]+$ ]] ; then { printf "Days -d argument given (%s) is not a number.\n" "${DaysOld}" ; exit 2 ; } ; fi ;; - h|*) Help ; exit 2 ;; + d) DaysOld="${OPTARG}" + if ! [[ $DaysOld =~ ^[0-9]+$ ]]; then + printf "Days -d argument given (%s) is not a number.\n" "${DaysOld}" + exit 2 + fi + ;; + v) printf "%s\n" "$VERSION"; exit 0 ;; + h|*) Help; exit 2 ;; esac done shift "$((OPTIND-1))" +# Now get the search name from the first remaining positional parameter +SearchName="${1:-}" + # Self-update functions self_update_curl() { cp "$ScriptPath" "$ScriptPath".bak - if [[ $(command -v curl) ]]; then - curl -L $RawUrl > "$ScriptPath" ; chmod +x "$ScriptPath" + if command -v curl &>/dev/null; then + curl -L "$RawUrl" > "$ScriptPath" + chmod +x "$ScriptPath" printf "\n%s\n" "--- starting over with the updated version ---" - exec "$ScriptPath" "${ScriptArgs[@]}" # Run the new script with old arguments - exit 1 # Exit the old instance - elif [[ $(command -v wget) ]]; then - wget $RawUrl -O "$ScriptPath" ; chmod +x "$ScriptPath" + exec "$ScriptPath" "${ScriptArgs[@]}" + exit 1 + elif command -v wget &>/dev/null; then + wget "$RawUrl" -O "$ScriptPath" + chmod +x "$ScriptPath" printf "\n%s\n" "--- starting over with the updated version ---" - exec "$ScriptPath" "${ScriptArgs[@]}" # Run the new script with old arguments - exit 1 # Exit the old instance + exec "$ScriptPath" "${ScriptArgs[@]}" + exit 1 else printf "curl/wget not available - download the update manually: %s \n" "$Github" fi } self_update() { - cd "$ScriptWorkDir" || { printf "Path error, skipping update.\n" ; return ; } - if [[ $(command -v git) ]] && [[ "$(git ls-remote --get-url 2>/dev/null)" =~ .*"sudo-kraken/podcheck".* ]] ; then + cd "$ScriptWorkDir" || { printf "Path error, skipping update.\n"; return; } + if command -v git &>/dev/null && [[ "$(git ls-remote --get-url 2>/dev/null)" =~ .*"sudo-kraken/podcheck".* ]]; then printf "\n%s\n" "Pulling the latest version." - git pull --force || { printf "Git error, manually pull/clone.\n" ; return ; } + git pull --force || { printf "Git error, manually pull/clone.\n"; return; } printf "\n%s\n" "--- starting over with the updated version ---" - cd - || { printf "Path error.\n" ; return ; } - exec "$ScriptPath" "${ScriptArgs[@]}" # Run the new script with old arguments - exit 1 # Exit the old instance + cd - || { printf "Path error.\n"; return; } + exec "$ScriptPath" "${ScriptArgs[@]}" + exit 1 else - cd - || { printf "Path error.\n" ; return ; } + cd - || { printf "Path error.\n"; return; } self_update_curl fi } -# Choose from list function choosecontainers() { - while [[ -z "$ChoiceClean" ]]; do + while [[ -z "${ChoiceClean:-}" ]]; do read -r -p "Enter number(s) separated by comma, [a] for all - [q] to quit: " Choice - if [[ "$Choice" =~ [qQnN] ]] ; then + if [[ "$Choice" =~ [qQnN] ]]; then exit 0 - elif [[ "$Choice" =~ [aAyY] ]] ; then + elif [[ "$Choice" =~ [aAyY] ]]; then SelectedUpdates=( "${GotUpdates[@]}" ) ChoiceClean=${Choice//[,.:;]/ } else ChoiceClean=${Choice//[,.:;]/ } - for CC in $ChoiceClean ; do - if [[ "$CC" -lt 1 || "$CC" -gt $UpdCount ]] ; then - echo "Number not in list: $CC" ; unset ChoiceClean ; break 1 + for CC in $ChoiceClean; do + if [[ "$CC" -lt 1 || "$CC" -gt $UpdCount ]]; then + echo "Number not in list: $CC" + unset ChoiceClean + break 1 else SelectedUpdates+=( "${GotUpdates[$CC-1]}" ) fi @@ -127,9 +163,14 @@ choosecontainers() { } datecheck() { - ImageDate=$($regbin -v error image inspect "$RepoUrl" --format='{{.Created}}' | cut -d" " -f1 ) - ImageAge=$(( ( $(date +%s) - $(date -d "$ImageDate" +%s) )/86400 )) - if [ "$ImageAge" -gt "$DaysOld" ] ; then + if [[ -z "${DaysOld:-}" ]]; then + return 0 + fi + if ! ImageDate=$($regbin -v error image inspect "$RepoUrl" --format='{{.Created}}' 2>/dev/null | cut -d" " -f1); then + return 1 + fi + ImageAge=$(( ( $(date +%s) - $(date -d "$ImageDate" +%s) ) / 86400 )) + if [ "$ImageAge" -gt "$DaysOld" ]; then return 0 else return 1 @@ -140,198 +181,249 @@ progress_bar() { QueCurrent="$1" QueTotal="$2" ((Percent=100*QueCurrent/QueTotal)) - ((Complete=50*Percent/100)) # Change first number for width (50) - ((Left=50-Complete)) # Change first number for width (50) + ((Complete=50*Percent/100)) + ((Left=50-Complete)) BarComplete=$(printf "%${Complete}s" | tr " " "#") BarLeft=$(printf "%${Left}s" | tr " " "-") - [[ "$QueTotal" == "$QueCurrent" ]] || printf "\r[%s%s] %s/%s " "$BarComplete" "$BarLeft" "$QueCurrent" "$QueTotal" - [[ "$QueTotal" == "$QueCurrent" ]] && printf "\r[%b%s%b] %s/%s \n" "$c_teal" "$BarComplete" "$c_reset" "$QueCurrent" "$QueTotal" + if [[ "$QueTotal" != "$QueCurrent" ]]; then + printf "\r[%s%s] %s/%s " "$BarComplete" "$BarLeft" "$QueCurrent" "$QueTotal" + else + printf "\r[%b%s%b] %s/%s \n" "$c_teal" "$BarComplete" "$c_reset" "$QueCurrent" "$QueTotal" + fi } -# Static binary downloader for dependencies +t_out=$(command -v timeout 2>/dev/null || echo "") +if [[ -n "$t_out" ]]; then + t_out=$(realpath "$t_out" 2>/dev/null || readlink -f "$t_out") + if [[ "$t_out" =~ "busybox" ]]; then + t_out="timeout ${Timeout}" + else + t_out="timeout --foreground ${Timeout}" + fi +else + t_out="" +fi + binary_downloader() { BinaryName="$1" BinaryUrl="$2" case "$(uname --machine)" in x86_64|amd64) architecture="amd64" ;; - arm64|aarch64) architecture="arm64";; - *) printf "\n%bArchitecture not supported, exiting.%b\n" "$c_red" "$c_reset" ; exit 1;; + arm64|aarch64) architecture="arm64" ;; + *) printf "\n%bArchitecture not supported, exiting.%b\n" "$c_red" "$c_reset"; exit 1 ;; esac GetUrl="${BinaryUrl/TEMP/"$architecture"}" - if [[ $(command -v curl) ]]; then curl -L $GetUrl > "$ScriptWorkDir/$BinaryName" ; - elif [[ $(command -v wget) ]]; then wget $GetUrl -O "$ScriptWorkDir/$BinaryName" ; - else printf "%s\n" "curl/wget not available - get $BinaryName manually from the repo link, exiting."; exit 1; + if command -v curl &>/dev/null; then + curl -L "$GetUrl" > "$ScriptWorkDir/$BinaryName" + elif command -v wget &>/dev/null; then + wget "$GetUrl" -O "$ScriptWorkDir/$BinaryName" + else + printf "%s\n" "curl/wget not available - get $BinaryName manually from the repo link, exiting." + exit 1 fi [[ -f "$ScriptWorkDir/$BinaryName" ]] && chmod +x "$ScriptWorkDir/$BinaryName" } distro_checker() { - if [[ -f /etc/arch-release ]] ; then PkgInstaller="pacman -S" - elif [[ -f /etc/redhat-release ]] ; then PkgInstaller="dnf install" - elif [[ -f /etc/SuSE-release ]] ; then PkgInstaller="zypper install" - elif [[ -f /etc/debian_version ]] ; then PkgInstaller="apt-get install" - else PkgInstaller="ERROR" ; printf "\n%bNo distribution could be determined%b, falling back to static binary.\n" "$c_yellow" "$c_reset" + if [[ -f /etc/arch-release ]]; then + PkgInstaller="pacman -S" + elif [[ -f /etc/redhat-release ]]; then + PkgInstaller="dnf install" + elif [[ -f /etc/SuSE-release ]]; then + PkgInstaller="zypper install" + elif [[ -f /etc/debian_version ]]; then + PkgInstaller="apt-get install" + else + PkgInstaller="ERROR" + printf "\n%bNo distribution could be determined%b, falling back to static binary.\n" "$c_yellow" "$c_reset" fi } -# Version check & initiate self update -if [[ "$VERSION" != "$LatestRelease" ]] && [[ -n "$LatestRelease" ]]; then - printf "New version available! %b%s%b ⇒ %b%s%b \n Change Notes: %s \n" "$c_yellow" "$VERSION" "$c_reset" "$c_green" "$LatestRelease" "$c_reset" "$LatestChanges" - if [[ -z "$AutoUp" ]] ; then - read -r -p "Would you like to update? y/[n]: " SelfUpdate - [[ "$SelfUpdate" =~ [yY] ]] && self_update - fi -fi - -# Set $1 to a variable for name filtering later -SearchName="$1" -# Create array of excludes -IFS=',' read -r -a Excludes <<< "$Exclude" ; unset IFS - -# Dependency check for jq in PATH or directory -if [[ $(command -v jq) ]]; then jqbin="jq" ; -elif [[ -f "$ScriptWorkDir/jq" ]]; then jqbin="$ScriptWorkDir/jq" ; +# Dependency check for jq +if command -v jq &>/dev/null; then + jqbin="jq" +elif [[ -f "$ScriptWorkDir/jq" ]]; then + jqbin="$ScriptWorkDir/jq" else printf "%s\n" "Required dependency 'jq' missing, do you want to install it?" read -r -p "y: With packagemanager (sudo). / s: Download static binary. y/s/[n] " GetJq - GetJq=${GetJq:-no} # set default to no if nothing is given - if [[ "$GetJq" =~ [yYsS] ]] ; then + GetJq=${GetJq:-no} + if [[ "$GetJq" =~ [yYsS] ]]; then [[ "$GetJq" =~ [yY] ]] && distro_checker - if [[ -n "$PkgInstaller" && "$PkgInstaller" != "ERROR" ]] ; then - (sudo $PkgInstaller jq) ; PkgExitcode="$?" + if [[ -n "$PkgInstaller" && "$PkgInstaller" != "ERROR" ]]; then + (sudo $PkgInstaller jq) + PkgExitcode="$?" [[ "$PkgExitcode" == 0 ]] && jqbin="jq" || printf "\n%bPackagemanager install failed%b, falling back to static binary.\n" "$c_yellow" "$c_reset" fi - if [[ "$GetJq" =~ [nN] || "$PkgInstaller" == "ERROR" || "$PkgExitcode" != 0 ]] ; then - binary_downloader "jq" "https://github.com/jqlang/jq/releases/latest/download/jq-linux-TEMP" - [[ -f "$ScriptWorkDir/jq" ]] && jqbin="$ScriptWorkDir/jq" + if [[ "$GetJq" =~ [nN] || "$PkgInstaller" == "ERROR" || "$PkgExitcode" != 0 ]]; then + binary_downloader "jq" "https://github.com/jqlang/jq/releases/latest/download/jq-linux-TEMP" + [[ -f "$ScriptWorkDir/jq" ]] && jqbin="$ScriptWorkDir/jq" fi - else printf "\n%bDependency missing, exiting.%b\n" "$c_red" "$c_reset" ; exit 1 ; + else + printf "\n%bDependency missing, exiting.%b\n" "$c_red" "$c_reset" + exit 1 fi fi -# Final check if binary is correct -$jqbin --version &> /dev/null || { printf "%s\n" "jq is not working - try to remove it and re-download it, exiting."; exit 1; } -# Dependency check for regctl in PATH or directory -if [[ $(command -v regctl) ]]; then regbin="regctl" ; -elif [[ -f "$ScriptWorkDir/regctl" ]]; then regbin="$ScriptWorkDir/regctl" ; +$jqbin --version &>/dev/null || { printf "%s\n" "jq is not working - try to remove it and re-download it, exiting."; exit 1; } + +# Dependency check for regctl +if command -v regctl &>/dev/null; then + regbin="regctl" +elif [[ -f "$ScriptWorkDir/regctl" ]]; then + regbin="$ScriptWorkDir/regctl" else read -r -p "Required dependency 'regctl' missing, do you want it downloaded? y/[n] " GetRegctl - if [[ "$GetRegctl" =~ [yY] ]] ; then + if [[ "$GetRegctl" =~ [yY] ]]; then binary_downloader "regctl" "https://github.com/regclient/regclient/releases/latest/download/regctl-linux-TEMP" - [[ -f "$ScriptWorkDir/regctl" ]] && regbin="$ScriptWorkDir/regctl" - else printf "\n%bDependency missing, exiting.%b\n" "$c_red" "$c_reset" ; exit 1 ; + if [[ -f "$ScriptWorkDir/regctl" ]]; then + regbin="$ScriptWorkDir/regctl" + else + printf "\n%bFailed to download regctl, exiting.%b\n" "$c_red" "$c_reset" + exit 1 + fi + else + printf "\n%bDependency missing, exiting.%b\n" "$c_red" "$c_reset" + exit 1 fi fi -# Final check if binary is correct -$regbin version &> /dev/null || { printf "%s\n" "regctl is not working - try to remove it and re-download it, exiting."; exit 1; } + +$regbin version &>/dev/null || { printf "%s\n" "regctl is not working - try to remove it and re-download it, exiting."; exit 1; } # Check podman compose binary -if podman compose version &> /dev/null ; then PodmanComposeBin="podman compose" ; -elif command -v podman-compose &> /dev/null; then PodmanComposeBin="podman-compose" ; -elif podman version &> /dev/null; then +if podman compose version &>/dev/null; then + PodmanComposeBin="podman compose" +elif command -v podman-compose &>/dev/null; then + PodmanComposeBin="podman-compose" +elif podman version &>/dev/null; then printf "%s\n" "No podman-compose binary available, using plain podman" else printf "%s\n" "No podman binaries available, exiting." exit 1 fi -# Numbered List function options() { -num=1 -for i in "${GotUpdates[@]}"; do - echo "$num) $i" - ((num++)) -done + num=1 + for i in "${GotUpdates[@]}"; do + echo "$num) $i" + ((num++)) + done } -# Listing typed exclusions -if [[ -n ${Excludes[*]} ]] ; then +if [[ -n "${Excludes[*]}" ]]; then printf "\n%bExcluding these names:%b\n" "$c_blue" "$c_reset" printf "%s\n" "${Excludes[@]}" printf "\n" fi -# Variables for progress_bar function ContCount=$(podman ps $Stopped --filter "name=$SearchName" --format '{{.Names}}' | wc -l) RegCheckQue=0 +start_time=$(date +%s) -# Testing and setting timeout binary -t_out=$(command -v timeout) -if [[ $t_out ]]; then - t_out=$(realpath $t_out 2>/dev/null || readlink -f $t_out) - if [[ $t_out =~ "busybox" ]]; then - t_out="timeout ${Timeout}" - else t_out="timeout --foreground ${Timeout}" - fi -else t_out="" -fi - -# Check the image-hash of every running container VS the registry -for i in $(podman ps $Stopped --filter "name=$SearchName" --format '{{.Names}}') ; do - ((RegCheckQue+=1)) +process_container() { + local container="$1" + ((RegCheckQue++)) progress_bar "$RegCheckQue" "$ContCount" - # Looping every item over the list of excluded names and skipping - for e in "${Excludes[@]}" ; do [[ "$i" == "$e" ]] && continue 2 ; done - RepoUrl=$(podman inspect "$i" --format='{{.ImageName}}') - LocalHash=$(podman image inspect "$RepoUrl" --format '{{.RepoDigests}}') - # Checking for errors while setting the variable - if RegHash=$(${t_out} $regbin -v error image digest --list "$RepoUrl" 2>&1) ; then - if [[ "$LocalHash" == *"$RegHash"* ]] ; then - NoUpdates+=("$i") - else - if [[ -n "$DaysOld" ]] && ! datecheck ; then - NoUpdates+=("+$i ${ImageAge}d") + >&2 echo "Processing container: $container" + + for e in "${Excludes[@]}"; do + if [[ "$container" == "$e" ]]; then + return 0 + fi + done + + local ImageId RepoUrl LocalHash RegHash + if ! ImageId=$(podman inspect "$container" --format='{{.Image}}'); then + return 0 + fi + if ! RepoUrl=$(podman inspect "$container" --format='{{.ImageName}}'); then + return 0 + fi + if ! LocalHash=$(podman image inspect "$ImageId" --format '{{.RepoDigests}}'); then + return 0 + fi + + if RegHash=$(${t_out} $regbin -v error image digest --list "$RepoUrl" 2>/dev/null | xargs); then + if [[ -n "$RegHash" ]]; then + if [[ "$LocalHash" == *"$RegHash"* ]]; then + NoUpdates+=("$container") else - GotUpdates+=("$i") + # Create a separate array for notifications + NotifyUpdates+=("$container") + # Add to GotUpdates for update logic + GotUpdates+=("$container") + + # If it's too recent based on age check, move it to NoUpdates for display + # but keep it in NotifyUpdates + if [[ -n "${DaysOld:-}" ]] && ! datecheck; then + NoUpdates+=("+$container ${ImageAge}d") + # Remove from GotUpdates for update logic + for i in "${!GotUpdates[@]}"; do + if [[ "${GotUpdates[i]}" = "$container" ]]; then + unset 'GotUpdates[i]' + break + fi + done + # Re-index array after removal + GotUpdates=("${GotUpdates[@]}") + fi fi + else + GotErrors+=("$container - No digest returned") fi else - # Here the RegHash is the result of an error code - GotErrors+=("$i - ${RegHash}") + GotErrors+=("$container - Error checking registry") fi +} + +# Main loop to process all containers +for container in $(podman ps $Stopped --filter "name=$SearchName" --format '{{.Names}}'); do + process_container "$container" || true done -# Sort arrays alphabetically IFS=$'\n' NoUpdates=($(sort <<<"${NoUpdates[*]}")) GotUpdates=($(sort <<<"${GotUpdates[*]}")) unset IFS -# Define how many updates are available -UpdCount="${#GotUpdates[@]}" - -# List what containers got updates or not -if [[ -n ${NoUpdates[*]} ]] ; then +echo "" +echo "===== Summary =====" +if [[ -n "${NoUpdates[*]}" ]]; then printf "\n%bContainers on latest version:%b\n" "$c_green" "$c_reset" printf "%s\n" "${NoUpdates[@]}" fi -if [[ -n ${GotErrors[*]} ]] ; then +if [[ -n "${GotErrors[*]}" ]]; then printf "\n%bContainers with errors; won't get updated:%b\n" "$c_red" "$c_reset" printf "%s\n" "${GotErrors[@]}" printf "%binfo:%b 'unauthorized' often means not found in a public registry.\n" "$c_blue" "$c_reset" fi -if [[ -n ${GotUpdates[*]} ]] ; then - printf "\n%bContainers with updates available:%b\n" "$c_yellow" "$c_reset" - [[ -z "$AutoUp" ]] && options || printf "%s\n" "${GotUpdates[@]}" - [[ -n "$Notify" ]] && { [[ $(type -t send_notification) == function ]] && send_notification "${GotUpdates[@]}" || printf "Could not source notification function.\n" ; } +if [[ -n "${GotUpdates[*]}" ]]; then + printf "\n%bContainers with updates available:%b\n" "$c_yellow" "$c_reset" + printf "%s\n" "${GotUpdates[@]}" fi -# Optionally get updates if there's any -if [ -n "$GotUpdates" ] ; then - if [ -z "$AutoUp" ] ; then - printf "\n%bChoose what containers to update.%b\n" "$c_teal" "$c_reset" - choosecontainers - else +if [[ -n "${GotUpdates[*]}" ]]; then + UpdCount="${#GotUpdates[@]}" + + # Send notification if -i flag was used, regardless of other options + [[ "${Notify:-}" == "yes" && -n "${NotifyUpdates[*]}" ]] && send_notification "${NotifyUpdates[@]}" + + if [[ "$NoUpdateMode" == true ]]; then + printf "\n%bNo updates will be performed due to -n flag.%b\n" "$c_blue" "$c_reset" + elif [[ "$AutoUp" == "yes" ]]; then SelectedUpdates=( "${GotUpdates[@]}" ) + else + printf "\n%bChoose what containers to update:%b\n" "$c_teal" "$c_reset" + options + choosecontainers fi - if [ "$AutoUp" == "${AutoUp#[Nn]}" ] ; then + + if [ "${#SelectedUpdates[@]}" -gt 0 ]; then NumberofUpdates="${#SelectedUpdates[@]}" CurrentQue=0 - for i in "${SelectedUpdates[@]}" - do + for i in "${SelectedUpdates[@]}"; do ((CurrentQue+=1)) unset CompleteConfs - # Extract labels and metadata ContLabels=$(podman inspect "$i" --format '{{json .Config.Labels}}') ContImage=$(podman inspect "$i" --format='{{.ImageName}}') ContPath=$($jqbin -r '."com.docker.compose.project.working_dir"' <<< "$ContLabels") @@ -347,52 +439,73 @@ if [ -n "$GotUpdates" ] ; then ContRestartStack=$($jqbin -r '."sudo-kraken.podcheck.restart-stack"' <<< "$ContLabels") [ "$ContRestartStack" == "null" ] && ContRestartStack="" - # Checking if compose-values are empty - possibly started with podman run or managed by Quadlet - if [ -z "$ContPath" ] ; then - # Check if a systemd unit exists with the same name as the container - if systemctl --user status "$i.service" &> /dev/null; then - echo "Detected Quadlet-managed container: $i (unit: $i.service)" - podman pull "$ContImage" - systemctl --user restart "$i.service" - echo "Quadlet container $i updated and restarted." - elif [ "$(id -u)" -eq 0 ] && systemctl status "$i.service" &> /dev/null; then - echo "Detected Quadlet-managed container: $i (unit: $i.service)" - podman pull "$ContImage" - systemctl restart "$i.service" - echo "Quadlet container $i updated and restarted." + if [ -z "$ContPath" ]; then + if systemctl --user status "$i.service" &>/dev/null; then + unit="$i.service" + elif [ "$(id -u)" -eq 0 ] && systemctl status "$i.service" &>/dev/null; then + unit="$i.service" else - if [ "$DRunUp" == "yes" ] ; then + pattern="^$(echo "$i" | sed 's/_/[_-]/g')\.service$" + candidates=$(systemctl --user list-units --type=service --no-legend | awk '{print $1}' | grep -iE "$pattern") + if [ "$(echo "$candidates" | wc -l)" -eq 1 ]; then + unit="$candidates" + elif [ "$(echo "$candidates" | wc -l)" -gt 1 ]; then + for cand in $candidates; do + if [[ "${cand,,}" == "${i,,}.service" ]]; then + unit="$cand" + break + fi + done + if [ -z "${unit:-}" ]; then + unit=$(echo "$candidates" | head -n 1) + fi + fi + fi + + if [ -n "${unit:-}" ]; then + echo "Detected Quadlet-managed container: $i (matched unit: $unit)" + podman pull "$ContImage" + if systemctl --user restart "$unit" &>/dev/null; then + echo "Quadlet container $i updated and restarted (user scope)." + elif [ "$(id -u)" -eq 0 ] && systemctl restart "$unit" &>/dev/null; then + echo "Quadlet container $i updated and restarted (system scope)." + else + echo "Failed to restart unit $unit for container $i." + fi + else + if [ "$DRunUp" == "yes" ]; then podman pull "$ContImage" printf "%s\n" "$i got a new image downloaded; rebuild manually with preferred 'podman run' parameters" else printf "\n%b%s%b has no compose labels or associated systemd unit; %bskipping%b\n\n" "$c_yellow" "$i" "$c_reset" "$c_yellow" "$c_reset" fi fi + continue fi - # cd to the compose-file directory to account for people who use relative volumes - cd "$ContPath" || { echo "Path error - skipping $i" ; continue ; } - # Reformatting path + multi compose - if [[ $ContConfigFile = '/'* ]] ; then - CompleteConfs=$(for conf in ${ContConfigFile//,/ } ; do printf -- "-f %s " "$conf"; done) + cd "$ContPath" || { echo "Path error - skipping $i"; continue; } + if [[ $ContConfigFile = /* ]]; then + CompleteConfs=$(for conf in ${ContConfigFile//,/ }; do printf -- "-f %s " "$conf"; done) else - CompleteConfs=$(for conf in ${ContConfigFile//,/ } ; do printf -- "-f %s/%s " "$ContPath" "$conf"; done) + CompleteConfs=$(for conf in ${ContConfigFile//,/ }; do printf -- "-f %s/%s " "$ContPath" "$conf"; done) fi printf "\n%bNow updating (%s/%s): %b%s%b\n" "$c_teal" "$CurrentQue" "$NumberofUpdates" "$c_blue" "$i" "$c_reset" - # Checking if Label Only option is set, and if container got the label - [[ "$OnlyLabel" == true ]] && { [[ "$ContUpdateLabel" != "true" ]] && { echo "No update label, skipping." ; continue ; } } + [[ "$OnlyLabel" == true ]] && { [[ "$ContUpdateLabel" != "true" ]] && { echo "No update label, skipping."; continue; } } podman pull "$ContImage" - # Check if the container got an environment file set and reformat it - if [ -n "$ContEnv" ]; then ContEnvs=$(for env in ${ContEnv//,/ } ; do printf -- "--env-file %s " "$env"; done) ; fi - # Check if the whole pod should be restarted - if [[ "$ContRestartStack" == "true" ]] || [[ "$ForceRestartPods" == true ]] ; then - $PodmanComposeBin ${CompleteConfs} down ; $PodmanComposeBin ${CompleteConfs} ${ContEnvs} up -d + if [ -n "$ContEnv" ]; then + ContEnvs=$(for env in ${ContEnv//,/ }; do printf -- "--env-file %s " "$env"; done) + fi + if [[ "$ContRestartStack" == "true" ]] || [[ "$ForceRestartPods" == true ]]; then + $PodmanComposeBin ${CompleteConfs} down + $PodmanComposeBin ${CompleteConfs} ${ContEnvs} up -d else $PodmanComposeBin ${CompleteConfs} ${ContEnvs} up -d ${ContName} fi done printf "\n%bAll done!%b\n" "$c_green" "$c_reset" - if [[ -z "$AutoPrune" ]] && [[ -z "$AutoUp" ]]; then read -r -p "Would you like to prune dangling images? y/[n]: " AutoPrune ; fi + if [[ -z "$AutoPrune" ]] && [[ "$AutoUp" == "no" ]]; then + read -r -p "Would you like to prune dangling images? y/[n]: " AutoPrune + fi [[ "$AutoPrune" =~ [yY] ]] && podman image prune -f else printf "\nNo updates installed, exiting.\n" @@ -401,4 +514,42 @@ else printf "\nNo updates available, exiting.\n" fi +# Export metrics if collector directory was specified +if [[ -n "${CollectorTextFileDirectory:-}" ]]; then + # Calculate check duration + end_time=$(date +%s) + check_duration=$((end_time - start_time)) + + # Source the prometheus collector script if it exists + if [[ -f "$ScriptWorkDir/addons/prometheus/prometheus_collector.sh" ]]; then + source "$ScriptWorkDir/addons/prometheus/prometheus_collector.sh" + # Call the prometheus_exporter with appropriate metrics + prometheus_exporter "${#NoUpdates[@]}" "${#GotUpdates[@]}" "${#GotErrors[@]}" "$ContCount" "$check_duration" + printf "\n%bPrometheus metrics exported to: %s/podcheck.prom%b\n" "$c_teal" "$CollectorTextFileDirectory" "$c_reset" + else + # Fallback if the collector script isn't found + cat > "$CollectorTextFileDirectory/podcheck.prom" <