diff --git a/add-on/blackbox-exporter.md b/add-on/blackbox-exporter.md new file mode 100644 index 0000000..33fb780 --- /dev/null +++ b/add-on/blackbox-exporter.md @@ -0,0 +1,536 @@ +BLACKBOX EXPORTER - GUIDA OPERATIVA KUBERNETES +============================================== + +Stack di partenza +----------------- +Nel cluster e' gia' installato kube-prometheus-stack nel namespace monitoring: + +```text +helm list -A | grep prometheus + +kube-prometheus-stack monitoring 4 2026-08-08 23:55:14.193903052 +0000 UTC deployed kube-prometheus-stack-88.1.5 +``` + +Obiettivo +--------- +Installare prometheus-blackbox-exporter per eseguire test HTTP/HTTPS su servizi Kubernetes interni, Ingress o URL esterni, esportando le metriche verso Prometheus tramite Prometheus Operator. + +Metriche principali: + +```text +probe_success +probe_duration_seconds +probe_http_status_code +probe_http_ssl +probe_dns_lookup_time_seconds +``` + +Legenda +------- +- COMANDO: istruzione da eseguire nel terminale. +- FILE: file da creare o aggiornare. +- NOTA: informazione da verificare prima di procedere. +- OUTPUT ATTESO: risultato indicativo del comando. + +Prerequisiti +------------ +- Accesso kubectl al cluster. +- Helm installato. +- Namespace monitoring presente o creabile. +- kube-prometheus-stack installato con Prometheus Operator. +- CRD Probe disponibile: probes.monitoring.coreos.com. + + +1. VERIFICA PROMETHEUS E PROMETHEUS OPERATOR +============================================ + +COMANDO - verifica release Helm Prometheus +```bash +helm list -A | grep prometheus +``` + +COMANDO - verifica pod Prometheus +```bash +kubectl get pods -A | grep prometheus +``` + +COMANDO - verifica CRD Probe +```bash +kubectl get crd probes.monitoring.coreos.com +``` + +COMANDO - verifica label delle risorse Prometheus +```bash +kubectl get prometheus -A --show-labels +``` + +COMANDO - controlla selector della risorsa Prometheus +```bash +kubectl get prometheus -n monitoring -o yaml +``` + +NOTA - label dei manifest +Dal comando `kubectl get prometheus -A --show-labels` risulta che Prometheus seleziona le risorse con la label `release=kube-prometheus-stack`. +Negli esempi sotto viene quindi usata questa label: + +```yaml +release: kube-prometheus-stack +``` + +Se in futuro il tuo Prometheus usa un selector diverso, sostituisci questa label nei manifest Probe e PrometheusRule. +Controlla soprattutto questi campi nella risorsa Prometheus: + +```yaml +spec: + probeSelector: + probeNamespaceSelector: + ruleSelector: + ruleNamespaceSelector: +``` + + +2. AGGIUNTA REPOSITORY HELM +=========================== + +COMANDO - aggiungi repository prometheus-community +```bash +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +``` + +COMANDO - aggiorna indice chart Helm +```bash +helm repo update +``` + + +3. CREA FILE VALUES HELM +======================== + +FILE - blackbox-values.yaml + +Descrizione: +- configura i moduli HTTP usati da blackbox-exporter; +- espone blackbox-exporter come Service ClusterIP sulla porta 9115; +- lascia disabilitato ServiceMonitor perche' lo scraping dei target viene configurato tramite Probe. + +COMANDO - crea blackbox-values.yaml +```bash +cat > blackbox-values.yaml <<'EOF' +fullnameOverride: blackbox-exporter + +config: + modules: + http_2xx: + prober: http + timeout: 10s + http: + method: GET + preferred_ip_protocol: ip4 + valid_http_versions: + - HTTP/1.1 + - HTTP/2.0 + valid_status_codes: + - 200 + - 204 + - 301 + - 302 + + http_post_2xx: + prober: http + timeout: 10s + http: + method: POST + preferred_ip_protocol: ip4 + valid_status_codes: + - 200 + - 201 + - 202 + - 204 + + http_k8s_health: + prober: http + timeout: 10s + http: + method: GET + preferred_ip_protocol: ip4 + fail_if_ssl: false + fail_if_not_ssl: false + valid_status_codes: + - 200 + +service: + type: ClusterIP + port: 9115 + +serviceMonitor: + enabled: false + +resources: + requests: + cpu: 50m + memory: 64Mi + limits: + cpu: 200m + memory: 256Mi +EOF +``` + + +4. INSTALLA BLACKBOX EXPORTER +============================= + +COMANDO - installa o aggiorna blackbox-exporter +```bash +helm upgrade --install blackbox-exporter prometheus-community/prometheus-blackbox-exporter \ + --namespace monitoring \ + --create-namespace \ + -f blackbox-values.yaml +``` + +COMANDO - verifica pod blackbox-exporter +```bash +kubectl get pods -n monitoring | grep blackbox +``` + +COMANDO - verifica Service blackbox-exporter +```bash +kubectl get svc -n monitoring | grep blackbox +``` + +OUTPUT ATTESO +```text +blackbox-exporter ClusterIP 9115/TCP +``` + + +5. CREA PROBE HTTP PER SERVIZI KUBERNETES INTERNI +================================================= + +FILE - blackbox-probes.yaml + +Descrizione: +- testa endpoint HTTP interni tramite DNS Kubernetes; +- usa il modulo http_k8s_health definito in blackbox-values.yaml; +- invia le metriche a Prometheus tramite la risorsa Probe del Prometheus Operator. + +Da personalizzare: +- my-service +- my-namespace +- porta del servizio +- path HTTP, ad esempio /health, /ready o / +- label release se diversa da prometheus + +COMANDO - crea blackbox-probes.yaml +```bash +cat > blackbox-probes.yaml <<'EOF' +apiVersion: monitoring.coreos.com/v1 +kind: Probe +metadata: + name: http-services-probe + namespace: monitoring + labels: + release: kube-prometheus-stack +spec: + jobName: http-services-probe + interval: 30s + scrapeTimeout: 15s + module: http_k8s_health + prober: + url: blackbox-exporter.monitoring.svc.cluster.local:9115 + scheme: http + path: /probe + targets: + staticConfig: + static: + - http://frontend.idcidp-dev.svc.cluster.local +EOF +``` + +COMANDO - applica Probe servizi interni +```bash +kubectl apply -f blackbox-probes.yaml +``` + +COMANDO - verifica Probe +```bash +kubectl get probe -n monitoring +kubectl describe probe http-services-probe -n monitoring +``` + +NOTA - campi importanti della Probe +```yaml +spec: + module: http_k8s_health + prober: + url: blackbox-exporter.monitoring.svc.cluster.local:9115 + targets: + staticConfig: + static: + - http://my-service.my-namespace.svc.cluster.local:8080/health +``` + + +6. CREA PROBE HTTP/HTTPS PER INGRESS O URL ESTERNI +================================================== + +FILE - blackbox-ingress-probes.yaml + +Descrizione: +- testa URL esposti tramite Ingress, Gateway, reverse proxy o endpoint pubblici; +- usa il modulo generico http_2xx. + +COMANDO - crea blackbox-ingress-probes.yaml +```bash +cat > blackbox-ingress-probes.yaml <<'EOF' +apiVersion: monitoring.coreos.com/v1 +kind: Probe +metadata: + name: ingress-http-probe + namespace: monitoring + labels: + release: kube-prometheus-stack +spec: + jobName: ingress-http-probe + interval: 30s + scrapeTimeout: 15s + module: http_2xx + prober: + url: blackbox-exporter.monitoring.svc.cluster.local:9115 + scheme: http + path: /probe + targets: + staticConfig: + static: + - https://app.example.com/health + - https://api.example.com/ready +EOF +``` + +COMANDO - applica Probe endpoint esterni +```bash +kubectl apply -f blackbox-ingress-probes.yaml +``` + + +7. VERIFICA METRICHE IN PROMETHEUS +================================== + +COMANDO - apri Prometheus in locale con port-forward +```bash +kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 +``` + +Poi apri nel browser: + +```text +http://localhost:9090 +``` + +QUERY PROMQL - tutte le probe +```promql +probe_success +``` + +QUERY PROMQL - probe servizi interni +```promql +probe_success{job="http-services-probe"} +``` + +QUERY PROMQL - status code HTTP +```promql +probe_http_status_code{job="http-services-probe"} +``` + +QUERY PROMQL - durata probe +```promql +probe_duration_seconds{job="http-services-probe"} +``` + +QUERY PROMQL - target falliti +```promql +probe_success{job="http-services-probe"} == 0 +``` + +Interpretazione: +- probe_success = 1: test riuscito. +- probe_success = 0: test fallito. +- probe_http_status_code: status HTTP ricevuto. +- probe_duration_seconds: durata della probe. + + +8. CREA ALERT PROMETHEUSRULE +============================ + +FILE - blackbox-http-alerts.yaml + +Descrizione: +- genera alert quando un target HTTP non risponde correttamente; +- genera alert quando un target risponde troppo lentamente. + +COMANDO - crea blackbox-http-alerts.yaml +```bash +cat > blackbox-http-alerts.yaml <<'EOF' +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: blackbox-http-alerts + namespace: monitoring + labels: + release: kube-prometheus-stack +spec: + groups: + - name: blackbox-http + rules: + - alert: BlackboxHttpProbeFailed + expr: probe_success == 0 + for: 2m + labels: + severity: warning + annotations: + summary: "HTTP probe fallita" + description: "Il target {{ $labels.instance }} non risponde correttamente da almeno 2 minuti." + + - alert: BlackboxHttpSlowResponse + expr: probe_duration_seconds > 2 + for: 5m + labels: + severity: warning + annotations: + summary: "HTTP probe lenta" + description: "Il target {{ $labels.instance }} risponde in piu' di 2 secondi da almeno 5 minuti." +EOF +``` + +COMANDO - applica alert +```bash +kubectl apply -f blackbox-http-alerts.yaml +``` + +COMANDO - verifica PrometheusRule +```bash +kubectl get prometheusrule -n monitoring | grep blackbox +kubectl describe prometheusrule blackbox-http-alerts -n monitoring +``` + + +9. TEST MANUALE BLACKBOX EXPORTER +================================= + +COMANDO - port-forward blackbox-exporter +```bash +kubectl port-forward -n monitoring svc/blackbox-exporter 9115:9115 +``` + +COMANDO - esegui probe manuale da un altro terminale +```bash +curl "http://localhost:9115/probe?target=http://my-service.my-namespace.svc.cluster.local:8080/health&module=http_k8s_health" +``` + +OUTPUT ATTESO - metriche indicative +```text +probe_success 1 +probe_http_status_code 200 +``` + + +10. TROUBLESHOOTING +=================== + +CASO - Prometheus non vede la Probe + +COMANDO +```bash +kubectl get probe -A +kubectl describe probe -n monitoring http-services-probe +kubectl get prometheus -n monitoring -o yaml +``` + +Controlla: +- spec.probeSelector +- spec.probeNamespaceSelector +- metadata.labels della Probe + +CASO - blackbox-exporter non parte + +COMANDO +```bash +kubectl get pods -n monitoring | grep blackbox +kubectl logs -n monitoring deploy/blackbox-exporter +``` + +CASO - servizio target non raggiungibile + +COMANDO +```bash +kubectl get svc -n my-namespace +kubectl get endpoints -n my-namespace my-service +kubectl describe svc -n my-namespace my-service +``` + +CASO - DNS Kubernetes non risolve + +COMANDO +```bash +kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- nslookup my-service.my-namespace.svc.cluster.local +``` + +CASO - endpoint HTTPS fallisce per certificati, redirect o header + +Azioni consigliate: +- usare il modulo http_2xx per endpoint esterni generici; +- verificare se l'endpoint richiede SNI, autenticazione, header custom o path diverso; +- aggiungere un modulo dedicato in blackbox-values.yaml se serve una configurazione HTTP specifica. + + +11. ORDINE DI ESECUZIONE CONSIGLIATO +==================================== + +COMANDO - sequenza completa +```bash +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update + +helm upgrade --install blackbox-exporter prometheus-community/prometheus-blackbox-exporter \ + --namespace monitoring \ + --create-namespace \ + -f blackbox-values.yaml + +kubectl apply -f blackbox-probes.yaml +kubectl apply -f blackbox-ingress-probes.yaml +kubectl apply -f blackbox-http-alerts.yaml +``` + +FILE CONSIGLIATI DA VERSIONARE +```text +blackbox-values.yaml +blackbox-probes.yaml +blackbox-ingress-probes.yaml +blackbox-http-alerts.yaml +``` + + +12. NOTE PER DEV, QA E PROD +=========================== + +Per separare gli ambienti e semplificare dashboard e alert, usa Probe distinte: + +```text +dev-http-services-probe +qa-http-services-probe +prod-http-services-probe +``` + +QUERY PROMQL - ambiente dev +```promql +probe_success{job="dev-http-services-probe"} +``` + +QUERY PROMQL - target prod falliti +```promql +probe_success{job="prod-http-services-probe"} == 0 +``` + +Per endpoint critici in produzione, valuta: +- interval piu' basso, ad esempio 15s o 30s; +- durata for degli alert tra 1m e 5m in base alla criticita'; +- dashboard Grafana con stato, status code e latenza per target. \ No newline at end of file diff --git a/pipeline/customize.sh b/pipeline/customize.sh index 35cb2a1..2289fb8 100644 --- a/pipeline/customize.sh +++ b/pipeline/customize.sh @@ -50,11 +50,15 @@ printf '\nenv=%s\n' "$ENV" >> "$TMP_VALUES_FILE" printf 'namespace=%s\n' "$NAMESPACE" >> "$TMP_VALUES_FILE" printf 'project=%s\n' "$PROJECT_NAME" >> "$TMP_VALUES_FILE" -# Ricava endpoint se presente e genera endpoint-nodot sostituendo '.' con '-' +# Ricava endpoint se presente e genera le variabili derivate per le sostituzioni YAML. ENDPOINT_VAL=$(grep -E '^[[:space:]]*endpoint[[:space:]]*=' "$TMP_VALUES_FILE" "$PROPERTIES_FILE" 2>/dev/null | tail -n 1 | sed -E 's/.*endpoint[[:space:]]*=[[:space:]]*//' | tr -d '\r\n[:space:]"' || true) if [ -n "$ENDPOINT_VAL" ]; then ENDPOINT_NODOT=$(echo "$ENDPOINT_VAL" | tr '.' '-') + ENDPOINT_NOSPACE=$(echo "$ENDPOINT_VAL" | tr -d '.') + printf 'endpoint-nodot=%s\n' "$ENDPOINT_NODOT" >> "$TMP_VALUES_FILE" + printf 'endpoint-nospace=%s\n' "$ENDPOINT_NOSPACE" >> "$TMP_VALUES_FILE" + fi # Trova tutti i file .yaml nella directory kubernetes e sottodirectory diff --git a/pipeline/deploy.sh b/pipeline/deploy.sh index 1ea3c62..869d222 100644 --- a/pipeline/deploy.sh +++ b/pipeline/deploy.sh @@ -7,44 +7,7 @@ ENVIRONMENT="${1:-dev}" YAML_DIR="kubernetes" -# --------------------------------------------------------------------------- -# Cerca risorse postgresql.cnpg.io/v1 nei manifest e deploya una ConfigMap -# --------------------------------------------------------------------------- -CNPG_FILE=$(grep -rl "postgresql.cnpg.io/v1" "$YAML_DIR" 2>/dev/null | head -1 || true) -if [ -n "$CNPG_FILE" ]; then - echo "Trovata risorsa postgresql.cnpg.io/v1 in: $CNPG_FILE" - - # Estrae metadata.name dal manifest CNPG preferendo yq, altrimenti awk - if command -v yq >/dev/null 2>&1; then - PG_NAME=$(yq eval 'select(.apiVersion == "postgresql.cnpg.io/v1") | .metadata.name' "$CNPG_FILE") - else - PG_NAME=$(awk '/postgresql\.cnpg\.io\/v1/{found=1} found && /^metadata:/{meta=1} meta && /^\s+name:/{print $2; exit}' "$CNPG_FILE") - fi - - # Ricava il namespace dal Role namespace-deployer presente nel cluster - PG_NS=$(kubectl --kubeconfig=./kubeconfig get role namespace-deployer \ - --no-headers \ - -o custom-columns='NS:.metadata.namespace' 2>/dev/null | head -1 || true) - #PG_NS="${PG_NS:-default}" - - if [ -z "$PG_NAME" ]; then - echo "⚠️ Impossibile estrarre metadata.name dal cluster CNPG, skip ConfigMap." >&2 - else - echo " → cluster: $PG_NAME namespace: $PG_NS" - kubectl --kubeconfig=./kubeconfig apply -f - <