ff

add-on/blackbox-exporter.md

@@ -0,0 +1,536 @@
+BLACKBOX EXPORTER - GUIDA OPERATIVA KUBERNETES
+==============================================
+
+Stack di partenza
+-----------------
+Nel cluster e' gia' installato kube-prometheus-stack nel namespace monitoring:
+
+```text
+helm list -A | grep prometheus
+
+kube-prometheus-stack  monitoring  4  2026-08-08 23:55:14.193903052 +0000 UTC  deployed  kube-prometheus-stack-88.1.5
+```
+
+Obiettivo
+---------
+Installare prometheus-blackbox-exporter per eseguire test HTTP/HTTPS su servizi Kubernetes interni, Ingress o URL esterni, esportando le metriche verso Prometheus tramite Prometheus Operator.
+
+Metriche principali:
+
+```text
+probe_success
+probe_duration_seconds
+probe_http_status_code
+probe_http_ssl
+probe_dns_lookup_time_seconds
+```
+
+Legenda
+-------
+- COMANDO: istruzione da eseguire nel terminale.
+- FILE: file da creare o aggiornare.
+- NOTA: informazione da verificare prima di procedere.
+- OUTPUT ATTESO: risultato indicativo del comando.
+
+Prerequisiti
+------------
+- Accesso kubectl al cluster.
+- Helm installato.
+- Namespace monitoring presente o creabile.
+- kube-prometheus-stack installato con Prometheus Operator.
+- CRD Probe disponibile: probes.monitoring.coreos.com.
+
+
+1. VERIFICA PROMETHEUS E PROMETHEUS OPERATOR
+============================================
+
+COMANDO - verifica release Helm Prometheus
+```bash
+helm list -A | grep prometheus
+```
+
+COMANDO - verifica pod Prometheus
+```bash
+kubectl get pods -A | grep prometheus
+```
+
+COMANDO - verifica CRD Probe
+```bash
+kubectl get crd probes.monitoring.coreos.com
+```
+
+COMANDO - verifica label delle risorse Prometheus
+```bash
+kubectl get prometheus -A --show-labels
+```
+
+COMANDO - controlla selector della risorsa Prometheus
+```bash
+kubectl get prometheus -n monitoring -o yaml
+```
+
+NOTA - label dei manifest
+Dal comando `kubectl get prometheus -A --show-labels` risulta che Prometheus seleziona le risorse con la label `release=kube-prometheus-stack`.
+Negli esempi sotto viene quindi usata questa label:
+
+```yaml
+release: kube-prometheus-stack
+```
+
+Se in futuro il tuo Prometheus usa un selector diverso, sostituisci questa label nei manifest Probe e PrometheusRule.
+Controlla soprattutto questi campi nella risorsa Prometheus:
+
+```yaml
+spec:
+  probeSelector:
+  probeNamespaceSelector:
+  ruleSelector:
+  ruleNamespaceSelector:
+```
+
+
+2. AGGIUNTA REPOSITORY HELM
+===========================
+
+COMANDO - aggiungi repository prometheus-community
+```bash
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+```
+
+COMANDO - aggiorna indice chart Helm
+```bash
+helm repo update
+```
+
+
+3. CREA FILE VALUES HELM
+========================
+
+FILE - blackbox-values.yaml
+
+Descrizione:
+- configura i moduli HTTP usati da blackbox-exporter;
+- espone blackbox-exporter come Service ClusterIP sulla porta 9115;
+- lascia disabilitato ServiceMonitor perche' lo scraping dei target viene configurato tramite Probe.
+
+COMANDO - crea blackbox-values.yaml
+```bash
+cat > blackbox-values.yaml <<'EOF'
+fullnameOverride: blackbox-exporter
+
+config:
+  modules:
+    http_2xx:
+      prober: http
+      timeout: 10s
+      http:
+        method: GET
+        preferred_ip_protocol: ip4
+        valid_http_versions:
+          - HTTP/1.1
+          - HTTP/2.0
+        valid_status_codes:
+          - 200
+          - 204
+          - 301
+          - 302
+
+    http_post_2xx:
+      prober: http
+      timeout: 10s
+      http:
+        method: POST
+        preferred_ip_protocol: ip4
+        valid_status_codes:
+          - 200
+          - 201
+          - 202
+          - 204
+
+    http_k8s_health:
+      prober: http
+      timeout: 10s
+      http:
+        method: GET
+        preferred_ip_protocol: ip4
+        fail_if_ssl: false
+        fail_if_not_ssl: false
+        valid_status_codes:
+          - 200
+
+service:
+  type: ClusterIP
+  port: 9115
+
+serviceMonitor:
+  enabled: false
+
+resources:
+  requests:
+    cpu: 50m
+    memory: 64Mi
+  limits:
+    cpu: 200m
+    memory: 256Mi
+EOF
+```
+
+
+4. INSTALLA BLACKBOX EXPORTER
+=============================
+
+COMANDO - installa o aggiorna blackbox-exporter
+```bash
+helm upgrade --install blackbox-exporter prometheus-community/prometheus-blackbox-exporter \
+  --namespace monitoring \
+  --create-namespace \
+  -f blackbox-values.yaml
+```
+
+COMANDO - verifica pod blackbox-exporter
+```bash
+kubectl get pods -n monitoring | grep blackbox
+```
+
+COMANDO - verifica Service blackbox-exporter
+```bash
+kubectl get svc -n monitoring | grep blackbox
+```
+
+OUTPUT ATTESO
+```text
+blackbox-exporter   ClusterIP   <cluster-ip>   <none>   9115/TCP
+```
+
+
+5. CREA PROBE HTTP PER SERVIZI KUBERNETES INTERNI
+=================================================
+
+FILE - blackbox-probes.yaml
+
+Descrizione:
+- testa endpoint HTTP interni tramite DNS Kubernetes;
+- usa il modulo http_k8s_health definito in blackbox-values.yaml;
+- invia le metriche a Prometheus tramite la risorsa Probe del Prometheus Operator.
+
+Da personalizzare:
+- my-service
+- my-namespace
+- porta del servizio
+- path HTTP, ad esempio /health, /ready o /
+- label release se diversa da prometheus
+
+COMANDO - crea blackbox-probes.yaml
+```bash
+cat > blackbox-probes.yaml <<'EOF'
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: http-services-probe
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  jobName: http-services-probe
+  interval: 30s
+  scrapeTimeout: 15s
+  module: http_k8s_health
+  prober:
+    url: blackbox-exporter.monitoring.svc.cluster.local:9115
+    scheme: http
+    path: /probe
+  targets:
+    staticConfig:
+      static:
+        - http://frontend.idcidp-dev.svc.cluster.local
+EOF
+```
+
+COMANDO - applica Probe servizi interni
+```bash
+kubectl apply -f blackbox-probes.yaml
+```
+
+COMANDO - verifica Probe
+```bash
+kubectl get probe -n monitoring
+kubectl describe probe http-services-probe -n monitoring
+```
+
+NOTA - campi importanti della Probe
+```yaml
+spec:
+  module: http_k8s_health
+  prober:
+    url: blackbox-exporter.monitoring.svc.cluster.local:9115
+  targets:
+    staticConfig:
+      static:
+        - http://my-service.my-namespace.svc.cluster.local:8080/health
+```
+
+
+6. CREA PROBE HTTP/HTTPS PER INGRESS O URL ESTERNI
+==================================================
+
+FILE - blackbox-ingress-probes.yaml
+
+Descrizione:
+- testa URL esposti tramite Ingress, Gateway, reverse proxy o endpoint pubblici;
+- usa il modulo generico http_2xx.
+
+COMANDO - crea blackbox-ingress-probes.yaml
+```bash
+cat > blackbox-ingress-probes.yaml <<'EOF'
+apiVersion: monitoring.coreos.com/v1
+kind: Probe
+metadata:
+  name: ingress-http-probe
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  jobName: ingress-http-probe
+  interval: 30s
+  scrapeTimeout: 15s
+  module: http_2xx
+  prober:
+    url: blackbox-exporter.monitoring.svc.cluster.local:9115
+    scheme: http
+    path: /probe
+  targets:
+    staticConfig:
+      static:
+        - https://app.example.com/health
+        - https://api.example.com/ready
+EOF
+```
+
+COMANDO - applica Probe endpoint esterni
+```bash
+kubectl apply -f blackbox-ingress-probes.yaml
+```
+
+
+7. VERIFICA METRICHE IN PROMETHEUS
+==================================
+
+COMANDO - apri Prometheus in locale con port-forward
+```bash
+kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090
+```
+
+Poi apri nel browser:
+
+```text
+http://localhost:9090
+```
+
+QUERY PROMQL - tutte le probe
+```promql
+probe_success
+```
+
+QUERY PROMQL - probe servizi interni
+```promql
+probe_success{job="http-services-probe"}
+```
+
+QUERY PROMQL - status code HTTP
+```promql
+probe_http_status_code{job="http-services-probe"}
+```
+
+QUERY PROMQL - durata probe
+```promql
+probe_duration_seconds{job="http-services-probe"}
+```
+
+QUERY PROMQL - target falliti
+```promql
+probe_success{job="http-services-probe"} == 0
+```
+
+Interpretazione:
+- probe_success = 1: test riuscito.
+- probe_success = 0: test fallito.
+- probe_http_status_code: status HTTP ricevuto.
+- probe_duration_seconds: durata della probe.
+
+
+8. CREA ALERT PROMETHEUSRULE
+============================
+
+FILE - blackbox-http-alerts.yaml
+
+Descrizione:
+- genera alert quando un target HTTP non risponde correttamente;
+- genera alert quando un target risponde troppo lentamente.
+
+COMANDO - crea blackbox-http-alerts.yaml
+```bash
+cat > blackbox-http-alerts.yaml <<'EOF'
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: blackbox-http-alerts
+  namespace: monitoring
+  labels:
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: blackbox-http
+      rules:
+        - alert: BlackboxHttpProbeFailed
+          expr: probe_success == 0
+          for: 2m
+          labels:
+            severity: warning
+          annotations:
+            summary: "HTTP probe fallita"
+            description: "Il target {{ $labels.instance }} non risponde correttamente da almeno 2 minuti."
+
+        - alert: BlackboxHttpSlowResponse
+          expr: probe_duration_seconds > 2
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "HTTP probe lenta"
+            description: "Il target {{ $labels.instance }} risponde in piu' di 2 secondi da almeno 5 minuti."
+EOF
+```
+
+COMANDO - applica alert
+```bash
+kubectl apply -f blackbox-http-alerts.yaml
+```
+
+COMANDO - verifica PrometheusRule
+```bash
+kubectl get prometheusrule -n monitoring | grep blackbox
+kubectl describe prometheusrule blackbox-http-alerts -n monitoring
+```
+
+
+9. TEST MANUALE BLACKBOX EXPORTER
+=================================
+
+COMANDO - port-forward blackbox-exporter
+```bash
+kubectl port-forward -n monitoring svc/blackbox-exporter 9115:9115
+```
+
+COMANDO - esegui probe manuale da un altro terminale
+```bash
+curl "http://localhost:9115/probe?target=http://my-service.my-namespace.svc.cluster.local:8080/health&module=http_k8s_health"
+```
+
+OUTPUT ATTESO - metriche indicative
+```text
+probe_success 1
+probe_http_status_code 200
+```
+
+
+10. TROUBLESHOOTING
+===================
+
+CASO - Prometheus non vede la Probe
+
+COMANDO
+```bash
+kubectl get probe -A
+kubectl describe probe -n monitoring http-services-probe
+kubectl get prometheus -n monitoring -o yaml
+```
+
+Controlla:
+- spec.probeSelector
+- spec.probeNamespaceSelector
+- metadata.labels della Probe
+
+CASO - blackbox-exporter non parte
+
+COMANDO
+```bash
+kubectl get pods -n monitoring | grep blackbox
+kubectl logs -n monitoring deploy/blackbox-exporter
+```
+
+CASO - servizio target non raggiungibile
+
+COMANDO
+```bash
+kubectl get svc -n my-namespace
+kubectl get endpoints -n my-namespace my-service
+kubectl describe svc -n my-namespace my-service
+```
+
+CASO - DNS Kubernetes non risolve
+
+COMANDO
+```bash
+kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- nslookup my-service.my-namespace.svc.cluster.local
+```
+
+CASO - endpoint HTTPS fallisce per certificati, redirect o header
+
+Azioni consigliate:
+- usare il modulo http_2xx per endpoint esterni generici;
+- verificare se l'endpoint richiede SNI, autenticazione, header custom o path diverso;
+- aggiungere un modulo dedicato in blackbox-values.yaml se serve una configurazione HTTP specifica.
+
+
+11. ORDINE DI ESECUZIONE CONSIGLIATO
+====================================
+
+COMANDO - sequenza completa
+```bash
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+
+helm upgrade --install blackbox-exporter prometheus-community/prometheus-blackbox-exporter \
+  --namespace monitoring \
+  --create-namespace \
+  -f blackbox-values.yaml
+
+kubectl apply -f blackbox-probes.yaml
+kubectl apply -f blackbox-ingress-probes.yaml
+kubectl apply -f blackbox-http-alerts.yaml
+```
+
+FILE CONSIGLIATI DA VERSIONARE
+```text
+blackbox-values.yaml
+blackbox-probes.yaml
+blackbox-ingress-probes.yaml
+blackbox-http-alerts.yaml
+```
+
+
+12. NOTE PER DEV, QA E PROD
+===========================
+
+Per separare gli ambienti e semplificare dashboard e alert, usa Probe distinte:
+
+```text
+dev-http-services-probe
+qa-http-services-probe
+prod-http-services-probe
+```
+
+QUERY PROMQL - ambiente dev
+```promql
+probe_success{job="dev-http-services-probe"}
+```
+
+QUERY PROMQL - target prod falliti
+```promql
+probe_success{job="prod-http-services-probe"} == 0
+```
+
+Per endpoint critici in produzione, valuta:
+- interval piu' basso, ad esempio 15s o 30s;
+- durata for degli alert tra 1m e 5m in base alla criticita';
+- dashboard Grafana con stato, status code e latenza per target.

pipeline/customize.sh

@@ -50,11 +50,15 @@ printf '\nenv=%s\n' "$ENV" >> "$TMP_VALUES_FILE"
 printf 'namespace=%s\n' "$NAMESPACE" >> "$TMP_VALUES_FILE"
 printf 'project=%s\n' "$PROJECT_NAME" >> "$TMP_VALUES_FILE"
 
-# Ricava endpoint se presente e genera endpoint-nodot sostituendo '.' con '-'
+# Ricava endpoint se presente e genera le variabili derivate per le sostituzioni YAML.
 ENDPOINT_VAL=$(grep -E '^[[:space:]]*endpoint[[:space:]]*=' "$TMP_VALUES_FILE" "$PROPERTIES_FILE" 2>/dev/null | tail -n 1 | sed -E 's/.*endpoint[[:space:]]*=[[:space:]]*//' | tr -d '\r\n[:space:]"' || true)
 if [ -n "$ENDPOINT_VAL" ]; then
   ENDPOINT_NODOT=$(echo "$ENDPOINT_VAL" | tr '.' '-')
+  ENDPOINT_NOSPACE=$(echo "$ENDPOINT_VAL" | tr -d '.')
+
   printf 'endpoint-nodot=%s\n' "$ENDPOINT_NODOT" >> "$TMP_VALUES_FILE"
+  printf 'endpoint-nospace=%s\n' "$ENDPOINT_NOSPACE" >> "$TMP_VALUES_FILE"
+
 fi
 
 # Trova tutti i file .yaml nella directory kubernetes e sottodirectory