Un hotfix non approuvé en prod k8s arrête le pipeline

Objectif Bloquer une mise en production quand un pod exécute une version que le comité n'a pas approuvée, par un code de sortie qu'une CI comprend.

Le catalogue des versions approuvées est un CSV. Ce qui tourne vraiment est dans le cluster. lumnik lit l'image de chaque pod par l'API Kubernetes et la joint au catalogue:

la version approuvée passe;
un hotfix poussé avec kubectl set image fait renvoyer le code 1 à la porte: la CI arrête le pipeline;
après le rollback, la même commande repasse.

lumnik Open
DevOps
Kubernetes
Livraison
Connector
Entity

manifests/lab-10-shop.yaml

# Three services, each image tagged with its version. run.sh builds the images (busybox, retagged) and loads them into kind.# The labels the connector reads: app (the service), environment, release-gate (what the reader lists).apiVersion: v1kind: Namespacemetadata: { name: shop }---apiVersion: v1kind: Namespacemetadata: { name: shop-staging }---apiVersion: apps/v1kind: Deploymentmetadata: { name: billing-api, namespace: shop }spec:  replicas: 1  selector: { matchLabels: { app: billing-api } }  template:    metadata: { labels: { app: billing-api, environment: production, release-gate: "true" } }    spec:      terminationGracePeriodSeconds: 1      containers:        - { name: api, image: "lumnik.local/billing-api:2.3.7", command: ["sleep", "infinity"] }---apiVersion: apps/v1kind: Deploymentmetadata: { name: catalog-api, namespace: shop }spec:  replicas: 1  selector: { matchLabels: { app: catalog-api } }  template:    metadata: { labels: { app: catalog-api, environment: production, release-gate: "true" } }    spec:      terminationGracePeriodSeconds: 1      containers:        - { name: api, image: "lumnik.local/catalog-api:5.8.1", command: ["sleep", "infinity"] }---# A release candidate in staging: not approved, and not production — the gate lets it through.apiVersion: apps/v1kind: Deploymentmetadata: { name: customer-portal, namespace: shop-staging }spec:  replicas: 1  selector: { matchLabels: { app: customer-portal } }  template:    metadata: { labels: { app: customer-portal, environment: staging, release-gate: "true" } }    spec:      terminationGracePeriodSeconds: 1      containers:        - { name: web, image: "lumnik.local/customer-portal:9.1.0-rc2", command: ["sleep", "infinity"] }

manifests/lab-30-reader.yaml

# What lumnik may do in the cluster: read pods, cluster-wide. Nothing else.apiVersion: v1kind: Namespacemetadata: { name: release-gate-reader }---apiVersion: v1kind: ServiceAccountmetadata: { name: release-gate-reader, namespace: release-gate-reader }---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata: { name: release-gate-reader }rules:  - apiGroups: [""]    resources: [pods]    verbs: [get, list]---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata: { name: release-gate-reader }roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: release-gate-reader }subjects: [{ kind: ServiceAccount, name: release-gate-reader, namespace: release-gate-reader }]---# A long-lived token for the hub's connector secret (projected tokens expire within the hour).apiVersion: v1kind: Secretmetadata:  name: release-gate-reader-token  namespace: release-gate-reader  annotations: { kubernetes.io/service-account.name: release-gate-reader }type: kubernetes.io/service-account-token

$ kubectl --context kind-lumnik-memory get pods -A -l release-gate -o "custom-columns=SERVICE:.metadata.labels.app,ENV:.metadata.labels.environment,IMAGE:.status.containerStatuses[0].image"
SERVICE           ENV          IMAGE
customer-portal   staging      lumnik.local/customer-portal:9.1.0-rc2
billing-api       production   lumnik.local/billing-api:2.3.7
catalog-api       production   lumnik.local/catalog-api:5.8.1

approved-versions.csv

service,owner,approved_version
billing-api,payments-on-call,2.3.7
catalog-api,commerce-on-call,5.8.1
customer-portal,web-on-call,9.0.4

manifests/10-approved-versions.yaml

# The catalogue of approved versions: a CSV the release board maintains, outside the cluster.apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: approved-versions  connector_type: csv-file  scopes: [devops-release-gate]spec:  transport:    kind: local    path: /tmp/lumnik-scenario/devops-release-gate/approved-versions.csv  parser:    has_header: true  after_process: none

manifests/20-pods.yaml

# The cluster's own API, read-only: the pods that run a service, and the image each one runs.# The token and the CA come from the release-gate-reader ServiceAccount (lab-30-reader.yaml): lm secret set.apiVersion: connectors.lumnik.io/v1kind: Connectormetadata:  name: k8s-pods  connector_type: rest-generic  scopes: [devops-release-gate]spec:  base_url: https://kubernetes.default.svc  auth: { kind: bearer, token_env: RELEASE_READER_TOKEN }  tls: { ca_secret: CLUSTER_CA }  defaults:    incremental: { mode: full }  endpoints:    - id: pods      method: GET      path: /api/v1/pods?labelSelector=release-gate%3Dtrue      pagination: { kind: cursor, cursor_param: continue, cursor_path: $.metadata.continue, response_path: $.items }      target:        kind: hybrid        table: connector.release_pods        unique_key: $.metadata.uid        promote:          - { name: pod_uid,     path: "$.metadata.uid",                     type: text }          - { name: service,     path: "$.metadata.labels.app",              type: text }          - { name: environment, path: "$.metadata.labels.environment",      type: text }          - { name: started_at,  path: "$.metadata.creationTimestamp",       type: timestamptz }          - { name: image,       path: "$.status.containerStatuses[0].image", type: text }          - { name: deployed_version, type: text }   # filled by image-version.kts      mapping:        cell:          - kotlin: { script: hooks:image-version.kts }

manifests/image-version.kts

// The version a pod really runs: its image's tag (lumnik.local/billing-api:2.3.7 → 2.3.7).@Suppress("UNCHECKED_CAST")val cells = bindings["cells"] as MutableMap<String, Any?>val image = cells["image"] as String?cells["deployed_version"] = image?.substringAfterLast('/')?.substringAfter(':', "")?.ifEmpty { null }

$ lm apply -f manifests/10-approved-versions.yaml
Applied connector "approved-versions" — 1 endpoint

$ lm apply -f manifests/20-pods.yaml
Applied connector "k8s-pods" — 1 endpoint
! 20-pods.yaml — 0 error(s), 1 warning(s)

  WARN     spec.endpoints[0].pagination.has_more_path
          no has_more_path: the cursor decides — the read stops on an empty cursor or an empty page
          → Declare has_more_path if the API sends a more-pages flag


$ lm connector run approved-versions
run scheduled: 1792aa2c-a9a5-46c3-ac4a-fe3186b0fb3b

$ lm run get 1792aa2c-a9a5-46c3-ac4a-fe3186b0fb3b
ID        CONNECTOR          ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
1792aa2c  approved-versions  default   Completed  3   3    0        0             2026-10-30 10:06  2026-10-30 10:06

$ lm connector run k8s-pods
run scheduled: f73a6843-4e26-44a2-962b-f5df3d0b2c14

$ lm run get f73a6843-4e26-44a2-962b-f5df3d0b2c14
ID        CONNECTOR  ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
f73a6843  k8s-pods   pods      Completed  3   3    0        0             2026-10-30 10:06  2026-10-30 10:06
• script hooks:image-version.kts  sha256 08ef0373ef0adfaed053c371d9b60f6af0e008d0f743401e497f7dd6bb71ea8f

manifests/30-deployment-compliance.yaml

kind: Entityname: DeploymentCompliancedescription: "Each pod a service ran, the version of its image, and the version the release board approved for that service."match: [pod_uid]scopes: [devops-release-gate]sources:  - table: release_pods    map:      pod_uid: pod_uid      service: service      environment: environment      deployed_version: deployed_version      started_at: started_atjoins:  - table: t_approved_versions    from: service    to: service    map:      owner: owner      approved_version: approved_version

$ lm entity apply -f manifests/30-deployment-compliance.yaml
Applied entity "DeploymentCompliance" → connector."v_deploymentcompliance"

$ lm entity get DeploymentCompliance --limit 1000 -o json
[
  {
    "pod_uid": "32a5ad62-c877-4128-bf90-9c272f3c4465",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.3.7",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "5873791e-7115-4a78-a312-d67e048dc44e",
    "service": "catalog-api",
    "environment": "production",
    "deployed_version": "5.8.1",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "commerce-on-call",
    "approved_version": "5.8.1",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "b0310305-67e3-4a00-bd69-9af4e84b6d86",
    "service": "customer-portal",
    "environment": "staging",
    "deployed_version": "9.1.0-rc2",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "web-on-call",
    "approved_version": "9.0.4",
    "_conflicts": "{}"
  }
]

$ run/release-gate.sh
[]

$ kubectl --context kind-lumnik-memory -n shop set image deploy/billing-api api=lumnik.local/billing-api:2.4.0-hotfix
deployment.apps/billing-api image updated

$ lm connector run k8s-pods
run scheduled: 7e0558c7-aec1-4865-9a9d-f1243040d98a

$ lm run get 7e0558c7-aec1-4865-9a9d-f1243040d98a
ID        CONNECTOR  ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
7e0558c7  k8s-pods   pods      Completed  3   3    0        0             2026-10-30 10:06  2026-10-30 10:06
• script hooks:image-version.kts  sha256 08ef0373ef0adfaed053c371d9b60f6af0e008d0f743401e497f7dd6bb71ea8f

$ lm entity get DeploymentCompliance --limit 1000 -o json
[
  {
    "pod_uid": "1b86c1e6-3538-47ad-b375-38995784ca70",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.4.0-hotfix",
    "started_at": "2026-10-30T09:06:13Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "32a5ad62-c877-4128-bf90-9c272f3c4465",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.3.7",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "5873791e-7115-4a78-a312-d67e048dc44e",
    "service": "catalog-api",
    "environment": "production",
    "deployed_version": "5.8.1",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "commerce-on-call",
    "approved_version": "5.8.1",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "b0310305-67e3-4a00-bd69-9af4e84b6d86",
    "service": "customer-portal",
    "environment": "staging",
    "deployed_version": "9.1.0-rc2",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "web-on-call",
    "approved_version": "9.0.4",
    "_conflicts": "{}"
  }
]

$ run/release-gate.sh
[
  {
    "service": "billing-api",
    "owner": "payments-on-call",
    "deployed_version": "2.4.0-hotfix",
    "approved_version": "2.3.7"
  }
]

$ kubectl --context kind-lumnik-memory -n shop rollout undo deploy/billing-api
deployment.apps/billing-api rolled back
Warning: resource deployments/billing-api was previously managed with 'kubectl apply'. Rolling back will not update the kubectl.kubernetes.io/last-applied-configuration annotation, which may cause unexpected behavior on future 'kubectl apply' operations. Consider using 'kubectl apply' with your previous configuration file instead.

$ lm connector run k8s-pods
run scheduled: 0f27922d-038c-41df-91ec-5469cde29f9c

$ lm run get 0f27922d-038c-41df-91ec-5469cde29f9c
ID        CONNECTOR  ENDPOINT  STATUS     IN  OUT  SKIPPED  DEAD-LETTERS  STARTED           ENDED
0f27922d  k8s-pods   pods      Completed  3   3    0        0             2026-10-30 10:06  2026-10-30 10:06
• script hooks:image-version.kts  sha256 08ef0373ef0adfaed053c371d9b60f6af0e008d0f743401e497f7dd6bb71ea8f

$ lm entity get DeploymentCompliance --limit 1000 -o json
[
  {
    "pod_uid": "1b86c1e6-3538-47ad-b375-38995784ca70",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.4.0-hotfix",
    "started_at": "2026-10-30T09:06:13Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "32a5ad62-c877-4128-bf90-9c272f3c4465",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.3.7",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "5873791e-7115-4a78-a312-d67e048dc44e",
    "service": "catalog-api",
    "environment": "production",
    "deployed_version": "5.8.1",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "commerce-on-call",
    "approved_version": "5.8.1",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "b0310305-67e3-4a00-bd69-9af4e84b6d86",
    "service": "customer-portal",
    "environment": "staging",
    "deployed_version": "9.1.0-rc2",
    "started_at": "2026-10-30T09:06:09Z",
    "owner": "web-on-call",
    "approved_version": "9.0.4",
    "_conflicts": "{}"
  },
  {
    "pod_uid": "daa99f47-0867-4fca-a2a8-cc78522350a1",
    "service": "billing-api",
    "environment": "production",
    "deployed_version": "2.3.7",
    "started_at": "2026-10-30T09:06:18Z",
    "owner": "payments-on-call",
    "approved_version": "2.3.7",
    "_conflicts": "{}"
  }
]

$ run/release-gate.sh
[]

service=billing-api: owner=payments-on-call deployed=2.3.7 approved=2.3.7
service=catalog-api: owner=commerce-on-call deployed=5.8.1 approved=5.8.1

preuves: replay.log, approved-statuses.txt, approved-violations.json, hotfix-statuses.txt, hotfix-violations.json, rollback-statuses.txt, rollback-violations.json, rollback-entity.json, summary.env · transform v1

Limites

La porte voit ce que le hub a lu: le pipeline lance la lecture du cluster juste avant la porte.
La version est le tag de l'image. Une image poussée sous un tag déjà approuvé n'est pas détectée: comparer le digest demanderait une autre colonne.
Un conteneur par pod est lu (le premier).
Cluster kind de démonstration, images construites pour le rejeu.

DevOps · mise en production

Un hotfix part en production sans validation.

Le pipeline doit le voir.

Cluster kind de démonstration

Le problème

Deux vérités, deux endroits.

Les versions approuvées: un fichier du comité.
Les versions qui tournent: le cluster.
Aucun outil ne compare les deux avant de livrer.

Ce qui n'est pas modifié

lumnik lit, il ne touche à rien.

Un compte de service qui ne peut que lister les pods.
Le CSV reste où il est.

Ce qu'il a fallu écrire

path: /api/v1/pods?labelSelector=…
joins:
  - table: t_approved_versions

Une lecture, une jointure.

4 fichiers, 72 lignes, et une porte shell de 18 lignes.

Le rouge

billing-api: 2.4.0-hotfix, approuvée 2.3.7.

Le pipeline s'arrête.
La violation nomme le service et son astreinte.

Le vert

Rollback: la même commande repasse.

Le hub garde chaque pod vu.
La porte lit le plus récent.

La limite

La porte voit ce que le hub a lu.

Un hotfix posé après la dernière lecture n'est pas encore vu.
Le pipeline lance la lecture juste avant la porte.

Essayer

Le scénario se rejoue sur kind.

Deux connecteurs, un script, une entité, une porte jq: chaque étape est publiée avec sa sortie.

Ce scénario tourne sur lumnik Open, gratuit et open source.

Essayer lumnik Open →