Skip to content

Secret Delivery Pipeline

Overview

Secret Delivery Pipeline follows the path from GCP Secret Manager to a live Kubernetes secret while keeping secret-value authority separate from the authentication mechanism used to read it. GCP Secret Manager owns the values, ESO performs the reconciliation, and applications consume materialised Kubernetes Secrets.

On-prem RKE2 authenticates ESO through a one-time bootstrap credential. GKE uses Workload Identity. Both surfaces converge on the same ClusterSecretStore and ExternalSecret delivery contract.

Case study

  • Context: application secrets were spread across environment files and manual operator steps. There was no traceable path from GCP Secret Manager to the application namespace, and credentials were embedded in GitOps state in some services.
  • Challenge: on-prem clusters needed a controlled path to cloud-held secrets while keeping the bootstrap credential out of Terraform state and Git history. GKE needed the same delivery contract without carrying that static credential model into the cloud cluster.
  • Approach: org/gcp/gsm-eso-sa provisions the reader identity. hyops init gcp --with-eso-sa generates the on-prem bootstrap key once and writes it to the local encrypted bootstrap vault. platform/k8s/gsm-bootstrap hands that credential into the on-prem cluster. GKE uses Workload Identity. In both environments, platform/k8s/gcp-secret-store configures ESO and applications declare ExternalSecret resources against the same store contract.
  • Outcome: the recorded exercise confirmed the full GCP Secret Manager → ESO → Kubernetes Secret delivery cycle on the on-prem RKE2 and GKE environments. The GKE path completed delivery and cleanup, while the on-prem bootstrap credential remained outside Terraform state and Git history.

Covers reader identity creation, on-prem key generation through hyops init, bootstrap handoff, ClusterSecretStore configuration, ExternalSecret reconciliation, and the verified secret landing in the application namespace.

Outcome

The result is one secret-delivery contract across on-prem and GKE with environment-specific authentication.

  • GCP Secret Manager holds the authoritative application secret values.
  • The ESO reader identity defines the permitted read scope.
  • On-prem RKE2 uses a one-time bootstrap credential handed through the encrypted bootstrap vault.
  • GKE uses Workload Identity for ESO authentication.
  • ExternalSecret reconciliation materialises the Kubernetes Secret consumed by the application.

Operating model

  1. org/gcp/gsm-eso-sa creates the GCP reader identity with Secret Manager read access. No service-account key is created by Terraform.
  2. For on-prem RKE2, hyops init gcp --with-eso-sa creates the key once and writes it to the operator's local encrypted bootstrap vault as HYOPS_GSM_SA_KEY_JSON.
  3. platform/k8s/gsm-bootstrap reads the bootstrap value and creates the cluster-side credential used by the on-prem store.
  4. On GKE, ESO authenticates through Workload Identity instead of a static key.
  5. platform/k8s/gcp-secret-store creates the ClusterSecretStore for the selected environment.
  6. Applications declare an ExternalSecret; the ESO controller reconciles it against GCP Secret Manager and materialises a Kubernetes Secret in the application namespace.

Architecture

Secret delivery architecture showing GCP Secret Manager as secret-value authority, the ESO reader identity, on-prem bootstrap and GKE Workload Identity authentication variants, ClusterSecretStore configuration, the ESO controller, ExternalSecret reconciliation, and final Kubernetes Secret delivery.

GCP Secret Manager remains authoritative for secret values. Authentication to that authority differs by environment: on-prem RKE2 uses a one-time bootstrap credential handed through the encrypted bootstrap vault, while GKE uses Workload Identity. Both surfaces configure the same ClusterSecretStore contract, which the ESO controller uses to reconcile ExternalSecret declarations into Kubernetes Secrets.

Platform state

Authority and on-prem secret-delivery proof: ESO reader authority is present, ClusterSecretStore is ready, ExternalSecret is synced, and the materialized secret exists without exposing values
Reader authority, ready ClusterSecretStore, synced ExternalSecret, and materialised secret without exposed values.

IP addresses, hostnames, and instance identifiers visible in screenshots and recordings reflect the ephemeral infrastructure provisioned during the recorded exercise.

Each stage is backed by a recorded run record. The captured exercise included a full GCP Secret Manager → ExternalSecret → Kubernetes Secret round trip on both delivery surfaces.

Stage Module Run ID Status
SA provisioning org/gcp/gsm-eso-sa apply-20260317T193309Z-465b4e8e ok
Bootstrap platform/k8s/gsm-bootstrap apply-20260313T230910Z-739e848c ok
GKE secret store platform/k8s/gcp-secret-store#gke_burst_secret_store apply-20260318T120046Z-bc73643d ok

The recorded GKE exercise completed a full GCP Secret Manager → ExternalSecret → Kubernetes Secret round trip and cleaned up the validation objects afterwards. Representative execution records are retained as part of the platform log.

Implementation

Surface Authentication Delivery contract
On-prem RKE2 Bootstrap SA key handed through the encrypted vault ClusterSecretStore → ExternalSecret → Kubernetes Secret
GKE burst Workload Identity ClusterSecretStore → ExternalSecret → Kubernetes Secret

Key components

  • Reader identity provisioning: org/gcp/gsm-eso-sa
  • On-prem key generation: hyops init gcp --with-eso-sa
  • On-prem bootstrap: platform/k8s/gsm-bootstrap
  • Secret-store contract: platform/k8s/gcp-secret-store
  • GKE variant: platform/k8s/gcp-secret-store#gke_burst_secret_store
  • ESO: deployed via the workload GitOps root alongside the cluster

Where it fits

  • application secrets with a single authoritative source in GCP Secret Manager
  • on-prem clusters that need a controlled path to cloud-held secrets
  • GKE clusters using Workload Identity with the same ESO delivery contract
  • platform services sharing a ClusterSecretStore across application namespaces
  • teams moving secrets out of environment files and into a reviewable delivery path

References

Further reading
Implementation references
  • org/gcp/gsm-eso-sa
  • platform/k8s/gsm-bootstrap
  • platform/k8s/gcp-secret-store
  • platform/k8s/gcp-secret-store#gke_burst_secret_store

What was verified

Verified during the recorded HybridOps v1.0.1 secret-delivery exercise. Reader identity provisioning was confirmed, the on-prem bootstrap path was validated, and ClusterSecretStore delivery was exercised on both the on-prem RKE2 and GKE surfaces.