Secrets Management with HashiCorp Vault
Module 3 · IT Tier Security — Vault, Pipelines, and ACLs

Secrets Management with HashiCorp Vault

Deploy HashiCorp Vault to eliminate hardcoded credentials from IT/OT systems — covering dynamic secrets, PKI, and Vault policies aligned to ISA/IEC 62443 FR 1 and FR 4.

The Hardcoded Credential Problem

Every industrial environment has them: API keys in configuration files, database passwords in shell scripts, WireGuard PSKs in plaintext /etc/wireguard/ files committed to a Git repository. These are not edge cases — a 2024 survey of industrial sites found hardcoded credentials in over 60% of OT engineering workstations.

The attack path is straightforward: an attacker who compromises one server reads a hardcoded credential and gains access to everything that credential unlocks. In a flat OT environment, that might be every PLC, every historian, and the SCADA configuration database.

HashiCorp Vault eliminates static credentials entirely. Instead of storing a password in a file, services request a credential from Vault at runtime, use it for a short-lived session, and let it expire. An attacker who intercepts the credential finds it expired before they can use it.


Vault Architecture for IT/OT

Vault runs in the IT Enterprise Zone (L4/L5). It is not deployed inside the OT network — OT systems with static configuration cannot support dynamic credential requests. Instead, Vault manages credentials for the IT-facing components: build systems, historian sync services, jump server session credentials, and WireGuard key material.

IT Enterprise Zone
┌─────────────────────────────────────┐
│  [Vault Server]                     │
│   ├── PKI secrets engine            │
│   │    └── Issues TLS certs for WG  │
│   ├── KV secrets engine             │
│   │    └── Stores WG PSKs, API keys │
│   ├── Database secrets engine       │
│   │    └── Dynamic historian creds  │
│   └── AWS/Cloud secrets engine      │
│        └── Dynamic IAM credentials  │
│                                     │
│  Authenticated by:                  │
│   - AppRole (services)              │
│   - LDAP (human engineers)          │
│   - Kubernetes (CI/CD pipelines)    │
└─────────────────────────────────────┘
         │ TLS (port 8200)
[OT DMZ jump server, historian sync, pipeline runners]

Step 1: Deploy Vault

For production, deploy Vault in HA mode with an encrypted backend (Consul or Integrated Storage). For this lab, we use single-node with the Integrated Storage (Raft) backend:

# Install Vault (adjust for your distro)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
  sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y vault

Configure Vault server (/etc/vault.d/vault.hcl):

ui = true

storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-01"
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault.d/vault.crt"
  tls_key_file  = "/etc/vault.d/vault.key"
}

api_addr     = "https://vault.it.internal:8200"
cluster_addr = "https://vault.it.internal:8201"
sudo systemctl start vault
sudo systemctl enable vault

# Initialize Vault (first time only) — save the unseal keys and root token securely
vault operator init -key-shares=5 -key-threshold=3

Store the unseal keys in a physical safe (SL 3 requirement: no digital-only storage of root credentials). Distribute them across at least 3 trusted individuals.


Step 2: Enable Secrets Engines

export VAULT_ADDR="https://vault.it.internal:8200"
export VAULT_TOKEN="<root-token-from-init>"  # use only during setup; rotate immediately

# Key-Value store for static secrets (WireGuard PSKs, vendor API keys)
vault secrets enable -path=secret kv-v2

# Database engine for dynamic historian credentials
vault secrets enable database

# PKI engine for certificate management
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki  # 10 years for root CA

Step 3: Store the WireGuard PSK

# Store the IT/OT tunnel PSK — never again lives in a plaintext file
vault kv put secret/it-ot/wireguard/psk \
  psk="$(sudo cat /etc/wireguard/it-ot-psk.key)"

# Verify storage (value is displayed in Vault UI but access-logged)
vault kv get secret/it-ot/wireguard/psk

Now remove the plaintext PSK from disk:

sudo shred -u /etc/wireguard/it-ot-psk.key
# Update wg0.conf to retrieve PSK from Vault at startup via a PostUp script

Step 4: Dynamic Historian Credentials

The historian sync service should not use a static database password. Configure Vault’s database secrets engine to generate credentials with a 1-hour TTL:

# Configure the historian database connection
vault write database/config/ot-historian \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@10.100.0.30:5432/historian" \
  allowed_roles="historian-sync" \
  username="vault-admin" \
  password="<one-time-setup-password>"

# Create the role that defines what credentials look like
vault write database/roles/historian-sync \
  db_name=ot-historian \
  creation_statements="CREATE ROLE '{{name}}' WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
    GRANT SELECT ON historian.process_data TO '{{name}}';" \
  default_ttl="1h" \
  max_ttl="4h"

The historian sync service requests credentials at startup:

# Service requests credentials at runtime — no password in config files
CREDS=$(vault read -format=json database/creds/historian-sync)
DB_USER=$(echo "$CREDS" | jq -r '.data.username')
DB_PASS=$(echo "$CREDS" | jq -r '.data.password')
# Connect using the dynamic credential
psql "host=10.100.0.30 user=${DB_USER} password=${DB_PASS} dbname=historian"
💻 Code Lab: Verify Credential Expiry

Test that Vault-issued credentials actually expire:

# Request a credential with a 1-minute TTL (for testing only)
vault write -f -format=json database/creds/historian-sync | \
  jq '{username: .data.username, password: .data.password, lease_duration: .lease_duration}'

# Try to connect with the credential — should succeed
psql "host=10.100.0.30 user=<username> password=<password> dbname=historian" -c "SELECT 1"

# Wait 65 seconds, then try again — should fail with authentication error
sleep 65
psql "host=10.100.0.30 user=<username> password=<password> dbname=historian" -c "SELECT 1"
# Expected: psql: error: connection to server failed: FATAL: password authentication failed

If the second connection succeeds, Vault’s database lease revocation is not working. Check the vault-agent logs and confirm the database plugin has REVOKE permissions on the historian database.


Step 5: Vault Policies (Least Privilege)

Every service or user that authenticates to Vault receives a policy defining exactly which paths they can read, write, or delete:

# /etc/vault.d/policies/historian-sync.hcl
# The historian sync service may only read dynamic database credentials
path "database/creds/historian-sync" {
  capabilities = ["read"]
}

# It may also read the WireGuard PSK for its tunnel
path "secret/data/it-ot/wireguard/psk" {
  capabilities = ["read"]
}

# Deny everything else (implicit deny in Vault)
vault policy write historian-sync /etc/vault.d/policies/historian-sync.hcl

The historian sync service authenticates using AppRole:

vault auth enable approle
vault write auth/approle/role/historian-sync \
  token_policies="historian-sync" \
  token_ttl=1h \
  token_max_ttl=4h \
  secret_id_ttl=24h

⚠️ Common Pitfall: Protect the Vault Token as Carefully as the Root Credential

Vault tokens are bearer tokens — whoever holds one has all the permissions associated with it. If the historian sync service’s AppRole token is leaked (e.g., written to a log file by a misconfigured application), an attacker can use it to request database credentials until the token expires.

Mitigations:

  • Set token_ttl to the minimum required (1 hour for a service that runs continuously and renews)
  • Enable response wrapping for token delivery: vault write -wrap-ttl=60s auth/approle/role/historian-sync/secret-id delivers a single-use wrapping token that must be unwrapped within 60 seconds
  • Audit Vault access logs daily — every credential request is logged with the requesting identity, IP, and timestamp

🧠 Knowledge Check

A build pipeline currently stores AWS IAM access keys as environment variables in the CI/CD system. The keys have broad permissions because the pipeline was built quickly and nobody scoped them down. What Vault features address this, and what is the residual risk?

Reveal Answer & Explanation

Two Vault features apply:

1. AWS secrets engine with dynamic credentials: instead of storing a static IAM key, the pipeline authenticates to Vault and receives a dynamically generated IAM key pair with a 15-minute TTL. The key is scoped to exactly the S3 bucket and ECR repository the pipeline needs. After 15 minutes, the key is automatically revoked by Vault.

2. Vault policy: the pipeline’s AppRole policy allows only read on the AWS credentials path — it cannot read credentials for other services or write to Vault’s configuration.

Residual risk: if the pipeline’s Vault AppRole credentials are compromised (e.g., a malicious pull request exfiltrates the Role ID and Secret ID via CI logs), an attacker can request new AWS credentials from Vault. Mitigations: response-wrap the Secret ID, restrict the AppRole to the pipeline’s CI IP range, and set a short secret_id_ttl.


What’s Next

HashiCorp Vault is deployed and managing credentials dynamically. No more hardcoded passwords in config files.

In Lesson 2, we use Vault’s PKI engine to sign build artifacts — ensuring that only cryptographically verified software reaches the OT environment through the build pipeline.