Secrets Management with HashiCorp Vault
Deploy HashiCorp Vault to eliminate hardcoded credentials from IT/OT systems — covering dynamic secrets, PKI, and Vault policies aligned to ISA/IEC 62443 FR 1 and FR 4.
The Hardcoded Credential Problem
Every industrial environment has them: API keys in configuration files, database passwords in shell scripts, WireGuard PSKs in plaintext /etc/wireguard/ files committed to a Git repository. These are not edge cases — a 2024 survey of industrial sites found hardcoded credentials in over 60% of OT engineering workstations.
The attack path is straightforward: an attacker who compromises one server reads a hardcoded credential and gains access to everything that credential unlocks. In a flat OT environment, that might be every PLC, every historian, and the SCADA configuration database.
HashiCorp Vault eliminates static credentials entirely. Instead of storing a password in a file, services request a credential from Vault at runtime, use it for a short-lived session, and let it expire. An attacker who intercepts the credential finds it expired before they can use it.
Vault Architecture for IT/OT
Vault runs in the IT Enterprise Zone (L4/L5). It is not deployed inside the OT network — OT systems with static configuration cannot support dynamic credential requests. Instead, Vault manages credentials for the IT-facing components: build systems, historian sync services, jump server session credentials, and WireGuard key material.
IT Enterprise Zone
┌─────────────────────────────────────┐
│ [Vault Server] │
│ ├── PKI secrets engine │
│ │ └── Issues TLS certs for WG │
│ ├── KV secrets engine │
│ │ └── Stores WG PSKs, API keys │
│ ├── Database secrets engine │
│ │ └── Dynamic historian creds │
│ └── AWS/Cloud secrets engine │
│ └── Dynamic IAM credentials │
│ │
│ Authenticated by: │
│ - AppRole (services) │
│ - LDAP (human engineers) │
│ - Kubernetes (CI/CD pipelines) │
└─────────────────────────────────────┘
│ TLS (port 8200)
[OT DMZ jump server, historian sync, pipeline runners]
Step 1: Deploy Vault
For production, deploy Vault in HA mode with an encrypted backend (Consul or Integrated Storage). For this lab, we use single-node with the Integrated Storage (Raft) backend:
# Install Vault (adjust for your distro)
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install -y vault
Configure Vault server (/etc/vault.d/vault.hcl):
ui = true
storage "raft" {
path = "/opt/vault/data"
node_id = "vault-01"
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_cert_file = "/etc/vault.d/vault.crt"
tls_key_file = "/etc/vault.d/vault.key"
}
api_addr = "https://vault.it.internal:8200"
cluster_addr = "https://vault.it.internal:8201"
sudo systemctl start vault
sudo systemctl enable vault
# Initialize Vault (first time only) — save the unseal keys and root token securely
vault operator init -key-shares=5 -key-threshold=3
Store the unseal keys in a physical safe (SL 3 requirement: no digital-only storage of root credentials). Distribute them across at least 3 trusted individuals.
Step 2: Enable Secrets Engines
export VAULT_ADDR="https://vault.it.internal:8200"
export VAULT_TOKEN="<root-token-from-init>" # use only during setup; rotate immediately
# Key-Value store for static secrets (WireGuard PSKs, vendor API keys)
vault secrets enable -path=secret kv-v2
# Database engine for dynamic historian credentials
vault secrets enable database
# PKI engine for certificate management
vault secrets enable pki
vault secrets tune -max-lease-ttl=87600h pki # 10 years for root CA
Step 3: Store the WireGuard PSK
# Store the IT/OT tunnel PSK — never again lives in a plaintext file
vault kv put secret/it-ot/wireguard/psk \
psk="$(sudo cat /etc/wireguard/it-ot-psk.key)"
# Verify storage (value is displayed in Vault UI but access-logged)
vault kv get secret/it-ot/wireguard/psk
Now remove the plaintext PSK from disk:
sudo shred -u /etc/wireguard/it-ot-psk.key
# Update wg0.conf to retrieve PSK from Vault at startup via a PostUp script
Step 4: Dynamic Historian Credentials
The historian sync service should not use a static database password. Configure Vault’s database secrets engine to generate credentials with a 1-hour TTL:
# Configure the historian database connection
vault write database/config/ot-historian \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@10.100.0.30:5432/historian" \
allowed_roles="historian-sync" \
username="vault-admin" \
password="<one-time-setup-password>"
# Create the role that defines what credentials look like
vault write database/roles/historian-sync \
db_name=ot-historian \
creation_statements="CREATE ROLE '{{name}}' WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON historian.process_data TO '{{name}}';" \
default_ttl="1h" \
max_ttl="4h"
The historian sync service requests credentials at startup:
# Service requests credentials at runtime — no password in config files
CREDS=$(vault read -format=json database/creds/historian-sync)
DB_USER=$(echo "$CREDS" | jq -r '.data.username')
DB_PASS=$(echo "$CREDS" | jq -r '.data.password')
# Connect using the dynamic credential
psql "host=10.100.0.30 user=${DB_USER} password=${DB_PASS} dbname=historian"
Test that Vault-issued credentials actually expire:
# Request a credential with a 1-minute TTL (for testing only)
vault write -f -format=json database/creds/historian-sync | \
jq '{username: .data.username, password: .data.password, lease_duration: .lease_duration}'
# Try to connect with the credential — should succeed
psql "host=10.100.0.30 user=<username> password=<password> dbname=historian" -c "SELECT 1"
# Wait 65 seconds, then try again — should fail with authentication error
sleep 65
psql "host=10.100.0.30 user=<username> password=<password> dbname=historian" -c "SELECT 1"
# Expected: psql: error: connection to server failed: FATAL: password authentication failedIf the second connection succeeds, Vault’s database lease revocation is not working. Check the vault-agent logs and confirm the database plugin has REVOKE permissions on the historian database.
Step 5: Vault Policies (Least Privilege)
Every service or user that authenticates to Vault receives a policy defining exactly which paths they can read, write, or delete:
# /etc/vault.d/policies/historian-sync.hcl
# The historian sync service may only read dynamic database credentials
path "database/creds/historian-sync" {
capabilities = ["read"]
}
# It may also read the WireGuard PSK for its tunnel
path "secret/data/it-ot/wireguard/psk" {
capabilities = ["read"]
}
# Deny everything else (implicit deny in Vault)
vault policy write historian-sync /etc/vault.d/policies/historian-sync.hcl
The historian sync service authenticates using AppRole:
vault auth enable approle
vault write auth/approle/role/historian-sync \
token_policies="historian-sync" \
token_ttl=1h \
token_max_ttl=4h \
secret_id_ttl=24h
Vault tokens are bearer tokens — whoever holds one has all the permissions associated with it. If the historian sync service’s AppRole token is leaked (e.g., written to a log file by a misconfigured application), an attacker can use it to request database credentials until the token expires.
Mitigations:
- Set
token_ttlto the minimum required (1 hour for a service that runs continuously and renews) - Enable response wrapping for token delivery:
vault write -wrap-ttl=60s auth/approle/role/historian-sync/secret-iddelivers a single-use wrapping token that must be unwrapped within 60 seconds - Audit Vault access logs daily — every credential request is logged with the requesting identity, IP, and timestamp
A build pipeline currently stores AWS IAM access keys as environment variables in the CI/CD system. The keys have broad permissions because the pipeline was built quickly and nobody scoped them down. What Vault features address this, and what is the residual risk?
Reveal Answer & Explanation
Two Vault features apply:
1. AWS secrets engine with dynamic credentials: instead of storing a static IAM key, the pipeline authenticates to Vault and receives a dynamically generated IAM key pair with a 15-minute TTL. The key is scoped to exactly the S3 bucket and ECR repository the pipeline needs. After 15 minutes, the key is automatically revoked by Vault.
2. Vault policy: the pipeline’s AppRole policy allows only read on the AWS credentials path — it cannot read credentials for other services or write to Vault’s configuration.
Residual risk: if the pipeline’s Vault AppRole credentials are compromised (e.g., a malicious pull request exfiltrates the Role ID and Secret ID via CI logs), an attacker can request new AWS credentials from Vault. Mitigations: response-wrap the Secret ID, restrict the AppRole to the pipeline’s CI IP range, and set a short secret_id_ttl.
What’s Next
HashiCorp Vault is deployed and managing credentials dynamically. No more hardcoded passwords in config files.
In Lesson 2, we use Vault’s PKI engine to sign build artifacts — ensuring that only cryptographically verified software reaches the OT environment through the build pipeline.