Database root credential rotation
Vault's database secrets engine provides a centralized workflow for managing credentials for various database systems. By leveraging this, every service instance gets a unique set of database credentials instead of sharing one. Having those credentials tied directly to each service instance and live only for the life of the service, any abnormal access pattern can be mapped to a specific service instance and its credential can be revoked immediately.
This reduces the manual tasks performed by the database administrator and makes the database access more efficient and secure.
In the dynamic secrets tutorial, you configured Vault to generate dynamic credentials for a PostgreSQL database. In this tutorial, you will learn how to rotate credentials used by Vault, known as the root user.
If you are not familar with how to configure Vault for dynamic credentials, follow the database secrets engine tutorial before you begin.
Scenario
Because Vault is managing the database credentials on behalf of the database administrator, it must also be given a set of highly privileged credentials which can grant and revoke access to the database system.
HashiCups needs to ensure the credentials used by Vault are secure, and do not violate the governance, risk and compliance (GRC) policies surrounding that data stored in the database.
Oliver with the operations team will use Vault's /database/rotate-root/:name
API endpoint to rotate the root credentials stored for the database connection.
Best Practice
Use this feature to rotate the root credentials immediately after the initial configuration of each database.
Prerequisites
This lab was tested on macOS using an x86_64 based and Apple silicon-based processors. You may also run this tutorial by clicking the Start interactive lab button.
To perform the tasks described in this tutorial, you need to have:
- Docker to run a Vault and PostgreSQL container.
- Vault binary installed.
- Git installed.
- PostgreSQL CLI (
psql
) client installed.
Policy requirements
Note
For the purpose of this tutorial, you can use your root
token to
work with Vault. However, it is recommended that root tokens are only used for
just enough initial setup or in emergencies. As a best practice, use tokens
with appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:
# Mount secrets engines
path "sys/mounts/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
# Configure the database secrets engine and create roles
path "database/*" {
capabilities = [ "create", "read", "update", "delete", "list" ]
}
If you are not familiar with policies, complete the policies tutorial.
Set up the lab
Clone the
learn-vault-dynamic-credentials
repository.$ git clone git@github.com:hashicorp-education/learn-vault-dynamic-credentials.git
Change into the
learn-vault-dynamic-credentials
directory.$ cd learn-vault-dynamic-credentials
Deploy the Vault and PostgreSQL containers.
$ terraform -chdir=vault-dynamic-creds-docker/ init && \ terraform -chdir=vault-dynamic-creds-docker/ apply -auto-approve
Example output:
Initializing the backend... Initializing provider plugins... - Finding kreuzwerker/docker versions matching "3.0.2"... - Installing kreuzwerker/docker v3.0.2... ...snip... Apply complete! Resources: 4 added, 0 changed, 0 destroyed. Outputs: POSTGRES_URL = "export TF_VAR_POSTGRES_URL=172.17.0.2:5432" POSTGRES_URL_HOST = "export 127.0.0.1:5432" VAULT_ADDR = "export VAULT_ADDR=http://127.0.0.1:8200" VAULT_TOKEN = "export VAULT_TOKEN=root"
Copy the export command from the Terraform output and export the environment variables.
Example:
$ export VAULT_ADDR=http://127.0.0.1:8200 \ VAULT_TOKEN=root \ TF_VAR_POSTGRES_URL=172.17.0.2:5432
Verify the PostgreSQL and Vault containers have started.
$ docker ps -f name=learn --format "table {{.Names}}\t{{.Status}}" NAMES STATUS learn-postgres Up 4 minutes learn-vault Up 4 minutes
Vault and PostgreSQL are running. Vault connects to PostgreSQL over the Docker bridge network.
Apply the PostgreSQL configuration used in the dynamic secrets tutorial.
$ terraform -chdir=vault-dynamic-creds-postgres/ init && \ terraform -chdir=vault-dynamic-creds-postgres/ apply -auto-approve
Example output:
Initializing the backend... Initializing provider plugins... - Finding cyrilgdn/postgresql versions matching "1.25.0"... - Installing cyrilgdn/postgresql v1.25.0... - Installed cyrilgdn/postgresql v1.25.0 (self-signed, key ID 418F268A88A6D481) ...snip... Plan: 2 to add, 0 to change, 0 to destroy. postgresql_role.ro: Creating... postgresql_grant.readonly_tables: Creating... postgresql_role.ro: Creation complete after 0s [id=ro] postgresql_grant.readonly_tables: Creation complete after 0s [id=ro_postgres_public_table] Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Apply the Vault configuration used in the dynamic secrets tutorial.
$ terraform -chdir=vault-dynamic-creds-vault/ init && \ terraform -chdir=vault-dynamic-creds-vault/ apply -auto-approve
Example output:
Initializing the backend... Initializing provider plugins... - Finding hashicorp/vault versions matching "4.5.0"... ...snip... vault_database_secrets_mount.database: Creating... vault_database_secrets_mount.database: Creation complete after 0s [id=database] vault_database_secret_backend_role.readonly: Creating... vault_database_secret_backend_role.readonly: Creation complete after 0s [id=database/roles/readonly] Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Vault and PostgreSQL are running and configured. You are ready to proceed with the tutorial.
Rotate the root credentials
Vault allows you to rotate the initial root database credentials used when configuring the database secrets engine.
Tip
Once you rotate the root credential, only the Vault knows the new root password. This is the same for all root database credentials given to Vault. A good practice is to create a separate superuser dedicated to Vault which is not used for other purposes.
Rotate the root credentials for the PostgreSQL database connection.
$ vault write -force database/rotate-root/postgres
Verify that the root credential was rotated by attempting to log into PostgreSQL and enter the password
rootpassword
.Entering the initial password (
rootpassword
) fails since you rotated the password with therotate-root
command.$ psql -h localhost -p 5432 -U root postgres Password for user root:
Example output:
psql -h localhost -p 5432 -U root Password for user root: psql: error: connection to server at "localhost" (::1), port 5432 failed: FATAL: password authentication failed for user "root"
You can invoke the
database/rotate-root/:name
endpoint periodically to secure the root credential.
Verify the configuration
Request credentials from the readonly
role to verify that the database secrets engine dynamically
generates credentials as expected.
Read credentials from the
readonly
database role.$ CREDS=$(vault read -format=json database/creds/readonly) && echo $CREDS { "request_id": "c72c9777-b35a-a8e9-81e2-83c5f2de5e37", "lease_id": "database/creds/readonly/7Gj6xNxZBRgQo881rGRB7Ueu", "lease_duration": 3600, "renewable": true, "data": { "password": "Ek783iXA-iwzWnewVgON", "username": "v-token-readonly-4ibY2gRaYfYaW8qMXAE3-1737564539" }, "warnings": null, "mount_type": "database" }
The command outputs the results in JSON and stores the output in the
CREDS
variable.Verify you can connect to the database using the Vault generated credentials.
$ psql -h localhost -p 5432 \ -U $(echo $CREDS | jq -r .data.username) postgres
Enter the password generated by Vault when prompted.
Example output:
Password for user v-token-readonly-x7v65y1xuprzxv9vpt80-1525378873:
Verify that you can run commands with the PostgreSQL role provided.
$ \du Role name | Attributes -------------------------------------------------+------------------------------------------------------------ ro | No inheritance, Cannot login | Password valid until infinity root | Superuser, Create role, Create DB, Replication, Bypass RLS v-token-readonly-x6q809467q98yp4yx4z4-1525378026 | Password valid until 2018-05-03 21:07:11+00
Exit the PostgreSQL prompt.
$ \q
You successfully rotated the root user credentials used by Vault and used the Vault generated credentials to successfully connected to your PostgreSQL server.
Clean up
Delete the Terraform state files for the PostgreSQL configuration.
Note
This step is only required because the root user password was rotated during the tutorial. In a production environment, you would use a dedicated superuser for Vault, and a dedicated user for Terraform.
$ rm -rf vault-dynamic-creds-postgres/terraform.tfstate*
Destroy the remaining Terraform resources.
$ terraform -chdir=vault-dynamic-creds-vault/ destroy -auto-approve && \ terraform -chdir=vault-dynamic-creds-docker/ destroy -auto-approve
Unset the environment variables.
$ unset VAULT_ADDR VAULT_TOKEN TF_VAR_POSTGRES_URL
Summary
In this tutorial, you learned how to manage the privileged root user credentials
used by Vault to generate dynamic credentials.
After rotating the root credentials, you requested credentials from Vault by
reading the readonly
to verify that the database secrets engine dynamically
generates credentials as expected.