Credentials returned from Vault with lease_duration of 0 stay in cache forever

[kberzinch]

Summary

I use Concourse with HashiCorp Vault as a credential manager. Caching is enabled with CONCOURSE_SECRET_CACHE_ENABLED=true.

My Vault server is configured with several secrets engines, including the aws engine. This engine is configured to provide STS assumed role credentials good for 15 minutes.

I recently upgraded Vault from v1.12.3 to v1.13.0, which changed the behavior of this engine to not generate leases for STS credentials: hashicorp/vault#15869

With this change, the lease_duration returned from this request is set to 0, rather than the actual TTL of the credentials.

Concourse only ever retrieves the secret once and then continues using it until ATC is restarted. It's unclear if this is intentional.

Steps to reproduce

Download and start up Vault 1.13.0.

$ wget --quiet https://releases.hashicorp.com/vault/1.13.0/vault_1.13.0_linux_amd64.zip

$ unzip vault_1.13.0_linux_amd64.zip

$ ./vault server -dev -dev-listen-address=0.0.0.0:8200 -dev-root-token-id=roottoken

Configure Concourse to use the Vault server, and start it up.

$ wget --quiet https://concourse-ci.org/docker-compose.yml

$ cat << "EOF" >> docker-compose.yml

      CONCOURSE_VAULT_URL: http://<redacted>:8200
      CONCOURSE_VAULT_CLIENT_TOKEN: roottoken
      CONCOURSE_SECRET_CACHE_ENABLED: true
      CONCOURSE_SECRET_CACHE_DURATION: 10s
EOF

$ docker compose up

Configure Vault to provide AWS STS credentials.

$ export VAULT_ADDR='http://0.0.0.0:8200'

$ ./vault secrets enable -path=concourse/main/ aws

$ ./vault write concourse/main/config/root access_key=<redacted> secret_key=<redacted> region=us-east-1

$ ./vault write concourse/main/roles/my-role credential_type=assumed_role role_arns="<redacted>" max_sts_ttl=900

Log in to Concourse and run a task that uses the credentials.

# task.yml
---
platform: linux

image_resource:
  type: registry-image
  source:
    repository: amazon/aws-cli
    tag: latest

params:
  AWS_ACCESS_KEY_ID: ((sts/my-role.access_key))
  AWS_SECRET_ACCESS_KEY: ((sts/my-role.secret_key))
  AWS_SESSION_TOKEN: ((sts/my-role.security_token))

run:
  path: aws
  args:
  - sts
  - get-caller-identity
  - --region
  - us-east-1
  - --no-cli-pager

$ wget --quiet http://localhost:8080/api/v1/cli?arch=amd64&platform=linux

$ chmod +x fly

$ ./fly --target local login --username test --password test

$ watch --beep --interval 10 --color --errexit ./fly execute --target local --config task.yml

Expected results

New credentials are used for each build, as the watch interval is the same as the Concourse secret cache duration.

Actual results

The first set of credentials is used until ATC is restarted.

Additional context

This is the raw JSON output from Vault 1.12.4 for vault read concourse/main/sts/my-role:

{
    "request_id": "7f1fadfb-8019-5d01-3fd9-b13285c9322a",
    "lease_id": "concourse/main/sts/my-role/1jibITRoa4cBljro73DQTrlq",
    "renewable": false,
    "lease_duration": 900,
    "data":
    {
        "access_key": "redacted",
        "arn": "arn:aws:sts::redacted:assumed-role/redacted",
        "secret_key": "redacted",
        "security_token": "redacted"
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

This is the JSON output from Vault 1.13.0:

{
    "request_id": "9d57d735-249e-c059-0cbc-ea58a39ef44b",
    "lease_id": "",
    "renewable": false,
    "lease_duration": 0,
    "data":
    {
        "access_key": "redacted",
        "arn": "arn:aws:sts::redacted:assumed-role/redacted",
        "secret_key": "redacted",
        "security_token": "redacted",
        "ttl": 899
    },
    "wrap_info": null,
    "warnings": null,
    "auth": null
}

Triaging info

• Concourse version: 7.9.1
• Browser (if applicable): N/A
• Did this used to work? Vault v1.12.4 does return lease information, which is respected by Concourse.