Cannot download raft storage snapshot from UI

[denmaddog]

Describe the bug
When I login to Vault UI and go to Status/Raft storage --> Snapshots --> Download It hangs for some time and then it shows message in browser:
{"errors":["permission denied"]}

URL is:
https://vault-cluster.example.com:8200/ui/vault/storage/raft

To Reproduce
Steps to reproduce the behavior:

1. Login to Vaults UI and go to Status/Raft storage --> Snapshots --> Download

Expected behavior
It should offer snapshot to download.

Environment:

• Vault Server Version (retrieve with vault status):
  vault status
  Key Value

---

Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
Version 1.12.0
Build Date 2022-10-10T18:14:33Z
Storage Type raft
Cluster Name vault-cluster-xxxxxx
Cluster ID xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
HA Enabled true
HA Cluster https://vault-node1.example.com:8201
HA Mode active
Active Since 2022-10-19T16:55:47.797228758Z
Raft Committed Index 2003050
Raft Applied Index 2003050

• Vault CLI Version (retrieve with vault version):
  Vault v1.12.0 (558abfa), built 2022-10-10T18:14:33Z
• Server Operating System/Architecture:
  PRETTY_NAME="Debian GNU/Linux 10 (buster)"
  NAME="Debian GNU/Linux"
  VERSION_ID="10"
  VERSION="10 (buster)"
  VERSION_CODENAME=buster
  ID=debian
  HOME_URL="https://www.debian.org/"
  SUPPORT_URL="https://www.debian.org/support"
  BUG_REPORT_URL="https://bugs.debian.org/"
  Vault server configuration file(s):

root@vault-node1:~# cat /etc/vault.d/vault.hcl
storage "raft" {
  path    = "/raft-data/"
  node_id = "node1"

#  retry_join {
#    leader_api_addr = "https://vault-node1.example.com:8200"
#  }

#  retry_join {
#    leader_api_addr = "https://vault-node2.example.com:8200"
#  }

#  retry_join {
#    leader_api_addr = "https://vault-node3.example.com:8200"
#  }

#  retry_join {
#    leader_api_addr = "https://vault-node4.example.com:8200"
#  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
#  tls_disable = true
  tls_cert_file    = "/etc/vault.d/example.com.crt"
  tls_key_file     = "/etc/vault.d/example.com.key"
}

disable_mlock = true
api_addr = "https://vault-node1.example.com:8200"
cluster_addr = "https://vault-node1.example.com:8201"
ui = true
####################


root@vault-node2:~# cat /etc/vault.d/vault.hcl
storage "raft" {
  path    = "/raft-data/"
  node_id = "node2"

  retry_join {
    leader_api_addr = "https://vault-node1.example.com:8200"
  }

  retry_join {
    leader_api_addr = "https://vault-node3.example.com:8200"
  }

  retry_join {
    leader_api_addr = "https://vault-node4.example.com:8200"
  }

  retry_join {
    leader_api_addr = "https://vault-node5.example.com:8200"
  }
}

listener "tcp" {
  address = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
#  tls_disable = true
  tls_cert_file    = "/etc/vault.d/example.com.crt"
  tls_key_file     = "/etc/vault.d/example.com.key"
}

disable_mlock = true
api_addr = "https://vault-node2.example.com:8200"
cluster_addr = "https://vault-node2.example.com:8201"
ui = true

vault operator raft autopilot state
Healthy: true
Failure Tolerance: 2
Leader: node1
Voters:
node1
node2
node3
node4
node5
Servers:
node1
Name: node1
Address: vault-node1.example.com:8201
Status: leader
Node Status: alive
Healthy: true
Last Contact: 0s
Last Term: 48
Last Index: 2003100
Version: 1.12.0
Node Type: voter
node2
Name: node2
Address: vault-node2.example.com:8201
Status: voter
Node Status: alive
Healthy: true
Last Contact: 139.461806ms
Last Term: 48
Last Index: 2003100
Version: 1.12.0
Node Type: voter
node3
Name: node3
Address: vault-node3.example.com:8201
Status: voter
Node Status: alive
Healthy: true
Last Contact: 434.560895ms
Last Term: 48
Last Index: 2003100
Version: 1.12.0
Node Type: voter
node4
Name: node4
Address: vault-node4.example.com:8201
Status: voter
Node Status: alive
Healthy: true
Last Contact: 1.706376111s
Last Term: 48
Last Index: 2003100
Version: 1.12.0
Node Type: voter
node5
Name: node5
Address: vault-node5.example.com:8201
Status: voter
Node Status: alive
Healthy: true
Last Contact: 156.711955ms
Last Term: 48
Last Index: 2003100
Version: 1.12.0
Node Type: voter

Additional context
Manual creation of snapshot works:
vault operator raft snapshot save 19102022.snapshot

ls -lah
total 9.2M
drwxr-xr-x 2 root root 4.0K Oct 19 17:01 .
drwx------ 8 root root 4.0K Oct 19 17:01 ..
-rw------- 1 root root 9.2M Oct 19 17:01 19102022.snapshot

[hsimon-hashicorp]

Just to clarify, the manual creation of the snapshot with the vault operator command is with the same credentials, correct?

[denmaddog]

Yes,
using root token for authentication with both vault cli and UI.

[hsimon-hashicorp]

Thank you! I'll have our engineering teams take a look. :)

[denmaddog]

np, thank you! :)

[atabacar]

I am having the same issue. I had Vault 1.8.8 and the download snapshot from UI was working fine. Then I upgraded to 1.11.3 and that functionality broke. In version 1.11.3 I tried both being logged as my usr and as root user. Same behaviour. I can take the snapshot using the cli but from UI it hangs and eventually I get permission denied message in the browser.

[denmaddog]

Same with v1.12.1.

[hashishaw]

This was fixed in PR #17769 -- will be released with 1.11.6 and 1.12.2. Thanks all!