[ kube-prometheus-stack ] Alertmanager with error 'field smtp_auth_password_file not found in type config.plain'

[gciria]

Describe the bug a clear and concise description of what the bug is.

In version 0.25.0 Alertmanager made available auth_password_file, to use the local or global SMTP password in file.
When using this parameter in the kube-prometheus-stack chart, alertmanager.config, Operator has the following error:

level=info ts=2023-01-03T12:08:01.585501494Z caller=operator.go:638 component=alertmanageroperator key=monitoring/kube-prometheus-stack-alertmanager msg="sync alertmanager"
level=info ts=2023-01-03T12:08:01.586027713Z caller=operator.go:1162 component=prometheusoperator key=monitoring/kube-prometheus-stack-prometheus msg="sync prometheus"
level=error ts=2023-01-03T12:08:01.591665869Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="sync \"monitoring/kube-prometheus-stack-alertmanager\" failed: provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors:\n  line 67: field auth_password_file not found in type config.plain"
level=info ts=2023-01-03T12:08:02.3353071Z caller=operator.go:1326 component=prometheusoperator key=monitoring/kube-prometheus-stack-prometheus msg="update prometheus status"

The kube-prometheus-stack v43.2.1 chart is using Alertmanager v0.25.0. Apparently the problem is in the CRDs that cannot verify the new parameter.

What's your helm version?

version.BuildInfo{Version:"v3.10.2", GitCommit:"50f003e5ee8704ec937a756c646870227d7c8b58", GitTreeState:"clean", GoVersion:"go1.18.8"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.7

Which chart?

kube-prometheus-stack

What's the chart version?

43.2.1

What happened?

New Alertmanager parameters smtp_auth_password_file and auth_password_file, used in values.alertmanager.config, are not being accepted by Prometheus Operator.

What you expected to happen?

New parameters need to be accepted by the Prometheus Operator CRD.

How to reproduce it?

Add a secret in Kubernetes:

---
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-config
  namespace: monitoring
type: Opaque
data:
  smtp_pass: aBcDe==

In values.yaml add:

alertmanager:
  config:
    global:
      smtp_hello: "host.example.org"
      smtp_smarthost: 'localhost:25'
      smtp_from: 'alertmanager@example.org'
      smtp_auth_username: 'globaluser'
      smtp_auth_password_file: '/etc/alertmanager/secrets/alertmanager-config/smtp_pass'
      smtp_hello: "host.example.org"

  alertmanagerSpec:
    secrets:
      - alertmanager-config

Update the chart.

Enter the changed values of values.yaml?

alertmanager:
  config:
    global:
      smtp_hello: "host.example.org"
      smtp_smarthost: 'localhost:25'
      smtp_from: 'alertmanager@example.org'
      smtp_auth_username: 'globaluser'
      smtp_auth_password_file: '/etc/alertmanager/secrets/alertmanager-config/smtp_pass'
      smtp_hello: "host.example.org"

  alertmanagerSpec:
    secrets:
      - alertmanager-config

Enter the command that you execute and failing/misfunctioning.

$ helm upgrade kube-prometheus-stack -f kube-prometheus-stack/values.yaml prometheus-community/kube-prometheus-stack --version 43.2.1 --namespace monitoring --timeout=1200s

Anything else we need to know?

No response

[joywaves]

thank you @gciria, same issue for me... to chime in: on initial setup, it won't deploy alertmanager pod if the setting is set to smtp_auth_password_file

[zeritti]

Support for smtp_auth_password_file and auth_password_file has been added in Prometheus operator through #5254 being included in the upcoming release v0.62.0

[gciria]

I tested smtp_auth_password_file and auth_password_file with Prometheus operator v0.62.0 and everything is working.
Thanks!