Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to find a valid CSRF token with Keycloak Authentication Switching From Dex #2664

Open
zenarcher007 opened this issue Apr 1, 2024 · 2 comments
Open

Unable to find a valid CSRF token with Keycloak Authentication Switching From Dex #2664

zenarcher007 opened this issue Apr 1, 2024 · 2 comments
Projects
Needs Triage

Comments

@zenarcher007
Copy link

zenarcher007 commented Apr 1, 2024 •
edited

/kind question

Question:
I am trying to switch authentication completely from using Dex to using KeyCloak. After logging in through KeyCloak, I am redirected back to https://kubeflow.mydomain.org/oauth2/callback?state=[160_chars_omitted]&session_state=77a15975-fa55-496e-8769-d0e45ff624bf&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=[110_chars_omitted], where it says "Error while loading CSRF cookie: http: named cookie not present" in the oauth2-proxy pod logs.

I am aware that in the README of istio-external-auth, it states that oauth2-proxy looks for a cookie named "oauth2_proxy_kubeflow". I notice that a cookie called "oidc_state_csrf" is created in the browser after logging in, although the same behavior is observed even when renaming it to "oauth2_proxy_kubeflow" in the browser. At this point, I am not sure what is causing this issue.

Note: based on this issue, I have tried the --cookie-csrf-per-request=true and --cookie-csrf-expire=5m options, along with --cookie-secure=true (yaml form), none of which changed this.

From the oauth2-proxy pod log:

[2024/03/31 23:30:09] [oauthproxy.go:874] &{GET /oauth2/callback?state=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw&session_state=bd69094d-dd48-4a81-8bcb-abf27dbe2590&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0 HTTP/1.1 1 1 map[Content-Length:[0] Cookie:[oauth2_proxy_kubeflow=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw] X-Envoy-Expected-Rq-Timeout-Ms:[600000] X-Envoy-Internal:[true] X-Forwarded-For:[10.42.4.75]] {} <nil> 0 [] false kubeflow.mydomain.org map[code:[3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0] iss:[https://main.mydomain.org/auth/realms/master] session_state:[bd69094d-dd48-4a81-8bcb-abf27dbe2590] state:[MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw]] map[] <nil> map[] 10.42.4.75:59538 /oauth2/callback?state=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw&session_state=bd69094d-dd48-4a81-8bcb-abf27dbe2590&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0 <nil> <nil> <nil> 0xc0002aa720} AuthFailure Invalid authentication via OAuth2. Error while loading CSRF cookie: http: named cookie not present

My current setup:

  • KubeFlow is installed from kubeflow/manifests [03ddeb8]
  • System: Ubuntu 22.04.3 LTS, k3s version v1.28.4+k3s2
  • KeyCloak is preinstalled and accessible externally.
  • Traefik is encrypting http to https traffic for both KeyCloak and Kubeflow
  • DNS provider is CloudFlare

Configuration

File: manifests/common/oidc-client/oidc-authservice/base/params.env:

AUTHSERVICE_URL_PREFIX=/oauth2 # Changed from /authservice (oddly, this seemed to make it match similar to the redirect behavior of dex)
OIDC_PROVIDER=https://main.mydomain.org/auth/realms/master
OIDC_AUTH_URL=https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth
OIDC_SCOPES=acr address email microprofile-jwt offline_access phone profile roles web-origins
REDIRECT_URL=https://kubeflow.mydomain.org/oauth2/callback
SKIP_AUTH_URLS=/login
USERID_HEADER=kubeflow-userid
USERID_PREFIX=
USERID_CLAIM=email
PORT="8080"
STORE_PATH=/var/lib/authservice/data.db # this is probably not needed anymore

File: manifests/common/oidc-client/oidc-authservice/base/secret_params.env:

CLIENT_ID=kubeflow
CLIENT_SECRET=<my_keycloak_generated_secret>

File: manifests/common/oidc-client/oauth2-proxy/base/kustomization.yaml

  • Has "FORCE_HTTPS" set to true

File: manifests/common/oidc-client/oauth2-proxy/components/istio-external-auth/kustomization.yaml

  • Has cloudflare config set (authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml)

File: manifests/common/oidc-client/oauth2-proxy/base/oauth2-proxy-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: oauth2-proxy
  labels:
    app: oauth2-proxy
data:
  oauth2_proxy.cfg: |
    provider = "oidc"
    oidc_issuer_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth" # URL to auth UI
    scope = "acr address email microprofile-jwt offline_access phone profile roles web-origins"
    upstreams = "static://200"
    email_domains = [ "*" ]
    skip_auth_regex=["/login/.*"] # Don't launch authenticator for these paths
    skip_oidc_discovery = true
    login_url = "/login/oidc" # The url for launching the authenticator? *Although this is not mentioned in browser network requests!*
    redeem_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/token"
    oidc_jwks_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/keys" # This is configured on KeyCloak
    skip_provider_button = true
   
    cookie_name = "oauth2_proxy_kubeflow"
    cookie_expire = "24h"
    cookie_refresh = 0 #"5m"
    code_challenge_method = "S256"
    redirect_url = "https://main.mydomain.org/oauth2/callback" # This is what it tells KeyCloak to redirect back to
    relative_redirect_url = false #true
binaryData: {}

Network path from Firefox:

  • [GET, 302] /
  • [GET] https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth?client_id=kubeflow&redirect_uri=https%3A%2F%2Fkubeflow.knoxds.org%2Foauth2%2Fcallback&response_type=code&scope=openid+acr+address+email+microprofile-jwt+offline_access+phone+profile+roles+web-origins&state=MTcxMTk0NDY0N3xOd3dBTkVsWlJsZEVTMUJGUjFsRFJVcEpTVFpPV1V4U01sUkhTVTVPTmpkVVFVNDNURWd5VUZReldsZEJVRXRSUlVWUE5FOUlRa0U9fIFkV4ZOKtDawkCocUHXiQLGf3CIsCZmBVsx8-xj6JnA
  • [GET] Many KeyCloak UI asset requests...
  • [POST, 302] https://main.mydomain.org/auth/realms/master/login-actions/authenticate?session_code=HZUik4W9zB76cd2j28AvOFsXGuNGqLuiZj-r5QG0I4A&execution=97e7b76a-90a2-4de8-b2ae-edbbc0f8d21a&client_id=kubeflow&tab_id=34CDD_rjOpg
  • [GET, 403] https://kubeflow.mydomain.org/oauth2/callback?state=MTcxMTk0NDY0N3xOd3dBTkVsWlJsZEVTMUJGUjFsRFJVcEpTVFpPV1V4U01sUkhTVTVPTmpkVVFVNDNURWd5VUZReldsZEJVRXRSUlVWUE5FOUlRa0U9fIFkV4ZOKtDawkCocUHXiQLGf3CIsCZmBVsx8-xj6JnA&session_state=133b466c-6d49-4e1b-8886-c1f1630f2425&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=af416245-40f8-4a04-85ad-0408460e709b.133b466c-6d49-4e1b-8886-c1f1630f2425.4dd21bf3-bbf5-4734-8de5-46329c4dddf0
    • Request Cookies: "oauth2_proxy_kubeflow", "oidc_state_csrf"
  • [GET] bulma.min.css
  • [GET] all.min.css
  • [GET, 302] favicon.ico
  • [GET, none] https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth?client_id=kubeflow&redirect_uri=https%3A%2F%kubeflow.mydomain.org%2Foauth2%2Fcallback&response_type=code&scope=openid+acr+address+email+microprofile-jwt+offline_access+phone+profile+roles+web-origins&state=MTcxMTk0NDY1MnxOd3dBTkVrMldWWldXRlZUUmxoTFExUkRSbGd5V1VOSFJGaEpRVXBGTlZwYVQxRkdTMGxFV2swME16WlRUVTlUVUVoRE5WVXpRbEU9fLVBVjtueQY7Yf_akLheNmfMBWeme_2PZZdJDFTLq_xL (NS_ERROR_FAILURE)
@google-oss-prow Google OSS Prow
Copy link

@zenarcher007: The label(s) kind/issue cannot be applied, because the repository doesn't have them.

In response to this:

/kind issue

Question:
I am trying to switch authentication completely from using Dex to using KeyCloak. After logging in through KeyCloak, I am redirected back to https://kubeflow.mydomain.org/oauth2/callback?state=[160_chars_omitted]&session_state=77a15975-fa55-496e-8769-d0e45ff624bf&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=[110_chars_omitted], where it says "Error while loading CSRF cookie: http: named cookie not present" in the oauth2-proxy pod logs.

I am aware that in the README of istio-external-auth, it states that oauth2-proxy looks for a cookie named "oauth2_proxy_kubeflow". I notice that a cookie called "oidc_state_csrf" is created in the browser after logging in, although the same behavior is observed even when renaming it to "oauth2_proxy_kubeflow" in the browser. At this point, I am not sure what is causing this issue.

Note: based on this issue, I have tried the --cookie-csrf-per-request=true and --cookie-csrf-expire=5m options, along with --cookie-secure=true (yaml form), none of which changed this.

From the oauth2-proxy pod log:

[2024/03/31 23:30:09] [oauthproxy.go:874] &{GET /oauth2/callback?state=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw&session_state=bd69094d-dd48-4a81-8bcb-abf27dbe2590&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0 HTTP/1.1 1 1 map[Content-Length:[0] Cookie:[oauth2_proxy_kubeflow=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw] X-Envoy-Expected-Rq-Timeout-Ms:[600000] X-Envoy-Internal:[true] X-Forwarded-For:[10.42.4.75]] {} <nil> 0 [] false kubeflow.mydomain.org map[code:[3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0] iss:[https://main.mydomain.org/auth/realms/master] session_state:[bd69094d-dd48-4a81-8bcb-abf27dbe2590] state:[MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw]] map[] <nil> map[] 10.42.4.75:59538 /oauth2/callback?state=MTcxMTkyNzc2MHxOd3dBTkVwRFZWaFFObFJTU1VOTE1rZzFSRmRMVjFOSlNrRkdTbFZSUWt0TVdsazFORGRXUVZWWFUweFdSVU5ZTWxwVVFqWldSMUU9fHripA8UhgH7NolCwPZjGhEdhUYVgzGzJHQ22qaLEjUw&session_state=bd69094d-dd48-4a81-8bcb-abf27dbe2590&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=3eb04cda-9269-4be8-9713-9e7261cb904f.bd69094d-dd48-4a81-8bcb-abf27dbe2590.4dd21bf3-bbf5-4734-8de5-46329c4dddf0 <nil> <nil> <nil> 0xc0002aa720} AuthFailure Invalid authentication via OAuth2. Error while loading CSRF cookie: http: named cookie not present

My current setup:

  • KubeFlow is installed from kubeflow/manifests [03ddeb8]
  • System: Ubuntu 22.04.3 LTS, k3s version v1.28.4+k3s2
  • KeyCloak is preinstalled and accessible externally.
  • Traefik is encrypting http to https traffic for both KeyCloak and Kubeflow
  • DNS provider is CloudFlare

Configuration

File: manifests/common/oidc-client/oidc-authservice/base/params.env:

AUTHSERVICE_URL_PREFIX=/oauth2 # Changed from /authservice (oddly, this seemed to make it match similar to the redirect behavior of dex)
OIDC_PROVIDER=https://main.mydomain.org/auth/realms/master
OIDC_AUTH_URL=https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth
OIDC_SCOPES=acr address email microprofile-jwt offline_access phone profile roles web-origins
REDIRECT_URL=https://kubeflow.mydomain.org/oauth2/callback
SKIP_AUTH_URLS=/login
USERID_HEADER=kubeflow-userid
USERID_PREFIX=
USERID_CLAIM=email
PORT="8080"
STORE_PATH=/var/lib/authservice/data.db # this is probably not needed anymore

File: manifests/common/oidc-client/oidc-authservice/base/secret_params.env:

CLIENT_ID=kubeflow
CLIENT_SECRET=<my_keycloak_generated_secret>

File: manifests/common/oidc-client/oauth2-proxy/base/kustomization.yaml

  • Has "FORCE_HTTPS" set to true

File: manifests/common/oidc-client/oauth2-proxy/components/istio-external-auth/kustomization.yaml

  • Has cloudflare config set (authorizationpolicy.istio-ingressgateway-oauth2-proxy-cloudflare.yaml)

File: manifests/common/oidc-client/oauth2-proxy/base/oauth2-proxy-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
 name: oauth2-proxy
 labels:
   app: oauth2-proxy
data:
 oauth2_proxy.cfg: |
   provider = "oidc"
   oidc_issuer_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth" # URL to auth UI
   scope = "acr address email microprofile-jwt offline_access phone profile roles web-origins"
   upstreams = "static://200"
   email_domains = [ "*" ]
   skip_auth_regex=["/login/.*"] # Don't launch authenticator for these paths
   skip_oidc_discovery = true
   login_url = "/login/oidc" # The url for launching the authenticator? *Although this is not mentioned in browser network requests!*
   redeem_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/token"
   oidc_jwks_url = "https://main.mydomain.org/auth/realms/master/protocol/openid-connect/keys" # This is configured on KeyCloak
   skip_provider_button = true
  
   cookie_name = "oauth2_proxy_kubeflow"
   cookie_expire = "24h"
   cookie_refresh = 0 #"5m"
   code_challenge_method = "S256"
   redirect_url = "https://main.mydomain.org/oauth2/callback" # This is what it tells KeyCloak to redirect back to
   relative_redirect_url = false #true
binaryData: {}

Network path from Firefox:

  • [GET, 302] /
  • [GET] https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth?client_id=kubeflow&redirect_uri=https%3A%2F%2Fkubeflow.knoxds.org%2Foauth2%2Fcallback&response_type=code&scope=openid+acr+address+email+microprofile-jwt+offline_access+phone+profile+roles+web-origins&state=MTcxMTk0NDY0N3xOd3dBTkVsWlJsZEVTMUJGUjFsRFJVcEpTVFpPV1V4U01sUkhTVTVPTmpkVVFVNDNURWd5VUZReldsZEJVRXRSUlVWUE5FOUlRa0U9fIFkV4ZOKtDawkCocUHXiQLGf3CIsCZmBVsx8-xj6JnA
  • [GET] Many KeyCloak UI asset requests...
  • [POST, 302] https://main.mydomain.org/auth/realms/master/login-actions/authenticate?session_code=HZUik4W9zB76cd2j28AvOFsXGuNGqLuiZj-r5QG0I4A&execution=97e7b76a-90a2-4de8-b2ae-edbbc0f8d21a&client_id=kubeflow&tab_id=34CDD_rjOpg
  • [GET, 403] https://kubeflow.mydomain.org/oauth2/callback?state=MTcxMTk0NDY0N3xOd3dBTkVsWlJsZEVTMUJGUjFsRFJVcEpTVFpPV1V4U01sUkhTVTVPTmpkVVFVNDNURWd5VUZReldsZEJVRXRSUlVWUE5FOUlRa0U9fIFkV4ZOKtDawkCocUHXiQLGf3CIsCZmBVsx8-xj6JnA&session_state=133b466c-6d49-4e1b-8886-c1f1630f2425&iss=https%3A%2F%2Fmain.mydomain.org%2Fauth%2Frealms%2Fmaster&code=af416245-40f8-4a04-85ad-0408460e709b.133b466c-6d49-4e1b-8886-c1f1630f2425.4dd21bf3-bbf5-4734-8de5-46329c4dddf0
    • Request Cookies: "oauth2_proxy_kubeflow", "oidc_state_csrf"
  • [GET] bulma.min.css
  • [GET] all.min.css
  • [GET, 302] favicon.ico
  • [GET, none] https://main.mydomain.org/auth/realms/master/protocol/openid-connect/auth?client_id=kubeflow&redirect_uri=https%3A%2F%kubeflow.mydomain.org%2Foauth2%2Fcallback&response_type=code&scope=openid+acr+address+email+microprofile-jwt+offline_access+phone+profile+roles+web-origins&state=MTcxMTk0NDY1MnxOd3dBTkVrMldWWldXRlZUUmxoTFExUkRSbGd5V1VOSFJGaEpRVXBGTlZwYVQxRkdTMGxFV2swME16WlRUVTlUVUVoRE5WVXpRbEU9fLVBVjtueQY7Yf_akLheNmfMBWeme_2PZZdJDFTLq_xL (NS_ERROR_FAILURE)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubeflow-bot kubeflow-bot added this to To Do in Needs Triage Apr 1, 2024
@zenarcher007 zenarcher007 changed the title Unable to find a valid CSRF token with Keycloak Authentication over Dex Unable to find a valid CSRF token with Keycloak Authentication Switching From Dex Apr 1, 2024
@juliusvonkohout
Copy link
Member

Maybe @kromanow94 can help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Needs Triage
To Do
Milestone
No milestone
Development

No branches or pull requests
2 participants
@juliusvonkohout @zenarcher007