failing kubernetes auth with oidc and azure apps and authorization with a specified Security Group

[guillaumedossantos]

I'm trying to set up SSO between our (regular, not AKS) kubernetes clusters and Azure AD. Currently trying with kubectl binary installed on my computer.
It works when no groups are involved, but we want to filter by security group (accounts on AAD are synced from our onprem Active Directory), no kube RBAC involved.

Setup is inspired by https://medium.com/@olemarkus/using-azure-ad-to-authenticate-to-kubernetes-eb143d3cce10 and https://docs.microsoft.com/fr-fr/azure/aks/azure-ad-integration :

• web app for kube api server configured to expose its API (add scope etc...) with app ID : abc123

• native app for client kubectl configured with addition of api permission from the web app, with app ID : xyz456

• kube api server yaml manifest , I add :

- --oidc-client-id=spn:abc123
- --oidc-issuer-url=https://sts.windows.net/OurAADTenantID

• config kubectl binary :

kubectl config set-cluster test-legacy-2 --server=https://192.168.x.y:4443 --certificate-authority=/somelocation/ca.pem

kubectl config set-credentials USER@mydomain.com 
--auth-provider=azure 
--auth-provider-arg=environment=AzurePublicCloud 
--auth-provider-arg=client-id=xyz456 
--auth-provider-arg=tenant-id=OurAADTenantID 
--auth-provider-arg=apiserver-id=abc123

Also in the Azure client app manifest, had to specify :
"allowPublicClient":true,
"oauth2AllowIdTokenImplicitFlow":true
Otherwise had error "Failed to acquire a token: acquiring a new fresh token: waiting for device code authentication to complete: autorest/adal/devicetoken: Error while retrieving OAuth token: Unknown Error". Found on MicrosoftDocs/azure-docs#10326

Issues start when trying to filter on some security group that I find in the JWT as per https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens

I am receiving a format error even though the JWT Azure sends me does contain the groups in the right format (json array of strings)

Config :

• In azure web app manifest to have the groups in my JWT :
  "groupMembershipClaims": "SecurityGroup",

• kube api server yaml manifest :
  - --oidc-groups-claim=groups
- --oidc-required-claim=groups=bbc2eedf-79cd-4505-9fb4-39856ed3790e

with the string here being the GUID of my target security group.

I am receiving error: You must be logged in to the server (Unauthorized) on output of kubectl and the kube api server logs provide me this authentication.go:62] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, oidc: parse claim groups: json: cannot unmarshal array into Go value of type string]]

But I don't understand why it is not happy cause when I decode the JWT I do have

"groups": [
    "00530f35-0013-4237-8947-6e3f6a7895ca",
    "bbc2eedf-79cd-4505-9fb4-39856ed3790e",
    "17dff614-fd68-4a38-906c-69561daec8b7"
  ],

which to my knowledge is a well-formatted json array of strings...

[guillaumedossantos]

Also tried as per https://kubernetes.io/docs/reference/access-authn-authz/authentication/#openid-connect-tokens to provide kubectl config with oidc arguments based on first successful authentication with the azure app (config above), but same errors :

kubectl config set-credentials USER@mydomain.com 
--auth-provider=oidc 
--auth-provider-arg=idp-issuer-url=https://sts.windows.net/OurAADTenantID
--auth-provider-arg=client-id=kubernetes 
--auth-provider-arg=client-secret= native app xyz456 client secret
--auth-provider-arg=refresh-token= Z Y X W V U T S R Q P O ...
--auth-provider-arg=idp-certificate-authority=/somelocation/ca.pem
--auth-provider-arg=id-token= A B C D E F G H I J K L ...

[athenabot]

/sig azure

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

[k8s-ci-robot]

@athenabot: The label(s) sig/azure cannot be applied. These labels are supported: api-review, community/discussion, community/maintenance, community/question, cuj/build-train-deploy, cuj/multi-user, platform/aws, platform/azure, platform/gcp, platform/minikube, platform/other

In response to this:

/sig azure

These SIGs are my best guesses for this issue. Please comment /remove-sig <name> if I am incorrect about one.

🤖 I am a bot run by vllry. 👩‍🔬

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@guillaumedossantos: The label(s) sig/platform/azure cannot be applied. These labels are supported: api-review, community/discussion, community/maintenance, community/question, cuj/build-train-deploy, cuj/multi-user, platform/aws, platform/azure, platform/gcp, platform/minikube, platform/other

In response to this:

/sig platform/azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[guillaumedossantos]

@kubernetes/sig-auth-test-failures

[k8s-ci-robot]

@guillaumedossantos: Reiterating the mentions to trigger a notification:
@kubernetes/sig-auth-test-failures

In response to this:

@kubernetes/sig-auth-test-failures

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[enj]

Required claims must be a string - an array of strings will not work.

kubernetes/staging/src/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/oidc.go

Lines 620 to 634 in 27cf50d

	// check to ensure all required claims are present in the ID token and have matching values.
	for claim, value := range a.requiredClaims {
	if !c.hasClaim(claim) {
	return nil, false, fmt.Errorf("oidc: required claim %s not present in ID token", claim)
	}
	// NOTE: Only string values are supported as valid required claim values.
	var claimValue string
	if err := c.unmarshalClaim(claim, &claimValue); err != nil {
	return nil, false, fmt.Errorf("oidc: parse claim %s: %v", claim, err)
	}
	if claimValue != value {
	return nil, false, fmt.Errorf("oidc: required claim %s value does not match. Got = %s, want = %s", claim, claimValue, value)
	}
	}

[guillaumedossantos]

Required claims must be a string - an array of strings will not work.

kubernetes/staging/src/k8s.io/apiserver/plugin/pkg/authenticator/token/oidc/oidc.go

Lines 620 to 634 in 27cf50d

	// check to ensure all required claims are present in the ID token and have matching values.
	for claim, value := range a.requiredClaims {
	if !c.hasClaim(claim) {
	return nil, false, fmt.Errorf("oidc: required claim %s not present in ID token", claim)
	}
	// NOTE: Only string values are supported as valid required claim values.
	var claimValue string
	if err := c.unmarshalClaim(claim, &claimValue); err != nil {
	return nil, false, fmt.Errorf("oidc: parse claim %s: %v", claim, err)
	}
	if claimValue != value {
	return nil, false, fmt.Errorf("oidc: required claim %s value does not match. Got = %s, want = %s", claim, claimValue, value)
	}
	}

As simple as that. Sorry I didn't read the code, I was sure it was expecting an array of strings because that is what - --oidc-groups-claim=groups expects as per doc

JWT claim to use as the user’s group. If the claim is present it must be an array of strings.

So I was under the impression that --oidc-required-claim did not bother searching an array of strings for the value - doc does not speak about the fact it must be a string

A key=value pair that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. Repeat this flag to specify multiple claims

So if I follow correctly, you just can't use required claim to require a specific group, that's a very important feature missing don't you think??
Maybe somebody should tag this as feature request?

[mikedanese]

Why wouldn't you just not grant permission to that group?

[guillaumedossantos]

Why wouldn't you just not grant permission to that group?

Sorry if I sound silly, but how so ?

Edit : I recall that I don't use RBAC in my tests, because we don't use RBAC yet.

[guillaumedossantos]

I found a workaround.

• Don't use oidc-groups-claim and oidc-required-claim
• In Azure, go to the Properties of the API server App.
• Select Yes in "User assignment required"
• In "Users and groups" add the specific Security Group you want to filter on
• To test : Remove yourself from the Security Group
• Wait for the token to expire (in my case it was 1 hour)
• You can't log in anymore

[ahjsun]

I am having the same error.

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code BJZQH4LQV to authenticate.
E0428 17:26:32.610157 2793 azure.go:154] Failed to acquire a token: acquiring a new fresh token: waiting for device code authentication to complete: autorest/adal/devicetoken: Error while retrieving OAuth token: Unknown Error

Setting up SSO between our (regular, not AKS) kubernetes clusters and Azure AD

Followed the document : https://docs.microsoft.com/en-us/azure/aks/azure-ad-integration

Could anyone pls help

[enj]

@guillaumedossantos @ahjsun with #121078 merged, this should be possible now as an alpha feature in v1.29.