NPE when token is to be refreshed #5817

adietish · 2024-03-19T14:06:26Z

Describe the bug

This bug was reported in redhat-developer/intellij-kubernetes#726
It happens in the kubernetes plugin for intellij when browsing an EKS cluster on AWS.

2024-03-18 15:05:59,803 [ 255348]   WARN - #com.redhat.devtools.intellij.kubernetes.tree.TreeStructure - Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
	at java.base/java.util.Base64$Decoder.decode(Base64.java:589)
	at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getDefaultHttpClientWithPemCert(OpenIDConnectionUtils.java:292)
	at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getOIDCProviderTokenEndpointAndRefreshToken(OpenIDConnectionUtils.java:330)
	at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.resolveOIDCTokenFromAuthConfig(OpenIDConnectionUtils.java:86)
	at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.extractNewAccessTokenFrom(TokenRefreshInterceptor.java:83)
	at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.refreshToken(TokenRefreshInterceptor.java:76)
	at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.before(TokenRefreshInterceptor.java:58)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.consumeBytes(StandardHttpClient.java:65)
	at io.fabric8.kubernetes.client.http.SendAsyncUtils.bytes(SendAsyncUtils.java:51)
	at io.fabric8.kubernetes.client.http.HttpResponse$SupportedResponses.sendAsync(HttpResponse.java:105)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.sendAsync(StandardHttpClient.java:52)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.retryWithExponentialBackoff(OperationSupport.java:604)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:581)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.submitList(BaseOperation.java:414)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:427)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:392)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:93)

The kube config looks as follows (anonymized):

---
apiVersion: "v1"
kind: "Config"
clusters:
- cluster:
    certificate-authority: "certs/xxxxx-id/k8s-ca.crt"
    server: "https://api.a-central-1.aws.xxxx.com"
  name: "xxxxx-id"
contexts:
- context:
    cluster: "xxxxx-id"
    namespace: "id-stag"
    user: "stefano-xxxxx-id"
  name: "xxxxx-id"

current-context: "xxxxx-id"
preferences: {}
users:
- name: "stefano-xxxxx-id"
  user:
    auth-provider:
      config:
        client-id: "de-k8s-authenticator"
        client-secret: "pUBnBOY8[...]ijwadxreNGQok"
        id-token: "eyJhbGciOiJSUzI1NiIsImtpZCI6IjRjZmRiYjcx[...]eiuhedeaZmQifQ.eyJpc3MiOiJodHRwczovL2RleC5pZC5hd3MuY3[...]6IkNpUXhZV1l5TlRnd05TMWtZ[...]aGVudGljYXRvciIsImV4cCI6MTcxMDgxMzc3NSwiaWF0IjoxNzEwNzcwNTc1LCJhdF9oYXNoIjoiVHAwenM1RFBNRjI2WnNpOEI1cmdyUSIsImVtYWlsIjoic3RlZmFuby5icnVzYUB2b"
        idp-issuer-url: "https://de.id.aws.xxxxx.com"
        refresh-token: "Chl4eG0zZmhzd2aeiuhE[...]kaHRhN3h6bXlqZmtoYXdrY2Zt"
      name: "oidc"

Fabric8 Kubernetes Client version

The client library used is 6.4.0

Steps to reproduce

In the plugin one needs to connect to the EKS instance and start to browse the available resources. It initially works and starts to fail once the token needs to be refreshed.

Expected behavior

In the plugin requesting resources starts to fail once the token fails to be refreshed with the stacktrace shown above

Runtime

Kubernetes (vanilla)

Kubernetes API Server version

1.25.3@latest

Environment

Amazon

Fabric8 Kubernetes Client Logs

See log pasted in issue description

Additional context

No response

The text was updated successfully, but these errors were encountered:

adietish · 2024-03-19T14:19:37Z

CC: @deejonz, @rohanKanojia

adietish · 2024-03-19T14:27:25Z

This issue looks very similar to #4960

…nd caCertData as a fallback option when `idp-certificate-authority-data` is not specified Related to fabric8io#5817 Currently, we fall back to caCertData specified in Config when `idp-certificate-authority-data` is not specified. We should also consider reading cert data from caCertFile. Signed-off-by: Rohan Kumar <rohaan@redhat.com>

adietish · 2024-04-02T15:19:03Z

#5818 seems to fix the issue for the user. He commented the issue in intellij-plugin redhat-developer/intellij-kubernetes#726 (comment) that issue is gone for him when using the private build I made for him (which is using the client with the fix)

…nd caCertData as a fallback option when `idp-certificate-authority-data` is not specified Related to fabric8io#5817 Currently, we fall back to caCertData specified in Config when `idp-certificate-authority-data` is not specified. We should also consider reading cert data from caCertFile. Signed-off-by: Rohan Kumar <rohaan@redhat.com>

deejonz · 2024-04-03T07:38:31Z

@adietish just in case, when I tried this morning with intellij already opened - I did right click -> "refresh" (as the plugin was not responding) and I've got again that error...

java.lang.NullPointerException: Cannot invoke "String.getBytes(java.nio.charset.Charset)" because "src" is null
	at java.base/java.util.Base64$Decoder.decode(Base64.java:589)
	at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getDefaultHttpClientWithPemCert(OpenIDConnectionUtils.java:277)
	at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.getOIDCProviderTokenEndpointAndRefreshToken(OpenIDConnectionUtils.java:315)
	at io.fabric8.kubernetes.client.utils.OpenIDConnectionUtils.resolveOIDCTokenFromAuthConfig(OpenIDConnectionUtils.java:94)
	at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.lambda$new$0(TokenRefreshInterceptor.java:50)
	at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.extractNewAccessTokenFrom(TokenRefreshInterceptor.java:132)
	at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.refreshToken(TokenRefreshInterceptor.java:124)
	at io.fabric8.kubernetes.client.utils.TokenRefreshInterceptor.before(TokenRefreshInterceptor.java:77)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.consumeBytesOnce(StandardHttpClient.java:101)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$consumeBytes$1(StandardHttpClient.java:92)
	at io.fabric8.kubernetes.client.utils.AsyncUtils.retryWithExponentialBackoff(AsyncUtils.java:75)
	at io.fabric8.kubernetes.client.utils.AsyncUtils.retryWithExponentialBackoff(AsyncUtils.java:68)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.retryWithExponentialBackoff(StandardHttpClient.java:166)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.consumeBytes(StandardHttpClient.java:90)
	at io.fabric8.kubernetes.client.http.SendAsyncUtils.bytes(SendAsyncUtils.java:51)
	at io.fabric8.kubernetes.client.http.HttpResponse$SupportedResponses.sendAsync(HttpResponse.java:105)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.sendAsync(StandardHttpClient.java:77)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:547)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.submitList(BaseOperation.java:440)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:450)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:418)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:97)
	at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.loadAllResources(NamespacedResourceOperator.kt:68)
	at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.getAllResources(NamespacedResourceOperator.kt:54)
	at com.redhat.devtools.intellij.kubernetes.model.resource.NamespacedResourceOperator.getAllResources(NamespacedResourceOperator.kt:36)
	at com.redhat.devtools.intellij.kubernetes.model.context.ActiveContext.getAllResources(ActiveContext.kt:148)
	at com.redhat.devtools.intellij.kubernetes.model.ResourceModel.getAllResources(ResourceModel.kt:122)
	at com.redhat.devtools.intellij.kubernetes.model.ResourceModel.getAllResources$default(ResourceModel.kt:121)
	at com.redhat.devtools.intellij.kubernetes.model.FilterableResources.list(ResourceModelQuery.kt:63)
	at com.redhat.devtools.intellij.kubernetes.tree.KubernetesStructure$createWorkloadElements$8$3.invoke(KubernetesStructure.kt:245)
	at com.redhat.devtools.intellij.kubernetes.tree.KubernetesStructure$createWorkloadElements$8$3.invoke(KubernetesStructure.kt:242)
	at com.redhat.devtools.intellij.kubernetes.tree.AbstractTreeStructureContribution$ElementNode.getChildElements(AbstractTreeStructureContribution.kt:97)
	at com.redhat.devtools.intellij.kubernetes.tree.AbstractTreeStructureContribution.getChildElements(AbstractTreeStructureContribution.kt:28)
	at com.redhat.devtools.intellij.kubernetes.tree.TreeStructure.getChildElements(TreeStructure.kt:73)
	at com.redhat.devtools.intellij.kubernetes.tree.TreeStructure.getChildElements(TreeStructure.kt:66)
	at com.intellij.ui.tree.StructureTreeModel.getValidChildren(StructureTreeModel.java:411)
	at com.intellij.ui.tree.StructureTreeModel.validateChildren(StructureTreeModel.java:329)
	at com.intellij.ui.tree.StructureTreeModel.getNode(StructureTreeModel.java:323)
	at com.intellij.ui.tree.StructureTreeModel.getChildren(StructureTreeModel.java:343)
	at com.intellij.ui.tree.AsyncTreeModel$CmdGetChildren.computeNode(AsyncTreeModel.java:613)
	at com.intellij.ui.tree.AsyncTreeModel$Command.computeNode(AsyncTreeModel.java:489)
	at com.intellij.util.concurrency.Invoker$Task.run(Invoker.java:381)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:192)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$12(CoreProgressManager.java:610)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:685)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:641)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:609)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:78)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:179)
	at com.intellij.util.concurrency.Invoker.startTask(Invoker.java:236)
	at com.intellij.util.concurrency.Invoker.invokeSafely(Invoker.java:194)
	at com.intellij.util.concurrency.Invoker.lambda$offerSafely$0(Invoker.java:177)
	at com.intellij.util.concurrency.Invoker$Background.lambda$offer$0(Invoker.java:508)
	at com.intellij.util.concurrency.ContextRunnable.run(ContextRunnable.java:27)
	at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:244)
	at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:30)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.executeFirstTaskAndHelpQueue(BoundedTaskExecutor.java:222)
	at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:218)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:210)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-04-03 09:34:12,896 [69649811]   INFO - #c.i.u.MemTester - not supported: not an executable file: null```

rohanKanojia · 2024-04-03T08:20:21Z

@deejonz : Hello, Are you sure you're using the right build of IntelliJ plugin (which is built on top of #5818) ?

I tested refresh flow on EKS+OIDC cluster provided by @adietish multiple times this morning. With fix I didn't get any NPE but on 6.11.0 I was getting NPE.

deejonz · 2024-04-03T08:24:14Z

I'm using 1.2.4-SNAPSHOT
did you try to leave the plugin open all night and try to refresh in the morning?

rohanKanojia · 2024-04-03T08:27:50Z

@deejonz : I'm not using the plugin, I checked it on the reproducer I shared with you. I followed these steps:

login to the K8s Cluster, run BasicPodListTest it should run as expected.
Check expiry timestamp of the access token from kubeconfig (in our case it was only 5 minutes)
Wait for 5 minutes and re run BasicPodListTest, KubernetesClient should automatically refresh token once it notices existing token has expired, test should pass and list pods
Check kubeconfig to see new access token has been persisted by KubernetesClient

deejonz · 2024-04-03T08:30:11Z

I tried to wait 1 or 2 hours and it was still working fine. But after the whole night it was broken again.
Also after restarting intellij it's saying "invalid token, password, ..." and it works again only after doing "kubectl get po -n mycontext" from the terminal...

adietish · 2024-04-03T09:09:40Z

@deejonz: thanks again for all the testing. Tremendous opportunity for us to test your setup.
I dont understand why:

you still get an NPE?
how is it that accessing the cluster using kubectl would heal it?
why would it fail after a night only?

@rohanKanojia for 2., is it possible that the client lib fails to refresh the token while kubectl then succeeds "healing" the client library?

@deejonz: can you please verify that the "fixed" kubernetes client is installed? Go to ~/Library/Application\ Support/JetBrains/IntelliJIdea2023.3/plugins/Kubernetes\ by\ Red\ Hat/lib/ and list the jars within. It should list kubernetes-client-6.11-SNAPSHOT.jar

I see that there are new commits in the kubernetes-client PR. I can thus rebuild the plugin and let you test it?

deejonz · 2024-04-03T09:15:11Z

I have kubernetes-client-6.10.0
please rebuild I will test

rohanKanojia · 2024-04-03T09:21:43Z

@adietish I had just rebased the pull request last night. I haven't added any additional commits.

is it possible that the client lib fails to refresh the token while kubectl then succeeds "healing" the client library?

Yes, I think you're right. It looks like kubectl succeeds in doing token refresh and updates kubeconfig with new token, which is then used by KubernetesClient.

adietish · 2024-04-03T09:38:20Z

@deejonz: https://drive.google.com/file/d/1R9_8JCm2c83fPxwg5K3fdyaMWS-VpUFb/view?usp=drive_link

md5 for it is 70dd1b665ada9f8c5fc53405da04d84c.

You need to uninstall the prior plugin, restart IDEA and then reinstall it. To verify that things are correct you can inspect ~/Library/Application\ Support/JetBrains/IntelliJIdea2023.3/plugins/Kubernetes\ by\ Red\ Hat/lib/ and make sure that you have kubernetes-client-6.11-SNAPSHOT.jar

deejonz · 2024-04-03T09:41:50Z

ok now I have 6.11-SNAPSHOT

adietish · 2024-04-09T08:08:13Z

@rohanKanojia: @deejonz reported that things are now working for him. I consider this issue in the client as fixed. Merge the PR and close?

…nd caCertData as a fallback option when `idp-certificate-authority-data` is not specified Related to fabric8io#5817 Currently, we fall back to caCertData specified in Config when `idp-certificate-authority-data` is not specified. We should also consider reading cert data from caCertFile. Signed-off-by: Rohan Kumar <rohaan@redhat.com>

…d caCertData as a fallback option when `idp-certificate-authority-data` is not specified (#5818) Related to #5817 Currently, we fall back to caCertData specified in Config when `idp-certificate-authority-data` is not specified. We should also consider reading cert data from caCertFile. Signed-off-by: Rohan Kumar <rohaan@redhat.com>

adietish mentioned this issue Mar 19, 2024

Getting error "src" is null when browsing tree redhat-developer/intellij-kubernetes#726

Closed

rohanKanojia mentioned this issue Mar 19, 2024

fix (kubernetes-client-api) : OpenIDConnectionUtils uses caCertFile and caCertData as a fallback option when idp-certificate-authority-data is not specified #5818

Merged

11 tasks

manusa assigned rohanKanojia Mar 28, 2024

manusa added this to the 6.12.0 milestone Apr 9, 2024

manusa closed this as completed in #5818 Apr 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NPE when token is to be refreshed #5817

NPE when token is to be refreshed #5817

adietish commented Mar 19, 2024 •

edited

adietish commented Mar 19, 2024

adietish commented Mar 19, 2024

adietish commented Apr 2, 2024 •

edited

deejonz commented Apr 3, 2024 •

edited

rohanKanojia commented Apr 3, 2024

deejonz commented Apr 3, 2024 •

edited

rohanKanojia commented Apr 3, 2024

deejonz commented Apr 3, 2024

adietish commented Apr 3, 2024 •

edited

deejonz commented Apr 3, 2024

rohanKanojia commented Apr 3, 2024 •

edited

adietish commented Apr 3, 2024 •

edited

deejonz commented Apr 3, 2024

adietish commented Apr 9, 2024 •

edited

NPE when token is to be refreshed #5817

NPE when token is to be refreshed #5817

Comments

adietish commented Mar 19, 2024 • edited

Describe the bug

Fabric8 Kubernetes Client version

Steps to reproduce

Expected behavior

Runtime

Kubernetes API Server version

Environment

Fabric8 Kubernetes Client Logs

Additional context

adietish commented Mar 19, 2024

adietish commented Mar 19, 2024

adietish commented Apr 2, 2024 • edited

deejonz commented Apr 3, 2024 • edited

rohanKanojia commented Apr 3, 2024

deejonz commented Apr 3, 2024 • edited

rohanKanojia commented Apr 3, 2024

deejonz commented Apr 3, 2024

adietish commented Apr 3, 2024 • edited

deejonz commented Apr 3, 2024

rohanKanojia commented Apr 3, 2024 • edited

adietish commented Apr 3, 2024 • edited

deejonz commented Apr 3, 2024

adietish commented Apr 9, 2024 • edited

adietish commented Mar 19, 2024 •

edited

adietish commented Apr 2, 2024 •

edited

deejonz commented Apr 3, 2024 •

edited

deejonz commented Apr 3, 2024 •

edited

adietish commented Apr 3, 2024 •

edited

rohanKanojia commented Apr 3, 2024 •

edited

adietish commented Apr 3, 2024 •

edited

adietish commented Apr 9, 2024 •

edited