Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove default-allowlist entries for Groovy source files related to Declarative #538

Merged
merged 3 commits into from
Jan 3, 2024

Conversation

dwnusbaum
Copy link
Member

@dwnusbaum dwnusbaum commented May 17, 2022

In 76a7681, we added an allowlist mechanism for loading arbitrary Groovy source files from the classpath. This included a default-allowlist file with entries related to various plugins that relied on the previous behavior (typically to implement some kind of DSL). Now that the security fix is out, we can update Declarative to use the new API and remove a lot of the entries in default-allowlist.

We will also need to update docker-workflow because of /org/jenkinsci/plugins/docker/workflow/declarative/AbstractDockerPipelineScript.groovy, but kubernetes and amazon-ecs will be fixed indirectly via Declarative without requiring any code changes.

Even once the downstream fixes are merged and released, we should wait a while to merge and release this to avoid unnecessary upgrade complexity for users, so I plan on leaving this in draft state for a few weeks even after downstream changes get released.

Downstream PRs:

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests - that demonstrates feature works or fixes the issue

@jglick
Copy link
Member

jglick commented Jul 6, 2022

Is there a reason this is still in draft?

@dwnusbaum
Copy link
Member Author

Is there a reason this is still in draft?

Yes, because it should not be released until the downstream PRs are in widespread use on the relevant baseline or else users who update will be unable to use pipeline-model-definition, docker-workflow, kubernetes, and amazon-ecs. docker-workflow got released two weeks ago, so I will check stats in a bit.

@dwnusbaum
Copy link
Member Author

dwnusbaum commented Jul 6, 2022

Currently, for users running Jenkins 2.332.1 or newer:

  • pipeline-model-definition: 31364 installations are running the required plugin version out of 121820 (45% of 270710) that are running a new enough Jenkins version, so around 26% of plugin users running 2.332.1 could run this PR without issues.
    • Roughly 28% are on the prior plugin version (so we should hit 50% soon). It looks like maybe another 25-30% are split across older CD plugin versions, and then the remaining 2.332.1 users are pretty staggered across plugin versions.
  • docker-workflow: 5973 installations are running the required plugin version out of 46072 (33% of 139612) that are running a new enough Jenkins version, so around 13% of plugin users running 2.332.1 could run this PR without issues.
    • Roughly 71% of 2.322.1+ users are on the prior plugin version and will probably be ready soon.

We can tell users to upgrade things simultaneously in the release notes, but given that this is just a developer-oriented cleanup and the result of not upgrading things simultaneously is pretty severe (even if it is easy to fix once you know what to do), I am inclined to wait another few weeks until at least a majority of eligible users are running the required versions.

@dwnusbaum
Copy link
Member Author

As of the August update to https://stats.jenkins.io/, for users running Jenkins 2.332.1 or newer:

  • pipeline-model-definition: 93004 installations are running the required plugin version out of 141134 (52% of 271413) that are running a new enough Jenkins version, so around 66% of plugin users running 2.332.1 or newer could run this PR without issues.
  • docker-workflow: 30204 installations are running the required plugin version out of 53344 (39% of 136779) that are running a new enough Jenkins version, so around 57% of plugin users running 2.332.1 or newer could run this PR without issues.

I know users don't like upgrading plugins, but I am still somewhat surprised that uptake is so low after 3 months. I am inclined to wait a bit longer for the same reasons as in my last comment.

@dwnusbaum
Copy link
Member Author

As of the October update to https://stats.jenkins.io/, for users running Jenkins 2.332.1 or newer:

  • pipeline-model-definition: 111482 installations are running the required plugin version (or newer) out of 151548 (57% of 265873) that are running a new enough Jenkins version, so around 74% of plugin users running 2.332.1 or newer could run this PR without issues.
  • docker-workflow: 37659 installations are running the required plugin version (or newer) out of 58068 (44% of 131973) that are running a new enough Jenkins version, so around 65% of plugin users running 2.332.1 or newer could run this PR without issues.

Maybe that is good enough to go ahead and release this along with a warning in the release notes? It's not really burning a hole in my pocket or anything, so we could keep waiting.

@jglick
Copy link
Member

jglick commented Nov 9, 2022

The question which our stats does not really answer is what portion of users update some but not all plugins.

@jglick
Copy link
Member

jglick commented Nov 28, 2023

@dwnusbaum ping, should we merge this finally?

@dwnusbaum dwnusbaum marked this pull request as ready for review January 2, 2024 19:37
@dwnusbaum dwnusbaum requested a review from a team as a code owner January 2, 2024 19:37
@dwnusbaum
Copy link
Member Author

Yes I think it should be fine now.

@jglick jglick added developer and removed internal labels Jan 3, 2024
@jglick jglick merged commit c43e04d into jenkinsci:master Jan 3, 2024
14 checks passed
@dwnusbaum dwnusbaum deleted the post-SECURITY-359 branch January 3, 2024 16:24
@chickenkiller
Copy link

Hello,

for information, I got an issue with the kubernetes plugin due to this. Here is the stack trace I have :

Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 81396a71-3130-44a4-a1db-04ad482fa418
java.lang.ClassNotFoundException: org.csanchez.jenkins.plugins.kubernetes.pipeline.KubernetesDeclarativeAgentScript
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:702)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:812)
at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:800)
at org.jenkinsci.plugins.pipeline.modeldefinition.withscript.WithScriptDescribable.getScript(WithScriptDescribable.java:55)
at org.jenkinsci.plugins.pipeline.modeldefinition.agent.DeclarativeAgent.getScript(DeclarativeAgent.java:56)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at org.jenkinsci.plugins.workflow.cps.LoggingInvoker.methodCall(LoggingInvoker.java:105)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:594)
at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(ModelInterpreter.groovy:76)
at WorkflowScript.run(WorkflowScript:65)
at _**cps.transform**_(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:90)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:116)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:85)
at jdk.internal.reflect.GeneratedMethodAccessor485.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:75)
at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:65)
at jdk.internal.reflect.GeneratedMethodAccessor489.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:152)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:146)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:423)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:331)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:295)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:97)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)

Re-installing version 3826.v3b_5707fe44da_ of workflow-cps plugin makes it work again. Is the kubernetes plugin team aware of this?

@jglick
Copy link
Member

jglick commented Jan 9, 2024

@chickenkiller have you neglected to update some other plugins? This PR should only have mattered for people running extremely outdated versions.

@chickenkiller
Copy link

@chickenkiller have you neglected to update some other plugins? This PR should only have mattered for people running extremely outdated versions.

No I updated absolutely everything. The stack trace shows that the kubernetes plugin is not even called in the stack, so I'm a bit puzzled. It seems to use a generic mechanism to load the groovy script stored in its resources. My jenkins version is 2.426.2, deployed via the latest Helm Chart.

@chickenkiller
Copy link

@jglick This line is the culprit: https://github.com/jenkinsci/workflow-cps-plugin/pull/538/files#diff-b12766f1439ade0a5b3ad1be7d7e93437350225fc182a52f178c8e75972d2bffL29
This file is not referenced anywhere in kubernetes-plugin though the declarative model loads it (and needs it). I wanted to make a P/R on kubernetes-plugin to solve this but I don't know how to solve it because I don't know where the file is dynamically found and loaded.

@jglick
Copy link
Member

jglick commented Jan 9, 2024

https://github.com/jenkinsci/kubernetes-plugin/blob/4230d0ccd951f44657b16fd8f867defb03180fc6/src/main/resources/org/csanchez/jenkins/plugins/kubernetes/pipeline/KubernetesDeclarativeAgentScript.groovy#L30 but hmm it seems that the downstream PRs were forgotten for kubernetes and also amazon-ecs @dwnusbaum?

@chickenkiller
Copy link

I mean, I found no occurrence of loadClass() method inside kubernetes-plugin repo, so I don't know where to override the isAllowed() method?

@jglick
Copy link
Member

jglick commented Jan 9, 2024

I am checking this now. We do cross-check compatibility of plugin updates in jenkinsci/bom but the CI builders used there cannot run in-Kubernetes tests unfortunately.

@dwnusbaum
Copy link
Member Author

Implementations of DeclarativeAgentScript are supposed to register themselves automatically. Not sure what is happening here.

@chickenkiller
Copy link

Thank you very much @jglick and @dwnusbaum ! Let me know if I can help testing anything.

@jglick
Copy link
Member

jglick commented Jan 9, 2024

CC @Vlatombe and any other @jenkinsci/kubernetes-plugin-developers

@jglick
Copy link
Member

jglick commented Jan 9, 2024

KubernetesDeclarativeAgentTest.declarative seems to pass even with this update…looking more.

@dwnusbaum
Copy link
Member Author

dwnusbaum commented Jan 9, 2024

See WithScriptDescriptor.getScriptResource and .getScriptClass along with WithScriptDescriptor$WithScriptAllowlist, and then KubernetesDeclarativeAgentImpl$DescriptorImpl extends DeclarativeAgentDescriptor which extends WithScriptDescriptor. Maybe some kind of class loading problem here and it needs to use uberClassLoader? Never mind, looking at the stack trace now the error is here.

@jglick
Copy link
Member

jglick commented Jan 9, 2024

@dwnusbaum unless you know what the problem is and how to fix it soon, would you mean filing a temporary revert PR?

@dwnusbaum
Copy link
Member Author

Yes, I am looking at this but will file a PR to revert things if I don't know what is happening in the next half hour or so.

@jglick
Copy link
Member

jglick commented Jan 9, 2024

KubernetesDeclarativeAgentTest.declarative seems to pass even with this update

It occurs to me that it might not exercise the right class loader code since it is not using RealJenkinsRule. I will see if I can reproduce the error in a test.

@dwnusbaum
Copy link
Member Author

@chickenkiller If you still have your Jenkins logs from earlier, do you see a message like "Preventing /org/csanchez/jenkins/plugins/kubernetes/pipeline/KubernetesDeclarativeAgentScript.groovy from being loaded without sandbox protection in . To allow access to this file, add any suffix of its URL to the system property ..."?

I tried to reproduce this in a non-test Jenkins instance but was not able to, at least with my simple case.

For now I will file a PR to add the scripts for the plugins that depend on Declarative back to the default allowlist.

dwnusbaum added a commit to dwnusbaum/workflow-cps-plugin that referenced this pull request Jan 9, 2024
…-359"

This reverts commit c43e04d, reversing
changes made to 3a59e40.
@chickenkiller
Copy link

chickenkiller commented Jan 9, 2024

@chickenkiller If you still have your Jenkins logs from earlier, do you see a message like "Preventing /org/csanchez/jenkins/plugins/kubernetes/pipeline/KubernetesDeclarativeAgentScript.groovy from being loaded without sandbox protection in . To allow access to this file, add any suffix of its URL to the system property ..."?

@dwnusbaum indeed I get the following message:

2024-01-09 15:51:42.803+0000 [id=72]	WARNING	o.j.p.w.c.GroovySourceFileAllowlist$ClassLoaderImpl#getResource: Preventing jar:file:/var/jenkins_home/plugins/kubernetes/WEB-INF/lib/kubernetes.jar!/org/csanchez/jenkins/plugins/kubernetes/pipeline/KubernetesDeclarativeAgentScript.groovy from being loaded without sandbox protection in MyPipeline/develop #192. To allow access to this file, add any suffix of its URL to the system property ‘org.jenkinsci.plugins.workflow.cps.GroovySourceFileAllowlist.DefaultAllowlist.ALLOWED_SOURCE_FILES’ (use commas to separate multiple files). If you want to allow any Groovy file on the Jenkins classpath to be accessed, you may set the system property ‘org.jenkinsci.plugins.workflow.cps.GroovySourceFileAllowlist.DISABLED’ to true.

I haven't spotted the message before, sorry for not communicating.

@jglick
Copy link
Member

jglick commented Jan 9, 2024

No luck so far either (jenkinsci/kubernetes-plugin#1494). My only suspicion so far is the use of an imperative idiom in WithScriptDescriptor.<init> which looks up a different extension (WithScriptAllowlist). Guice initialization in Jenkins is notoriously weird (jenkinsci/script-security-plugin#552, e.g.) so this seems risky; perhaps the presence of some unnamed plugin triggers an unusual initialization sequence. Could WithScriptAllowlist.isAllowed look up WithScriptDescriptors on the fly, or would this be a performance bottleneck?

jglick added a commit that referenced this pull request Jan 9, 2024
Revert "Merge pull request #538 from dwnusbaum/post-SECURITY-359"
@dwnusbaum
Copy link
Member Author

dwnusbaum commented Jan 9, 2024

IDK. While testing I sometimes saw two instances of WithScriptAllowlist when trying to print WithScriptAllowlist.scriptUrls from the script console, but this only ever happened when loading plugins dynamically.

Could WithScriptAllowlist.isAllowed look up WithScriptDescriptors on the fly, or would this be a performance bottleneck?

I am not sure how performance-sensitive this code is, or how much of a difference that approach would make. The allowlists are only checked for resources that end with .groovy, and I guess each class will only be loaded once per build, but from some quick testing it looks like GroovySourceFileAllowlist.isAllowed is called twice for each class load. I guess we can switch to that approach and then add a global cache of allowed resources to GroovySourceFileAllowlist if performance is a concern.

@jglick
Copy link
Member

jglick commented Jan 9, 2024

add a global cache of allowed resources to GroovySourceFileAllowlist

Or to WithScriptAllowlist.

@jglick
Copy link
Member

jglick commented Jan 9, 2024

@chickenkiller
Copy link

@jglick & @dwnusbaum, I confirm the latest version 3837.v305192405b_c0 works and fixes my issue!

Thank you very much for your help on the subject and for your amazing reactivity. Jenkins community is awesome!

@jglick
Copy link
Member

jglick commented Jan 16, 2024

Good. Unfortunately we remain unable to reproduce the problem and do not have a solid hypothesis regarding its cause, making it hard to know when if ever we could clean up the obsolete entries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants